Documentation

Signing in

There are no passwords in this application. Signed out, you land on a page with two buttons, Sign in with Keycloak and Create account, and both hand you to Keycloak. The second adds action=register to the authorization request so you arrive on the registration form rather than the login form; whether that form is reachable at all depends on whether self-registration is enabled in the realm.

Your username, display name and email come from Keycloak on every login and cannot be edited inside the chat. The application creates its own user row the first time it sees you, keyed on the OIDC subject.

Sessions last 8 hours and the two clocks are kept in step deliberately: Keycloak's SSO session idle timeout and the servlet session timeout are both 8h. The browser also watches for mouse, key, scroll, touch and focus events, and after 8 hours of genuine inactivity it posts to /logout itself rather than leaving a stale tab open until someone clicks something.

Channels

A channel is public or private, and the distinction is narrower than it looks. Public means anyone signed in can read it, search it, and download its attachments without joining. Writing always requires membership, in both kinds. So a public channel you have not joined is readable but the composer is not rendered, and the header offers Join channel instead.

Private channels are invisible to non-members: not readable, not searchable, and not returned by channel search. A non-member who reaches one by link gets a short notice telling them to ask an admin for an invitation.

Creating one

The + next to the sidebar's first group heading opens the create form, and the same form appears on the channels landing page when you have no channels yet. It takes a name (up to 120 characters), an optional description (up to 500), and a public or private choice. The name is slugified for the URL: lowercased, every run of non-alphanumerics collapsed to a hyphen, cut at 80 characters. A name with no letters or digits in it is rejected, and so is one whose slug collides with an existing channel.

Whoever creates a channel becomes its first channel admin. There is a workspace-level policy controlling who may create channels at all, either everyone or admins only, defaulting to everyone; it is stored in the database and has no admin screen yet, so changing it today means changing the row.

Members, invites and roles

The people icon in the channel header opens the member list, which anyone who can read the channel may open. It shows each member's avatar, display name and handle, marks channel admins and workspace admins, and, if you are a channel admin, gives every other row a Make admin or Demote button. The server refuses to demote the last remaining admin, on the grounds that an ownerless private channel cannot be repaired from inside the app.

Inviting is done by exact username, from the channel settings panel. Any member can invite, not only admins. There is no bulk invite and no invite link.

Renaming, archiving, deleting

A channel admin can rename a channel and edit its description from the cog in the channel header. Renaming moves the channel's short name in links; existing links keep working, because every route is keyed on the channel's id rather than its name.

Archiving makes a channel read-only and takes it out of the way: it disappears from sidebars and from channel search, refuses posts, replies, reactions, uploads, invites and joins, and stays fully readable and searchable. That is the point of archiving rather than deleting — the project ended, the record did not. It is reversible by a channel admin or a workspace admin.

Deleting is a workspace-admin action, and asks you to type the channel's name first. It destroys the messages, the threads, the attachments on disk, the search index entries, the pins, the saves and the memberships, and credits every uploader's storage back. It cannot be undone. A channel admin — who is often just whoever happened to create the room — gets archive instead, which is the reversible answer to the same problem.

Leaving

Any member can leave, from the same cog. Your messages stay; you are leaving, not retracting. Leaving a private channel is one-way from your side — you will need another invitation to return — and the confirmation says so.

If the last admin leaves, the role passes to the longest-standing remaining member rather than the channel being left with nobody who can invite, rename or archive it. Refusing to let the last admin go would trap the one person who took responsibility for the room, which is the opposite of what you want to encourage.

The sidebar

The sidebar lists every channel you are in, alphabetically, with the ones you have starred lifted into a Favourites group above the rest. Below them are your direct messages and group conversations. That is the whole model: what you see is what you have joined, in an order that does not move.

It used to be a ranked shortlist — your five largest channels and your five most active ones, with everything else behind a line reading N more channels. Ranking is a reasonable idea and a bad sidebar. A list that reorders itself as membership and traffic drift defeats the thing a sidebar is actually for, which is knowing where a channel is without reading; and a channel that falls off the list is one you stop seeing at all.

Starring is how you pin the handful you use constantly. Click the star on any channel row. A favourite is yours alone — it changes nothing for anyone else in the channel.

Unread follows the convention Slack and Mattermost both use, because it is the one that scales: a channel with unread messages is bold, and a number appears only when someone used your name or it is a direct message. A count on every busy channel is noise, and noise teaches people to ignore the badge that matters. Muted channels are dimmed and neither bold nor badged — but they still count unread, because muting is "stop interrupting me", not "pretend nothing happened". A mention in a muted channel still gets a badge, in neutral grey rather than red: it makes no sound, and it is still findable later.

Each row carries a lock icon if the channel is private. Being an admin of a channel is not shown here — it is in the channel's settings panel, where it is something you can act on.

The box at the top does two jobs. It filters the channels already in your sidebar as you type, and it queries the server for channels you have not joined, showing them in the main pane with their description, member count and a Join button. Private channels you are not in never appear.

Live updates follow the channels you have joined, not the rows that happen to be rendered. A mention in any channel you are in raises its badge, plays its sound and fires its notification while you are reading somewhere else.

Messages and Markdown

The composer takes up to 8,000 characters. Enter sends, Shift+Enter inserts a newline, and the box grows to about 260 pixels before it starts scrolling. A Preview pane below it renders what you have typed, and it renders it on the server, using the same code path that will render the posted message. A preview that looks right is therefore a guarantee, not an approximation.

What Markdown actually survives

Bodies are parsed with CommonMark plus GFM tables and autolinking, then sanitized with jsoup before anyone sees them. Sanitization is not a formality here; it is the reason the strict Content-Security-Policy can stay strict. What comes out the other side:

Removed on the way through: raw <script>, <div>, <span>, <img> and inline style attributes. Because images are not on the allowlist, inline Markdown images are stripped; to share a picture, attach it. The one exception to the no-embedding rule is video: a link to YouTube or Vimeo gets a responsive player appended after it, using the privacy-preserving youtube-nocookie host, and those three origins are the only ones the CSP allows in a frame.

Editing, deleting, permalinks

Hovering a message reveals its actions. Edit appears on your own messages only, with no time window, and adds an (edited) marker. Delete appears for the author and for channel admins, and takes the thread replies, attachments, reactions, mentions and search index entries with it. Copy link to message puts a permalink on your clipboard; opening one renders 25 messages either side of the target and shows a banner offering to jump back to the latest.

On a touch device there is no hover, so a 500 ms long press opens an action sheet from the bottom of the screen with the same actions plus a twelve-emoji quick-reaction strip.

Sending, and what you see while it happens

Your own message appears the instant you press Enter, in a pending state, and is reconciled when the server broadcasts it back. That is not cosmetic: the server deliberately does not broadcast a message until its database row has committed, so that nobody is ever shown a line that then failed to persist. The optimistic echo is what hides the few milliseconds that costs. If no confirmation arrives within 12 seconds the bubble is marked failed and offers a Retry.

Typing indicators are published at most once every two seconds while your composer has text in it, and expire four seconds after the last one. They exist in channels only.

Read state is one timestamp per channel per user. A channel is marked read when you open it, and when a message arrives while the tab is both visible and focused. A background tab is deliberately not counted as reading, so unreads do not evaporate because a window was open behind something else. Your own messages never count toward your unread total.

History loads fifty messages at a time as you scroll up. After a reconnect the client backfills forward from the last message it saw, up to 2,500 messages, so a laptop that slept through a meeting catches up rather than showing a hole.

Threads

Threads are a real parent-child relationship on the message, not quoted text, and they are exactly one level deep. Replying to a reply is refused by the server with a message telling you to reply to its parent instead.

Start one from the Reply in thread action, or from the N replies indicator that appears under any message that has them. Either opens a panel to the right of the conversation containing the parent, the replies and a reply composer. Esc closes it. Reply counts update live.

Thread replies stay out of the main channel feed, which is the point: the channel keeps its shape while a side discussion runs. Reactions, editing, deleting and permalinks all work inside the panel. Attachments do not, the thread composer has no attach button.

Direct and group messages

Direct messages live in the same sidebar as channels. They are a separate model underneath — their own tables, their own topics, their own access rule — but they are not a smaller product: a DM does what a channel does.

A one-to-one DM starts from an avatar hovercard. Hover anyone's avatar, name or handle for about a fifth of a second and a card appears with their status and a Send direct message button — or from the + beside the Direct messages heading in the sidebar. Conversations are deduplicated by participant pair, so you always return to the same thread with the same person.

A group is created from the + beside the Direct messages heading. It takes a title and a list of usernames separated by commas or spaces. At most 50 members can be seeded in one go; more can be added afterwards from the group's member panel, by any existing member. If any username is wrong the error is deliberately vague, one or more of those members could not be found, so the form cannot be used to test whether an account exists.

What a DM can do

The same things a channel can, with the differences that follow from a DM being a private room rather than a public one. Threads, typing indicators, read state and a "new messages" line, reactions, editing and deleting your own messages, attachments, Markdown with live preview, mentions, unread badges, permalinks, and search — conversations are in the Lucene index, and only ever visible to their own members.

Each conversation has its own notification level — Default, Every message, Mentions only or Nothing — so a group DM that has become a standing meeting can be turned down without muting the person in it.

There are two account-wide defaults, one for channels and one for conversations, and it is worth knowing why. Channels default to Mentions only: most traffic in a room you joined is other people's business, and you want to be told when it becomes yours. Conversations default to Every message, because a message sent to you and nobody else is addressed to you whether or not it spells your name — a one-to-one where the other person has to type your name to reach you is a broken product. Both live on your profile page.

The split is what lets Mentions only mean what it says on a conversation. With a single default it could not: conversations inherited the channel default, so honouring "mentions only" would have stopped delivering direct messages to every existing account at once, and the setting had to be ignored instead — which in turn made it impossible to say "only tell me when someone names me" about a twenty-person group DM. Now you can, and a 1:1 still notifies. This is the shape Slack uses, reached the same way.

You can leave a group. You cannot leave a one-to-one, and the option is not offered: messaging that person again resolves the same conversation, so leaving could only ever mean hiding it until the next message. Close it and move on instead.

Forwarding a message out of a DM is not offered, deliberately. A DM is the one room with a real expectation of confinement, and no amount of confirmation wording makes a one-click "send this somewhere else" button appropriate there. Copy and paste exists; the difference is that it is your decision rather than a control the product handed you.

Slash commands do not work in conversations.

There is no way to leave a conversation, remove someone from a group, rename a group or delete one. Workspace admins can delete an individual DM message; nobody can delete the conversation.

Voice and video calls

Open a direct message and press the handset in the header for a voice call, or the camera beside it for video. The other person's devices ring; they answer or decline. That is the whole of it.

Calls are one-to-one, and the buttons appear nowhere else. Not in a channel, not in a group DM, not in a note-to-self. This is a limit of how the media travels rather than a policy: a call here is a single connection between two browsers, and a third participant has nowhere to go. Group calls need a media server that receives one stream per person and forwards it to the others, which is a service to operate rather than a feature to switch on. Offering a button that could not work would be worse than not offering one.

Being called

A call rings wherever you are in the app — reading a channel, on your profile page, anywhere. It also rings on every tab and device you have open, and the moment you answer on one the others stop. If you are on another call already, the caller is told you are busy rather than being left listening to a ring nobody will pick up.

An unanswered call stops ringing after 45 seconds. If the person calling you closes their laptop mid-ring, your phone stops immediately rather than waiting that out.

Do Not Disturb silences the ring, not the call. The panel still appears, so a call that arrives while you are looking at the app can be answered — what DND removes is the noise. It is the same rule the rest of the app follows: Do Not Disturb suppresses interruption, never information.

During and after

While connected you get a timer, a mute button, and on a video call a camera toggle. Hanging up takes a deliberate click — Esc declines a call that is ringing and does nothing to one in progress, because the key people press to close things should not end a conversation.

Every call leaves a line in the conversation: Call · 4 min for one that happened, Missed call for one that did not. It is an ordinary message — searchable, exportable, and still legible in five years without anything that knows what a call is.

A declined call is archived as a missed call, and nothing more. The person calling is told in the moment that you declined, because that is useful to them right then. The permanent record does not say it. Writing "declined" into someone's message history puts a verdict on their behalf, and the fact of the call is the part worth keeping.

Where the audio and video actually go

Not through the chat server. It carries the ringing and the connection setup and nothing else — it never receives a packet of audio or video, and could not read one if it did. The media goes through a TURN relay you run yourself (see Calls and TURN), which forwards encrypted traffic without holding the keys to it.

By default every call is relayed, including two people sitting in the same office. That costs bandwidth and buys two things. Calls behave the same for everybody: the alternative is a direct connection that works for most people and fails behind corporate firewalls and symmetric NAT — late, after the ringing has already promised a call. And neither participant learns the other's IP address, which a direct connection necessarily reveals.

Calls need HTTPS. Browsers refuse access to a camera or microphone on an insecure origin, and localhost is the only exception. Over plain http:// on a hostname or LAN address the buttons appear and pressing one reports that calls need HTTPS — that is the browser, not the app. Put the site behind TLS; the proxy guide covers it.

Pinning, saving, forwarding, quoting

Four things you can do with a message that already exists. They sit on the message's hover row — the common ones inline, the rest behind ⋯, because nine icons on hover is worse than five.

• Pin is for the channel. Any member can pin; the channel header shows the count and opens the list. It is where a room keeps the thing you should read first.
• Save is for you. Private, cross-channel, and nobody in the room can tell — it is a personal queue, listed on its own page behind your avatar. Saving is a read, not a write, so you can save from a channel you have only read access to.
• Forward sends a copy to another channel or conversation with an optional comment. The copy quotes the original and links back to it rather than impersonating its author. Forwarding out of a private channel asks you to acknowledge that first, by name; forwarding out of a direct message is not offered at all.
• Quote pulls the message into your composer as a blockquote with a permalink, which is the cheapest of the four and the one you will use most.

Reactions

Click an existing reaction to add or remove yours, or use the Add reaction action to open a picker with a search box and ten category tabs over roughly 650 emoji. Hovering a reaction shows who reacted.

You cannot react to your own message. The button is hidden on your own messages and the server refuses the request as well. It is an opinion rather than a technical limit, and it is worth knowing before you file it as a bug.

There is no custom or uploaded emoji, and no administration screen for emoji. The rate limit is 60 reaction toggles a minute.

Mentions and the inbox

Write @username in a message. The handle is matched after the message has been sanitized, so a mention cannot be forged by writing the markup by hand, and it is skipped inside inline code and fenced blocks, so a code sample containing an @ does not page anyone. A handle that does not resolve to a real user stays plain text.

Typing @ opens a typeahead. It matches display names as well as handles, so @and finds "Alice Anderson", and it leads with the people actually in the room. That matters more than it sounds: mentions resolve against the exact username, but the interface shows people by display name, so before the typeahead existed the only way to mention someone was to already know a string the UI never showed you — and a handle that resolved to nobody notified nobody, silently. Unresolved handles still stay plain text; resolved ones render as a tinted pill, so the composer's live preview tells you whether it worked before you send.

Mentioning everyone

@channel notifies every member of the channel. @here notifies the ones currently connected. @everyone is a synonym for @channel — Slack scopes it to the default channel, a concept this app does not have, and refusing the word outright would fail silently for the people most likely to type it. All three render as a stronger pill than a personal mention, and the typeahead shows how many people each one would reach before you commit.

A mute still wins. A channel set to Nothing stays silent through @channel, because a mute with exceptions is not a mute.

Mentioning someone who is not a member of a private channel creates nothing, so a mention cannot be used to signal into a room the target cannot read. Broadcast mentions obey the same rule by construction: their audience is the membership.

In a group conversation, @channel and @here notify the other participants the same way. In a one-to-one DM they are decoration — the other person is already being notified.

The bell in the top bar is the mention inbox. It carries a count, and opens a list of your most recent mentions with author, channel, relative time and a snippet, each linking straight to the message in context. Mark all as read clears the lot.

Being mentioned raises an in-page toast, and optionally a desktop notification. Permission is not requested on page load; the first toast of a session carries an Enable desktop alerts button instead, so the browser prompt only appears if you ask for it. Notifications are suppressed for the channel you are already looking at in a focused tab.

Notification levels and sounds

Two separate questions, deliberately kept apart: what interrupts you, which is about you and is saved to your account, and which noise it makes, which is about the room you are sitting in and is saved in that browser only.

What interrupts you

An account-wide default plus a per-channel override, the same shape Slack and Mattermost use. Set the default under Profile → Notifications; set one channel under the cog in its header.

Which noise it makes

Mentions and direct messages have their own switch and their own sound, so you can tell them apart without looking — a direct message is someone waiting on you, a mention is your name going past in a room that was talking anyway. Fifteen sounds, synthesised in the browser rather than shipped as files, so there is no audio asset to serve and nothing extra to allow through the content security policy.

These settings live in the browser, not the account: whether you want a noise depends on where you are sitting. The choice is remembered per device and per browser profile.

Browsers refuse to play audio until you have interacted with the page, so the first sound of a session arrives only after your first click or keypress. Nothing is lost — the toast still appears — and it is why the settings page plays the sound when you switch it on.

Presence and custom status

Presence is derived from your WebSocket connection, not from a heartbeat and not from a periodic ping. If you have at least one live session you are connected; if you have none you are offline. On top of that:

1. A manual choice wins over everything and persists until you change it.
2. Otherwise, no live session means Offline.
3. Otherwise, no activity for longer than the away threshold (10 minutes by default) means Away.
4. Otherwise Active.

Because the away transition is computed rather than pushed, clients re-poll presence once a minute to pick it up. Manual changes broadcast immediately.

Click your own avatar in the top bar for the picker: Active, Away, Do not disturb, Appear offline, then View profile, Set a status, Admin console if you have the role, About, and Sign out. It is fully keyboard driven: Enter or Space opens it, arrows and Home/End move, Esc closes.

A custom status is an emoji and up to 120 characters of text, set on your profile page, with an optional auto-clear after 30 minutes, an hour, four hours, or the end of the day. Expiry is lazy: once the clear time has passed the status stops being returned, and the stored row is tidied the next time you set or clear one. The emoji is drawn as a small badge over your avatar everywhere. Clearing a status does not reset a manual presence choice, the two are independent.

Search

Search runs against an embedded Apache Lucene index on local disk. There is no separate search service to operate, and no Postgres full-text machinery. Indexing happens after the message's transaction commits, so the index never contains a message the database rolled back, and the index rebuilds itself from Postgres on startup if the directory is empty or was written by an older document schema.

One box, in the top bar. While you are reading a channel it is scoped to that channel, so the common case takes no extra clicks; the scope control on the results page widens it. Typing offers a short dropdown for the "jump straight to the thing I remember" case, and Enter opens the full results page.

The default scope covering public channels you have not joined is deliberate: you can already read them, and a search that pretends otherwise just makes shared knowledge unfindable. Hits from a channel you are not in are marked as such. Authorisation is a clause inside the Lucene query rather than a filter applied to its results — post-filtering leaks through hit counts, ranking and paging even when the rows themselves are hidden.

The results page shows a count, the scope you searched, and each hit with its author, channel, time and a highlighted snippet. Long result sets page. "Back to #channel" returns you where you started, and the browser's own Back button works, because it is a normal navigation rather than a panel that swallowed your place.

Query syntax

• Two-character minimum. Shorter queries return nothing rather than everything.
• Words are combined with AND. Every term must appear.
• from:@alice (or from:alice) filters by author: from:alice deploy is "messages alice wrote containing deploy".
• @alice on its own finds messages that mention alice — where she was called by name, not what she wrote. This is the meaning the token has in Slack. It used to mean the author here, which is the opposite, and silently: searching @alice to find where she pinged you returned her own messages instead, with nothing to indicate the mix-up.
• in:#channel narrows to one channel. A channel you cannot read and a channel that does not exist give the same refusal, so the box cannot be used to discover private channel names.
• Plain words are matched fuzzily: one edit for three to five characters, two edits for six to thirty-one, exact below and above that. wrold finds world.
• Quoted "exact phrases", boolean operators and -exclusion work.
• Wildcards, prefixes and regular expressions are refused. deploy* returns nothing, silently. So does a query with an unbalanced quote. That is the cost of not letting a user hand the index a query that scans everything.
• An unrecognised word: prefix is searched as literal text rather than quietly dropped — a typo'd modifier that changes the result set without saying so is worse than one that finds nothing.

Search covers message bodies and attachment filenames, in channels and in conversations. quarterly-report.pdf finds the message that carries it. It does not read inside file contents. A file its uploader has deleted stops matching.

Files and attachments

Attach with the paperclip in the composer. You can select several files at once; each becomes its own message, and any text you typed becomes the caption of the first one. Drag-and-drop and paste-to-upload do not exist, the file dialog is the only route.

Uploads are sent as the raw request body rather than as a multipart form, with the filename and caption riding along in headers. That is a throughput decision: multipart has to scan every byte looking for its boundary, which caps a transfer well below what the network can carry.

Uploads are also refused when the attachments volume drops below a configured free-space floor, 64 MiB by default. That headroom is not for attachments, it is so Postgres and the Lucene segment merger do not run out of disk at the same moment.

The content type is sniffed from the bytes with Apache Tika and the sniffed type wins over whatever the browser claimed. Images render inline and, in channels, open in a full-screen lightbox with download and open-in-new-tab actions; Esc closes it. Everything else renders as a card with the filename, type and size. Downloads are served as attachments by default, and only real raster images may be displayed inline, which is why SVG is excluded.

Finding a file again

Two views. Your files (behind your avatar) is everything you have uploaded, anywhere, with the storage it accounts for — it is where you delete things. Files in this channel (the folder icon in the channel header) is everything anybody shared in that room, newest first, each linking back to the message it arrived on. Anyone who can read the channel can see its files, which is already true of the messages that carry them.

Filenames are also searchable: quarterly-report.pdf in the search box finds the message carrying it, in channels and in conversations. Search does not read inside files. A file its uploader has deleted stops matching, and its tombstone stops listing.

Polls

Polls are created with a slash command and are channel-only.

/poll Where for the offsite? | Lisbon | Tallinn | Somewhere with mountains

You need a question and at least two options, up to a maximum of ten. Options are separated by |; escape a literal pipe as \|. The question can be 500 characters, each option 200.

The poll posts as a message whose body contains the question in plain text, which is deliberate: it means polls are findable by search. The interactive widget renders on top of that. Each option is a full-width button with a proportional fill bar and its vote count, and once you have voted a Remove vote appears.

Voting is single choice. Clicking a different option moves your vote rather than adding one. Votes broadcast to everyone in the channel as they land.

There is no closing, no expiry, no anonymity setting, no list of who voted, and no editing a poll after it is posted. If you get the options wrong, delete the message and post another.

Usage errors are shown only to you, as a red banner above the composer that clears itself after a few seconds. A malformed /poll never becomes a message in the channel.

Slash commands

There are three: /help, /poll and /remind. Anything else beginning with a slash is refused privately — you are told it is not a command here, the text goes back into your composer, and nothing reaches the channel.

That refusal is the point. People arrive from Slack and Mattermost with muscle memory for /leave, /dnd, /away, /invite, /me, /shrug, /topic, /archive and a dozen more that do not exist here — and every one of them used to be broadcast to the room verbatim. Typing /dnd in a busy channel and watching it land as a message is a small, entirely avoidable humiliation.

If you genuinely want a line that starts with a slash, escape it (\/leave), wrap it in backticks, or type one space before it. Every refusal says so.

/help

Lists the commands that exist, privately. Built from the live registry, so it cannot drift from what is actually installed.

/poll

Covered above.

/remind

/remind me in 25m to restart the ingest job
/remind @bob at 14:00 to review the migration
/remind in 2d that the certificate expires

The shape is /remind [me|@username] <when> [to|that] <text>.

• The target defaults to you. Naming @someone sends it to your direct message with them, attributed — "Reminder from @alice: …" — so they know who set it.
• in N<unit> accepts seconds, minutes, hours and days in the usual abbreviations. The maximum is about a year.
• at HH:MM accepts 24-hour or 12-hour time and rolls over to tomorrow if today's time has already passed. It resolves in your timezone — set on the profile page, or taken from your identity provider if it supplies one. It used to resolve in the server's, which quietly meant the wrong hour for anyone not sitting next to the machine.
• Nothing is posted to the channel. The confirmation is a private notice showing the resolved time and the zone it used, and the reminder itself arrives as a direct message — from your conversation with yourself when the target is you. Both used to be ordinary channel messages, which announced "ask about my salary review" to everyone in the room twice. The scheduler checks for due reminders every 30 seconds, so delivery is accurate to roughly that.

There is no way to list or cancel a pending reminder, and no UI showing what you have set.

Your profile and themes

Reach /profile from the avatar menu. Four sections, of which three are editable.

• Profile picture. PNG, JPEG, WEBP or GIF, up to 5 MiB, resized server-side so the longer edge is at most 256 pixels. Changes broadcast, so other people's open tabs swap the image without a reload.
• Status. Emoji, text and auto-clear, described under presence.
• Account. Read only. Username, display name and email come from Keycloak and are changed there.
• Theme. Eight fixed palettes: default, dark, orange, pink, green, purple, red, cyan. Only dark is dark; the other seven are light variants with different accents. Clicking a tile saves it, there is no save button, and the choice is stored on your user row rather than in the browser, so it follows you between machines.

Keyboard shortcuts

The complete list. There is no global command palette and no quick switcher.

No shortcut exists for strikethrough, quote or list; those are toolbar buttons only. There is no up-arrow-to-edit-your-last-message.

The admin console

/admin, reachable from the avatar menu, and gated on the Keycloak realm role ichat-admin. It has four sections:

• Branding. The workspace title, and a logo (PNG, JPEG or WEBP, up to 256 KB) used in the top bar, on the landing page and as the favicon. SVG is rejected on purpose, an uploaded SVG is a script-execution surface. A reset button appears once a custom logo is set.
• Channels. A read-only table of every channel with type, member count, message count and creation date.
• Privacy. One checkbox controlling whether the user table below shows full email addresses or masks them to al…@example.com. Default is to show them. The database keeps the real value either way, so turning it off and on again loses nothing.
• Users. A read-only table: username, display name, email, last active, joined.

Admins also see more in the About dialog: Java and JVM version, OS and architecture, CPU count, maximum heap, uptime, timezone, and the exact version of every bundled library. That is deliberately admin-only, it is a fingerprint of your deployment.

One consequence is visible to users: a suspended account (a users.suspended_at that is set) gets a plain Account suspended page, and API calls get a JSON error, from a filter that runs before anything else.

How settings are layered

Later entries win over earlier ones:

1. application.yml, the defaults compiled into the jar.
2. The active Spring profile's application-<profile>.properties.
3. Environment variables and -D system properties.
4. Vault or OpenBao, if enabled, which is layered on top of everything for the five keys it owns.

Two settings are non-negotiable and have no knob. spring.jpa.hibernate.ddl-auto is validate, so the application refuses to start against a schema that does not match the entities; schema changes go through Flyway migrations. And spring.jpa.open-in-view is false, which is why lazy associations have to be touched inside a service transaction.

Running it as a service

One jar, one systemd unit, one environment file. The unit lives in the manual-install guide, annotated directive by directive, and scripts/install-almalinux.sh writes exactly that text — so there is one copy of it and it cannot drift. An earlier revision of the install guide carried a second, weaker unit that disagreed with the first on paths and on where the environment file lived, and following both in sequence produced an install that started and then failed on its first write. Copy the unit from there rather than from anywhere else.

What it gives you, and what you have to keep in agreement with it:

sudo systemctl daemon-reload
sudo systemctl enable --now intellistream-chat
systemctl status intellistream-chat
journalctl -u intellistream-chat -f     # Flyway migrations, then Tomcat on :8080

Relocating from /opt/intellistream-chat moves four things together: WorkingDirectory, ReadWritePaths, wherever the jar is installed, and the SELinux file context for the data directory (selinux-harden.sh --data-dir).

HTTP and the proxy

server.forward-headers-strategy is set to framework, which is what makes the rest of the proxy story work. A request arriving with X-Forwarded-Proto: https is treated as secure, so the session and CSRF cookies get their Secure flag automatically and no separate setting is needed. If your proxy strips that header you must force it with server.servlet.session.cookie.secure=true.

Database

Flyway is enabled and reads classpath:db/migration. Migrations run before Hibernate validates the schema, and a mismatch is a startup failure rather than a silent difference.

Identity and Keycloak

A dedicated startup check refuses to boot a confidential client with an empty secret. That matters because an empty string is a perfectly valid property value, so without the check the application would start and then fail at first login with something unhelpful.

Storage and quotas

There is no per-upload cap to configure. If you want one for a particular account, set the Keycloak user attribute chat_max_upload_bytes — mapped into the token by the bundled realm — to a byte count. Workspace admins bypass it, and bypass the storage quota too.

Back up ./data/ and the Postgres database and you have the whole product: attachments, avatars, branding and the search index, plus every message.

Search

Calls and TURN

Calls are off until you configure a TURN server, and the buttons are not rendered until you do. That is deliberate: relaying is on by default, so without a relay there is no path for the media and a call button would be a control that cannot work.

The application is never in the media path. It relays the connection setup over the WebSocket it already has, and the audio and video go through coturn, which forwards encrypted UDP it has no way to read. The shared secret below is how the app mints short-lived credentials for it — there is no user database on the TURN side and no accounts to provision.

Running coturn

The bundled compose file starts one with the rest of the stack, bound to loopback — which is all two browsers on the same machine need, and the only safe default for a container that comes up on its own with a shared secret published in the repository. Point it at a real address only together with a real secret:

podman compose up -d                       # loopback, for trying calls on this machine

ICHAT_TURN_RELAY_IP=<public or LAN address> \
ICHAT_TURN_SECRET=<something unguessable> \
  podman compose up -d                     # reachable from elsewhere

For a real deployment, run it on the host instead and configure two URLs. The UDP entry on 3478 is the fast path. The turns: entry on a TLS port is the one that gets through a corporate firewall, because on the wire it is indistinguishable from HTTPS — put it on 443 if you can give coturn its own address or an SNI route from your proxy. A deployment with only the UDP entry works in the office it was tested in and fails at a customer site.

The secret is a real secret. ichat.calls.turn-secret must equal coturn's static-auth-secret, and a TURN server with a guessable one is an open relay — anyone who finds it can push traffic through your server, and the first sign is a bandwidth bill. There is no default anywhere in the stack for exactly this reason.

Sizing

This is the one feature that does not fit the small box the rest of the application does. A relayed voice call costs the relay about 128 kbit/s and a video call about 4 Mbit/s, counting both directions. Bandwidth binds long after coturn's packet-rate CPU does. A gigabit line carries more concurrent 1:1 video calls than a workspace of a few thousand people will ever place at once.

Single instance. Which calls are ringing is held in memory, like the rate limiter, so a second application node would not know about a call started on the first. Calls work on one node; running several needs the shared state that the rest of horizontal scaling does.

The write path

Messages are inserted in batches. Ids are allocated up front in blocks from the sequence, so a message has its real primary key the moment it is accepted; the row is handed to a queue and flushed a few milliseconds later as part of a multi-row insert. Roughly 14,000 transactions a second become about 55 batches a second, and that single change was the largest throughput lever in the whole system.

The queue is sharded by channel, so one flusher owns a channel and its messages commit and publish in the order they were accepted, while different channels commit in parallel. Nothing is broadcast or indexed until its batch has committed. Bodies containing an @ take the synchronous path instead, because mention rows need the message row to exist for their foreign key.

WebSocket tuning

A diagnostics line at startup logs the executors that were actually resolved. Check it before trusting any throughput number, because a missing executor is invisible in every obvious metric: the box is not busy, the database is idle, and latency merely looks like a slow dependency.

Moderation and retention

An admin removing a message is a soft delete. The row stays, drops out of every read path, and remains recoverable until the retention window passes, on the reasoning that the first ban is sometimes the wrong ban. A scheduled purge then removes it for good.

Background cleanup

Two sweeps run on a timer: one deletes files on disk with no surviving database row, the other reconciles the Lucene index against the messages table.

Caches and presence

Rate limits

Limits are per user, per action, over a sliding window, held in memory in the process. There is exactly one property, ichat.ratelimit.enabled, defaulting to true. The individual numbers are compiled in, not configurable. Turning limits off is done only by the bench profile, where every synthetic connection authenticates as one user and the 30-per-minute message cap would otherwise cap the entire benchmark at 30 messages.

Assets and dev tools

Spring profiles

Three, and they are small on purpose. Each one overrides a handful of keys, nothing more.

dev

Activated automatically by ./gradlew bootRun when SPRING_PROFILES_ACTIVE is unset. Not activated by java -jar and not by the tests. It points the issuer at a local Keycloak, allows the localhost origins, turns on dev tools and serves unbundled assets. The file itself is gitignored and there is a committed .example beside it, because it is where a maintainer keeps machine-specific values such as a LAN address for testing on a phone.

prod

Binds loopback, turns dev tools off, and takes the issuer URI, the allowed origins and the client secret from the environment. Two of those, the issuer and the secret, have no default, so a production start with either missing fails immediately rather than coming up pointed at a development Keycloak.

bench

For the load harness, and labelled in the file itself as never for production. It disables rate limiting, binds all interfaces, raises the Tomcat connection and accept limits far above stock, uses a separate Lucene directory, and exposes the health and metrics actuator endpoints read-only. Every one of those is a reason not to run it anywhere real.

Vault and OpenBao

Optional, off by default, and the application is perfectly happy reading credentials from the environment. Turn it on when you would rather the database password and the OIDC client secret did not sit in a file on the host.

It runs before Spring builds any beans, fetches one KV v2 record, and layers it above everything else, so what Vault says wins over the environment, the profile and application.yml alike. It reads five fields and ignores anything else in the record:

db.username              -> spring.datasource.username
db.password              -> spring.datasource.password
keycloak.client-id       -> OIDC client id
keycloak.client-secret   -> OIDC client secret
keycloak.issuer-uri      -> both the OIDC client and the resource server issuer

The path is a KV v2 path. The default resolves to secret/data/intellistream-chat; a value containing a slash, such as mymount/myapp/secrets, uses the part before the first slash as the mount.

Failure behaviour is deliberately loud. Enabled with a missing URI or token is a startup crash, not a fallback, because a silent fallback is how a production instance ends up running on a development password. A transport error or a malformed record is likewise fatal. The one soft case is a record that fetched cleanly but contained none of the five field names: that logs a warning listing what it expected and carries on with the environment, since it is far more likely to be a typo in the path than an attack.

Locally, a profile-gated OpenBao container and a seeding script are in the repository. That container runs in memory with a single root token, so it is for testing the integration, not for running it.

The Keycloak realm

Change the host or port and you must change the redirect URIs and web origins to match, or Keycloak rejects the login with 400 invalid_redirect_uri.

Roles

The role converter grants ROLE_ADMIN on an exact match for ichat-admin and on nothing else. Keycloak's own admin role, which administers the identity provider, is not treated as a grant here, and that is the entire point: whoever operates your Keycloak is not thereby an administrator of your chat. The realm ships the admin role defined and assigned to nobody, so the negative case has something to test against. Both directions are pinned by a unit test.

Promote a person by assigning ichat-admin to them individually. It should never be in the default role set.

The login theme

The sign-in page belongs to Keycloak, not to this application, so it inherits none of the app's styling. Left alone, a user goes from the product's landing page to stock Keycloak and back, which is the one moment a self-hosted deployment looks least like a product. The repository ships a login theme, keycloak/themes/intellistream, that closes the gap: the same navy chrome as the app shell, the same orbit mark, the same self-hosted Figtree, and light and dark driven by prefers-color-scheme.

Installing it

Themes live in $KEYCLOAK_HOME/themes/<name>/<type>/. For a distribution unzipped under /opt/keycloak that is /opt/keycloak/themes/intellistream/login/. Keycloak reads that directory at runtime, so installing a theme needs no kc.sh build and no rebuild of an optimized image. Only themes packaged as a JAR under providers/ need one.

sudo cp -r /path/to/repo/keycloak/themes/intellistream /opt/keycloak/themes/
sudo chown -R root:keycloak /opt/keycloak/themes/intellistream
sudo find /opt/keycloak/themes/intellistream -type d -exec chmod 750 {} +
sudo find /opt/keycloak/themes/intellistream -type f -exec chmod 640 {} +

The server only ever reads a theme, so it never needs write access. On an SELinux host, files copied into /opt/keycloak inherit the correct label but files moved in from a home directory keep the old one; sudo restorecon -R /opt/keycloak/themes fixes a permission error that otherwise makes no sense.

In the compose stack the same directory is bind-mounted read-only, with the SELinux relabel rootless Podman needs: ./keycloak/themes:/opt/keycloak/themes:ro,Z.

Not U. That flag chowns the source to the container's remapped UID, which a container needs only when it has to write — Keycloak only reads a theme, and world-readable files are the whole requirement. What U costs is that the theme files stop being yours to edit, and the moment anyone hands them back with podman unshare chown, the container loses access. Keycloak then falls back to its built-in theme: the login page still works, it is simply unbranded, and the only evidence is a Failed to find LOGIN theme line in the server log. A theme that silently un-applies itself is worse than one that fails to start.

Selecting it for the realm

Three equivalent routes. Use whichever matches how the realm is managed.

kcadm.sh config credentials --server https://auth.your-domain \
         --realm master --user admin
kcadm.sh update realms/ichat-realm -s loginTheme=intellistream

The wordmark under the mark is not part of the theme. It renders Realm settings → General → HTML Display name, so change your name there rather than in the stylesheet; the bundled realm sets <strong>IntelliStream</strong> Chat.

Caching, which will confuse you exactly once

kc.sh start caches resolved themes and compiled FreeMarker templates in memory, and serves theme resources with a thirty-day Cache-Control. Both defaults are correct for production and both hide your edits.

So on a production server: restart Keycloak after changing a theme, and expect browsers that already loaded the page to keep the old stylesheet regardless. A restart cannot reach a client cache. The only reliable fix is a new URL — rename the file (intellistream.css → intellistream.2.css, updated in theme.properties). Telling users to hard-reload is not a deployment strategy.

If you fork it

The theme overrides exactly one FreeMarker template: footer.ftl, which Keycloak ships as an empty macro specifically so themes can fill it in. Everything else is theme.properties, one stylesheet, two SVGs and the font. That is a deliberate constraint, not laziness. A theme that copies login.ftl or template.ftl keeps rendering your copy after an upgrade rewrites the original, and the failure is silent: a new required action, credential type or security fix to the login form simply never appears, and nothing in the logs mentions it. Before copying a template, check whether a CSS rule or one of the kc*Class properties gets you there.

Postgres is not the bottleneck

Take that seriously before you spend an evening in postgresql.conf. If your instance is slow, tuning the database is unlikely to be the fix, and the checks worth running first are elsewhere: the STOMP inbound executor, the broker's subscription cache limit, and whether the batch writer is enabled. All three of those have produced order-of-magnitude differences. None of the Postgres settings below will.

What the settings below are good for is keeping the database out of trouble as the message table grows past the point where the defaults were sensible: not falling into a checkpoint storm, not letting autovacuum drift a week behind, not planning a query on statistics from last month. That is a different goal from unlocking throughput, and it is worth doing.

What the app actually asks of the database

Four properties of this workload determine which knobs matter.

1. Writes arrive batched. The application turns roughly 14,000 individual inserts a second into about 55 multi-row transactions a second. Per-transaction costs, which is most of what commit-path tuning addresses, are therefore divided by 256 before Postgres ever sees them.
2. Reads are small, indexed and keyset-paginated. The channel view is fifty rows ordered by (channel_id, created_at) straight off an index built for exactly that. Nothing sorts a large result set, nothing hashes a big join, and full-text search does not touch Postgres at all, it goes to Lucene.
3. The working set is recent. People read today's messages and occasionally jump to an old permalink. A cache sized to hold the last few weeks behaves almost as well as one sized to hold everything.
4. The messages table is where all the volume is. A new top-level, live, unpinned message writes four index entries: the primary key, (channel_id, created_at), the live-rows partial index on the same columns, and (author_id, created_at). The partial indexes on parent_id, pinned_at and deleted_at cost nothing on insert because the row does not qualify for them.

That last point is the one worth carrying away: the cheapest performance decision available to you is not adding an index to messages. Every index you add is paid on every insert forever, and index maintenance is a far larger share of this workload's database cost than anything in the memory settings.

Memory settings

shared_buffers

Postgres's own page cache, and the only one of these that reserves memory at startup. The default of 128 MB is a compatibility choice, not a recommendation, and it is the one setting almost everybody should change. Start at 25% of the memory available to Postgres. The operating system's page cache covers the rest, and it does so well enough that pushing past roughly 40% usually buys nothing and can cost you double-buffering.

effective_cache_size

A planner hint, not an allocation. It tells the optimiser roughly how much memory the whole machine is likely to have available for caching pages, which is what decides whether an index scan looks cheap enough to prefer over a sequential scan. Set it to about 70% of RAM. It costs nothing to get generous and it is the single cheapest correctness fix for a planner that has started choosing sequential scans over ix_messages_channel_created.

work_mem

Memory for one sort or hash node, not one query and not one connection. A single query with several such nodes can use a multiple of it, and every concurrent query has its own budget, so the worst case is roughly work_mem times nodes times active queries.

For this workload the default 4 MB is very nearly right, because almost nothing sorts. Raising it to 8 or 16 MB is harmless insurance for the admin console's aggregate counts and for migrations. Raising it to 256 MB "for performance" is how a server that was fine at ten concurrent queries falls over at forty. Change it only after log_temp_files shows you spilling.

maintenance_work_mem

Used by VACUUM, CREATE INDEX and ALTER TABLE. This one is worth raising well above its 64 MB default, to somewhere between 256 MB and 1 GB, because it directly sets how much of a table autovacuum can clean in one pass and how fast a Flyway migration that adds an index to a large messages table completes. Remember that autovacuum workers each get their own allowance, so multiply by autovacuum_max_workers for the worst case.

Connections and the pool

Postgres defaults to max_connections = 100. The application ships with no pool configuration at all, so it runs on HikariCP's Spring Boot default of 10. The benchmark harness raises that to 50, which is where the number in scalability.md comes from.

Neither number was ever the limit. At 13,000 messages a second, acquiring a connection took 0.03 ms and nothing ever waited. Batching is why: 55 transactions a second do not need many connections, they need a few connections held briefly.

Leave max_connections at 100. If you must raise the pool, raise it to a size that keeps the STOMP inbound executor from queueing, and remember the relationship runs the other way too: inbound threads beyond the pool size simply wait inside Hikari instead of waiting in the executor.

Checkpoints and WAL

This is the section that earns its place. Checkpoint settings are the one part of postgresql.conf where the defaults are genuinely wrong for a sustained-insert table, and the symptom is periodic latency spikes rather than a uniformly slow server.

After each checkpoint, the first write to any page also writes a full copy of that page into the WAL. Frequent checkpoints therefore mean more pages are "first written" more often, and WAL volume climbs out of proportion to the data. With max_wal_size at its 1 GB default, a busy message table triggers checkpoints on volume rather than on time, repeatedly, and you pay that tax continuously.

The check that tells you whether this mattered is in what to monitor: a checkpoint count dominated by requested rather than timed means max_wal_size is still too small.

synchronous_commit, and what it is worth

Seven percent is not nothing, and if you want it, take it knowingly. What you are turning off is the guarantee that a transaction reported as committed is on disk. A crash or a power loss can lose the last fraction of a second of transactions that Postgres already told the application were durable, up to roughly three times wal_writer_delay, so a few hundred milliseconds by default. This is not the same as fsync = off: the database still recovers to a consistent state, it simply recovers to a slightly earlier one.

For this application specifically, that has a consequence worth spelling out. The whole write design rests on broadcasting a message only after its row has committed, so that nobody is shown a line that then failed to persist. With synchronous_commit = off, "committed" weakens to "committed unless the machine dies in the next half second", and the property becomes a very strong likelihood rather than a guarantee. Messages that were delivered to everybody in the room could be missing after a hard crash.

Whether that is acceptable is a judgement about your workspace, not a technical question. It is a defensible trade for an internal team chat and an indefensible one for a system of record. What is not defensible is taking it by accident, which is why it is off by default and why the 7% is quoted here rather than left as folklore.

If you want most of the benefit without the global change, note that synchronous_commit can be set per transaction, per role or per database.

Autovacuum on the messages table

Autovacuum's defaults are scale factors, which means the bigger a table gets the longer it waits. That is fine for a table of thousands and wrong for one of hundreds of millions. On messages the defaults mean waiting for 20% growth before an insert-triggered vacuum and 10% before an analyze; on a 200-million-row table that is 40 million inserts of drift.

Three separate jobs are at stake, and only one of them is about bloat:

• Statistics. The planner's row estimates for created_at ranges are what keep the channel query on its index. Stale statistics on a fast-growing table are the most likely cause of a query plan that was fine last month and is not now. This is the job that matters most here.
• The visibility map. Vacuum marks all-visible pages, which is what allows index-only scans. An insert-only table never gets vacuumed by the dead-tuple rules at all, which is why modern Postgres added the insert-based trigger.
• Dead tuples. Genuinely present here, because moderation soft-deletes rows and the retention purge then hard-deletes them, up to 100,000 rows an hour at the shipped batch settings. That is real vacuum work, arriving in bursts.

Set per-table overrides rather than changing the global defaults for everything:

ALTER TABLE messages SET (
  autovacuum_analyze_scale_factor       = 0.01,
  autovacuum_analyze_threshold          = 5000,
  autovacuum_vacuum_scale_factor        = 0.02,
  autovacuum_vacuum_threshold           = 5000,
  autovacuum_vacuum_insert_scale_factor = 0.01
);

Globally, the one change worth making on solid-state storage is autovacuum_vacuum_cost_limit = 2000 (from an effective default of 200). Autovacuum is throttled by an I/O cost budget designed for spinning disks; on NVMe that throttle is the only reason it falls behind. Consider autovacuum_max_workers = 4 if you run other large tables beside this one, remembering that each worker draws its own maintenance_work_mem.

Starting values by RAM

Percentages are of the memory available to Postgres. If the JVM is on the same machine, subtract its maximum heap plus about a third again for off-heap overhead before applying them. The 2 GB column assumes exactly that co-location, which is why it is not simply a quarter of the machine.

Everything above except shared_buffers, max_connections and track_io_timing can be changed with ALTER SYSTEM plus SELECT pg_reload_conf(). Those three want a restart.

Running on ZFS

ZFS changes several of the answers above, because it is copy on write. A block is never overwritten in place: the new version is written elsewhere and the pointer is switched when the transaction group commits. Nothing is ever half old and half new. That single property is what makes the settings below safe, and it is worth understanding rather than copying, because every one of them becomes wrong the day you move the data to ext4 or XFS.

Dataset layout

Give Postgres its own dataset, and give the WAL a second one. They want opposite things: the heap is read and written in 8 KB pages at random offsets, the WAL is appended sequentially in large runs. One dataset cannot be tuned for both.

zfs create -o recordsize=16K  -o compression=zstd-1 -o atime=off -o xattr=sa  tank/pgdata
zfs create -o recordsize=128K -o compression=zstd-1 -o atime=off -o xattr=sa  tank/pgwal

Postgres settings that change on ZFS

Two knobs people copy from old advice

logbias=throughput on the data dataset is genuinely situational: it stops synchronous writes going through the ZIL twice, which helps when synchronous_commit is on and there is no separate log device, and hurts latency when there is one. Measure it, do not inherit it.

primarycache=metadata is not situational — do not set it. It tells ZFS to cache metadata and throw every data block away, on the theory that shared_buffers is already caching those and ARC is duplicating the work. That was never a good trade and the compression point above is why: the ARC is the most memory-efficient cache in the system precisely because it keeps blocks compressed, so this turns off the cache that holds the most per byte and leaves the one that holds the least. Every miss then becomes a disk read plus a decompress. The symptom is a server that looks correctly tuned, has plenty of free memory, and is inexplicably I/O bound.

The application's own data

Attachments and the search index deserve the same treatment, and one of them has a sharp edge. A dataset quota on the attachments directory is the right backstop, it is what turns "the host filled up" into "one upload failed", and the app handles that cleanly, refusing the write and returning 507 rather than leaving a partial file behind.

Give the Lucene index its own dataset, not a subdirectory of the attachments one. If a write to the index fails because the attachments quota is exhausted, Lucene's IndexWriter marks itself failed and closes, and every subsequent index update throws until the process restarts. Search then silently stops reflecting new messages while the rest of the application carries on working normally. Separate datasets mean a full attachment volume cannot take search down with it. Default recordsize is fine for both.

A ZFS snapshot of the Postgres dataset is crash consistent, which is to say Postgres will replay the WAL and come up cleanly from one, the same as it would after a power cut.

Splitting the data across datasets costs nothing in consistency, which is the part that makes this layout work. A recursive snapshot is atomic across every dataset in the pool: one transaction group, one instant, no window in which the database has moved on and the attachments have not. So put them under a common parent and take the whole set at once — database, WAL, attachments and index, mutually consistent.

zfs snapshot -r tank/intellistream-chat@$(date +%Y%m%d-%H%M)

What a snapshot is not is a backup. It shares the pool it is protecting you against, so it survives a bad migration and not a dead pool; keep sending it somewhere else with zfs send, or keep taking pg_dump off the machine. The index does not need backing up at all — it rebuilds from the messages table on startup when the directory is empty, so the reason to include it is restore speed on a large archive, not durability.

What not to bother with

What to monitor

Four queries and two application metrics will tell you almost everything.

Cache hit ratio

SELECT blks_hit, blks_read,
       round(100.0 * blks_hit / nullif(blks_hit + blks_read, 0), 2) AS hit_pct
FROM pg_stat_database
WHERE datname = 'intellistream_chat';

Below about 99% on a chat workload means shared_buffers is too small for the recent working set. Above it, stop looking here.

Checkpoints: timed against requested

SELECT num_timed, num_requested, write_time, sync_time
FROM pg_stat_checkpointer;   -- pg_stat_bgwriter before Postgres 17

num_requested counts checkpoints forced by hitting max_wal_size. If it is a meaningful fraction of num_timed, raise max_wal_size. This is the single most useful number in this section.

The messages table's vacuum health

SELECT n_live_tup, n_dead_tup, n_mod_since_analyze,
       last_autovacuum, last_autoanalyze
FROM pg_stat_user_tables
WHERE relname = 'messages';

Watch for last_autoanalyze falling days behind while n_mod_since_analyze climbs, which is the shape of a plan about to go wrong, and for n_dead_tup that never comes back down after a retention purge.

Temp files

temp_files and temp_bytes in pg_stat_database are the only honest reason to raise work_mem. If they stay at zero, leave it alone. Setting log_temp_files = 0 logs every spill with the query attached.

From the application side

• Hikari, through Micrometer: pending connections and acquire time. Anything but zero pending, sustained, means the pool is genuinely short. That is not what was observed at 13,000 messages a second.
• The ichat.write.stage timers, which break each message down by stage: id allocation, Markdown render, broker handoff, cached channel lookup, cached access check. At full rate those sum to about 0.7 ms. If a change makes that grow, it will show up here before it shows up anywhere in Postgres.

Finally, pg_stat_statements is worth enabling on any instance you intend to keep. It needs shared_preload_libraries and therefore a restart, so it is easier to add on day one than on the day you need it.

Back to top