What batching did for Speechwave's reaction limit

What batching did for Speechwave’s reaction limit
In a previous post I measured how many emoji reactions Speechwave can handle at once. The short version: every reaction is pushed to every audience phone, so the work grows with the square of the room, and one core topped out at about 35,000 of those pushes per second. Past that, a database insert would time out inside someone’s page, the page would crash, and their phone would have to reconnect.
I ended that post with a list of fix ideas. I’ve now made two of them, and this post covers what I changed, what the numbers did, and a few things that got in my way.
What I did and didn’t do
The list was to batch the pushes to audience phones, to batch the database writes, and to catch an insert failure so one timeout doesn’t take a page down.
I did the first and third. I skipped batching the db writes. SQLite managed about 7,500 inserts per second in my tests, and nothing I measured pointed at the writes as the main problem. Batching them would add moving parts and change when each reaction gets its timestamp, so I’d rather wait until the numbers call for it.
Batching the pushes
Here is how an audience page used to receive a reaction:
def handle_info(
%Phoenix.Socket.Broadcast{event: "new_reaction", payload: %{emoji: emoji}},
socket
) do
{:noreply, push_event(socket, "new_reaction", %{emoji: emoji})}
endOne message per reaction, per page. Now a single GenServer collects the taps, and a timer fires every 250 milliseconds. Each tick sends one message per talk, holding everything collected since the last tick:
defp send_batches(%{pending: pending, max_batch: max_batch} = state) do
for {slug, newest_first} <- pending do
emojis = newest_first |> Enum.reverse() |> cap(max_batch)
SpeechwaveWeb.Endpoint.broadcast("audience:#{slug}", "reactions_batch", %{emojis: emojis})
end
%{state | pending: %{}}
endA page now receives about four messages a second, whether the room is quiet or going wild. If more than 50 emoji pile up in one tick, I send a random sample of 50, since nobody can read more than that floating up a phone anyway. The JavaScript on the page spreads each batch across the next 250 milliseconds with a random animation delay, so it looks like a stream and not a burst.
We couldn’t just batch in place and have the pages ignore the old per-reaction message. That doesn’t work, because Phoenix delivers a broadcast to every process subscribed to the topic, and the delivery itself is the cost. A page that ignores the message has still paid to receive it. So audience pages now subscribe to their own topic, and the Chrome extension keeps getting one message per tap on the old one. As a bonus, the extension needed no update.
The theoretical price is that the person tapping sees their own emoji up to 250 milliseconds later than before. I think that’s probably okay.
Catching the insert failure
When the database can’t take a write in time, Ecto doesn’t return an error. It raises, and an unhandled exception in a LiveView takes the page down. So recording a reaction now looks like this:
def record_reaction(talk_id, emoji, slide_number, client_id) do
# ... find the active session and insert the reaction ...
rescue
error in [DBConnection.ConnectionError, Exqlite.Error] ->
if busy_error?(error) do
Logger.warning("reaction not recorded, database busy: #{first_line(error)}")
:telemetry.execute([:speechwave, :reaction, :insert_failed], %{count: 1}, %{})
{:error, :db_busy}
else
reraise error, __STACKTRACE__
end
endThe reaction is lost from the speaker’s analytics in that case, which is why it gets logged and counted. But the audience still sees the emoji, and nobody’s page crashes.
The first version only rescued the pool timeout, because that was the failure
I’d seen. Then I ran the production image and checked both ways the database
can say no. Holding every pooled connection did what I expected. But SQLite
only allows one writer at a time, and when I held its write lock with a
connection free, the insert raised Exqlite.Error "Database busy" after about
two seconds, and the rescue missed it. Now it catches both. Anything else still
raises, because anything else is a real bug.
Results
I reran the old and new builds on the same day, on the same machine, with the same limits as before: one pinned core, 512 MB, a database pool of 2.
| People | Reactions/s | Build | CPU | Delivered | Delay (median / p95) | Page crashes |
|---|---|---|---|---|---|---|
| 400 | 100 | before | 99% | 100% | 38 / 205 ms | 0 |
| 400 | 100 | after | 27% | 100% | 3 / 25 ms | 0 |
| 400 | 125 | before | 86% | 77% | 29 / 135 ms | 96 |
| 400 | 125 | after | 32% | 100% | 3 / 23 ms | 0 |
| 800 | 45 | before | 87% | 100% | 37 / 173 ms | 0 |
| 800 | 45 | after | 28% | 100% | 5 / 40 ms | 0 |
| 800 | 60 | before | 92% | 81% | 709 / 1,312 ms | 171 |
| 800 | 60 | after | 30% | 100% | 3 / 19 ms | 0 |
| 1,200 | 40 | before | 100% | 83% | 921 ms (2 probes) | 213 |
| 1,200 | 40 | after | 30% | 100% | 4 / 40 ms | 0 |
At the same load, CPU fell to about a third, every reaction arrived, and the crashes went away at every step that used to produce them. The 1,200-person row is everyone tapping once every 30 seconds, which is closer to a real audience.
A few caveats. These are single 60-second runs, and each has only 11 timing probes, so treat the delays as rough. “Delay” is measured at the extension-style listener, which still gets one message per tap. It says nothing about audience pages, which now wait up to 250 milliseconds longer by design. And the old numbers here differ a little from the table in the last post, because I remeasured them instead of copying them.
Limitations
I didn’t find the new limit. I only reran the old steps, and the new build didn’t break at any of them. The old rule of thumb (keep total reactions per second under 35,000 divided by the number of people) no longer describes this build, because each page’s message rate is now fixed at about four a second.
Memory is probably next. The 1,200-person run used about 417 MB of the 512 MB available, and the last post’s 3,200-person run was killed for running out of memory. If you have a few thousand people in a room tapping at once, that’s where we’d look first. We should also measure the extension side, since the speaker’s laptop still gets every reaction individually.
For talks of a few hundred people, the answer to “how many reactions can the audience send at once?” was already “plenty”. Now it’s “plenty, with room to spare.”