Contents

What batching did for Speechwave's reaction limit

What batching did for Speechwave’s reaction limit

In a previous post I measured how many emoji reactions Speechwave can handle at once. The short version: every reaction is pushed to every audience phone, so the work grows with the square of the room, and one core topped out at about 35,000 of those pushes per second. Past that, a database insert would time out inside someone’s page, the page would crash, and their phone would have to reconnect.

I ended that post with a list of fix ideas. I’ve now made two of them, and this post covers what I changed, what the numbers did, and a few things that got in my way.

What I did and didn’t do

The list was to batch the pushes to audience phones, to batch the database writes, and to catch an insert failure so one timeout doesn’t take a page down.

I did the first and third. I skipped batching the db writes. SQLite managed about 7,500 inserts per second in my tests, and nothing I measured pointed at the writes as the main problem. Batching them would add moving parts and change when each reaction gets its timestamp, so I’d rather wait until the numbers call for it.

Batching the pushes

Here is how an audience page used to receive a reaction:

def handle_info(
      %Phoenix.Socket.Broadcast{event: "new_reaction", payload: %{emoji: emoji}},
      socket
    ) do
  {:noreply, push_event(socket, "new_reaction", %{emoji: emoji})}
end

One message per reaction, per page. Now a single GenServer collects the taps, and a timer fires every 250 milliseconds. Each tick sends one message per talk, holding everything collected since the last tick:

defp send_batches(%{pending: pending, max_batch: max_batch} = state) do
  for {slug, newest_first} <- pending do
    emojis = newest_first |> Enum.reverse() |> cap(max_batch)
    SpeechwaveWeb.Endpoint.broadcast("audience:#{slug}", "reactions_batch", %{emojis: emojis})
  end

  %{state | pending: %{}}
end

A page now receives about four messages a second, whether the room is quiet or going wild. If more than 50 emoji pile up in one tick, I send a random sample of 50, since nobody can read more than that floating up a phone anyway. The JavaScript on the page spreads each batch across the next 250 milliseconds with a random animation delay, so it looks like a stream and not a burst.

We couldn’t just batch in place and have the pages ignore the old per-reaction message. That doesn’t work, because Phoenix delivers a broadcast to every process subscribed to the topic, and the delivery itself is the cost. A page that ignores the message has still paid to receive it. So audience pages now subscribe to their own topic, and the Chrome extension keeps getting one message per tap on the old one. As a bonus, the extension needed no update.

The theoretical price is that the person tapping sees their own emoji up to 250 milliseconds later than before. I think that’s probably okay.

Catching the insert failure

When the database can’t take a write in time, Ecto doesn’t return an error. It raises, and an unhandled exception in a LiveView takes the page down. So recording a reaction now looks like this:

def record_reaction(talk_id, emoji, slide_number, client_id) do
  # ... find the active session and insert the reaction ...
rescue
  error in [DBConnection.ConnectionError, Exqlite.Error] ->
    if busy_error?(error) do
      Logger.warning("reaction not recorded, database busy: #{first_line(error)}")
      :telemetry.execute([:speechwave, :reaction, :insert_failed], %{count: 1}, %{})
      {:error, :db_busy}
    else
      reraise error, __STACKTRACE__
    end
end

The reaction is lost from the speaker’s analytics in that case, which is why it gets logged and counted. But the audience still sees the emoji, and nobody’s page crashes.

The first version only rescued the pool timeout, because that was the failure I’d seen. Then I ran the production image and checked both ways the database can say no. Holding every pooled connection did what I expected. But SQLite only allows one writer at a time, and when I held its write lock with a connection free, the insert raised Exqlite.Error "Database busy" after about two seconds, and the rescue missed it. Now it catches both. Anything else still raises, because anything else is a real bug.

Results

I reran the old and new builds on the same day, on the same machine, with the same limits as before: one pinned core, 512 MB, a database pool of 2.

PeopleReactions/sBuildCPUDeliveredDelay (median / p95)Page crashes
400100before99%100%38 / 205 ms0
400100after27%100%3 / 25 ms0
400125before86%77%29 / 135 ms96
400125after32%100%3 / 23 ms0
80045before87%100%37 / 173 ms0
80045after28%100%5 / 40 ms0
80060before92%81%709 / 1,312 ms171
80060after30%100%3 / 19 ms0
1,20040before100%83%921 ms (2 probes)213
1,20040after30%100%4 / 40 ms0

At the same load, CPU fell to about a third, every reaction arrived, and the crashes went away at every step that used to produce them. The 1,200-person row is everyone tapping once every 30 seconds, which is closer to a real audience.

A few caveats. These are single 60-second runs, and each has only 11 timing probes, so treat the delays as rough. “Delay” is measured at the extension-style listener, which still gets one message per tap. It says nothing about audience pages, which now wait up to 250 milliseconds longer by design. And the old numbers here differ a little from the table in the last post, because I remeasured them instead of copying them.

Limitations

I didn’t find the new limit. I only reran the old steps, and the new build didn’t break at any of them. The old rule of thumb (keep total reactions per second under 35,000 divided by the number of people) no longer describes this build, because each page’s message rate is now fixed at about four a second.

Memory is probably next. The 1,200-person run used about 417 MB of the 512 MB available, and the last post’s 3,200-person run was killed for running out of memory. If you have a few thousand people in a room tapping at once, that’s where we’d look first. We should also measure the extension side, since the speaker’s laptop still gets every reaction individually.

For talks of a few hundred people, the answer to “how many reactions can the audience send at once?” was already “plenty”. Now it’s “plenty, with room to spare.”