How many reactions per second can Speechwave handle?

A user recently sent us this question:

How many reactions can the audience send at once before the emoji display slows down or drops some of them?

It’s a fair question to ask before you put an app in front of a few hundred people at a conference. We haven’t had a reported issue yet, but I realized I had never actually measured it. So I did, and here are the results.

If you haven’t seen it, Speechwave is a live emoji reaction app for talks. The audience taps emoji on their phones, and a Chrome extension floats them over the speaker’s slides. The speaker gets analytics afterward.

Great expectations

I already knew from developing the code that each audience member is limited to one reaction every 3 seconds. So the most any room can send is the number of people divided by 3, per second. A 300-person room tops out at 100 reactions per second, and only if everyone taps the instant their button comes back, and that’s if you get 100% room participation.

That gives us our theoretical maximum, but tells us nothing about what the server does at 100 reactions per second.

What about SQLite?

I next guessed that SQLite could be a weak spot. Every accepted reaction is saved to the database, and SQLite has a reputation for being a single-writer database. However, it turns out SQLite can write far more than a few hundred rows a second.

Measuring it inside the same container used for the rest of the test, it did about 7,500 inserts per second. It didn’t seem like the database was going to be out limiting factor.

Testing methods

Wanting the test to look like production, I built the production Docker image and ran it locally with the same limits as my Fly machine: one CPU, 512 MB of memory, and a database pool of 2. Then I wrote a small Node script that simulates an audience. Each fake person loads the talk page, joins the LiveView over a websocket, and taps on a timer. A second connection plays the Chrome extension and times how long a reaction takes to arrive.

There were two gotchas.

First, Docker’s --cpus=1 limits how much CPU time a container gets, but the Erlang VM still looked at my laptop and started eight schedulers. On Fly it would see one. When I pinned the container to a single core, the numbers got a worse, but a lot more realistic.

Second, the load generator was the bottleneck. Every reaction is sent to every audience phone, so with 800 fake people one Node process had to read about 200,000 messages per second. It couldn’t, and that backed things up on the server in a way real phones wouldn’t. I split the generator across several processes and made it count messages without parsing them.

Results

I held the audience at a fixed size and raised the total number of reactions per second until something gave.

Of course, the answer depends on how many people are in the room. With 400 people, things slow down at about 100 reactions per second and start dropping soon after. With 800 people, the same thing happens at about 45.

Showing everybody everything

Here’s the part I hadn’t initially thought about. The audience page also shows the floating emoji, so that people can see their reaction appear. That means every reaction is pushed to every connected phone, not only to the speaker’s laptop.

Here’s the handler from the audience LiveView:

def handle_info(
      %Phoenix.Socket.Broadcast{event: "new_reaction", payload: %{emoji: emoji}},
      socket
    ) do
  {:noreply, push_event(socket, "new_reaction", %{emoji: emoji})}
end

Every audience member runs that for every reaction. So the work per second is the number of people times the number of reactions per second, which grows with the square of the room. In my runs one core topped out at about 35,000 to 40,000 of those pushes per second, and both of my break points were right there:

• 400 people × 100 reactions/s = 40,000
• 800 people × 45 reactions/s = 36,000

That gives us a rule of thumb: total reactions per second should stay under about 35,000 divided by the number of people. A room of 1,000 can handle about 35 reactions per second. A room of 200 can handle about 175, though it can’t reach that, because the 3-second limit caps it at about 67.

Real audiences don’t tap constantly. If each person reacts about once every 30 seconds, 800 people send 27 reactions per second, and that ran fine at 55% CPU. Raise it to 1,200 people at the same pace and the core saturates. Obviously, we could handle higher loads if we assume less than 100% participation.

What failure mode looks like

We want things to degrade gracefully. When the core is saturated, the database insert inside someone’s LiveView waits too long for a pool connection and raises. That crashes the process, so the reaction is lost, and the phone has to reconnect. While it’s not ideal, the audience is unlikely to notice a few lost reactions. However…

Memory was the other ceiling. Each connected person costs about 0.23 MB on top of a baseline around 160 MB. A 3,200-person run was killed for running out of memory. That’s not something we’ve seen in production yet, but it wouldn’t be great.

How we can make it even better

The obvious fixes are to batch the pushes to audience phones (one message every quarter second instead of one per reaction), to batch the database writes, and to catch an insert failure so one timeout doesn’t take a page down with it. The first one attacks the primary issue. Of course, if we need to, we can always scale our servers at Fly as well.

Caveats: These were single runs, so the edges are fuzzy: one 800-person run lost 5% of its reactions at a rate where a harder run lost none, so there’s some variance here. Fly’s shared CPU can be throttled harder than a pinned core on my laptop, so the real limits may be lower. Also, I only measured the server. Whether the emoji on the speaker’s screen keep up at these rates is a separate test. For really large talks we may need to throttle/summarize the reaction stream going to the speaker’s broswer extension.

So in summary…

For a typical talk with a few hundred participants, life should be good. For a thousand active particpants (congratulations!), reactions might start to drop above roughly 35 reactions per second across the whole audience, which is what you’d see if everyone tapped about once every 30 seconds and then something exciting happened in the talk. That’s a good problem to have, and now we know where the edge is. Even better, we know how to push the edge out futher if we need to.