Why every message into Synapse waits in a bounded queue first

The problem with one handler per channel

Four channel adapters (WhatsApp, Telegram, Discord, Slack) all feed the same agent core. The naive version of that is a synchronous handler per channel: a message comes in, the handler runs the agent loop, and only then does it look at the next message. That serializes every channel behind whichever one is currently mid-request, which is already a problem, but the sharper failure shows up under load. An unbounded queue in front of a synchronous handler turns a traffic burst, or a webhook a platform retries because it didn't see a fast-enough acknowledgment, into unbounded memory growth on one side and duplicate processing on the other.

Three stages before the agent core sees anything

I built one async ingestion layer, shared across all four adapters, and every message crosses three stages before the agent core ever sees it. The numbers below are configuration constants read out of that layer's source, not a benchmark result. FloodGate holds incoming messages in a three-second batch window, so a burst of messages arriving in quick succession lands as one unit of work instead of many separate ones. Dedup checks each message against a five-minute TTL cache, so a platform-level retry of a webhook that already succeeded doesn't get processed twice. What comes out the other side lands in a bounded asyncio FIFO queue, capped at 100 messages, drained by two workers:

queue: asyncio.Queue[Message] = asyncio.Queue(maxsize=100)

async def worker() -> None:
    while True:
        message = await queue.get()
        try:
            await handle(message)
        finally:
            queue.task_done()

workers = [asyncio.create_task(worker()) for _ in range(2)]

That's the shape of it, not a literal excerpt. The cap is the actual point: a producer that hits maxsize blocks instead of growing the process's memory without limit, which turns "the queue is falling behind" into backpressure the caller can see and handle, instead of a slow memory leak that only shows up as an out-of-memory kill hours later.

What the test actually proves

A load test fans out 500 concurrent sends through the ingestion queue in one run and asserts, specifically: zero drops from deduplication misfiring, zero drops from the queue being full, no missing message ids on the way out, no duplicated ids, and the whole batch drained inside 60 seconds. That's a How I measured concurrent burst, exactly-once: 500 messagesMethodtest_no_dropped_under_burst_500 fans out 500 concurrent sends through the ingestion queue and asserts exactly-once processing: zero duplicate drops, zero full-queue drops, no missing ids, no duplicated ids, queue drained inside 60s. Stated as what the test asserts: no CI run log is committed, so this is a verified test, not a published run resultDate2026-08-11Sample sizen = 500Sourcegithub.com/UpayanGhosh/Synapse-OSS/blob/develop/workspace/tests/load/test_pipeline_burst.py test, and I'm stating it exactly that way on purpose, as what the test asserts, not as a published result. No CI run log for this test is committed anywhere in the repository, so I don't have a dated, citable "it ran on this date and passed" artifact the way some of the other figures on this site do. The test exists, it's real, and it's strict about what counts as success. What I don't have yet is a dated log proving the last time it ran green, and I'd rather say that plainly than round it up to a number that sounds more finished than the evidence actually is.

What's still manual

The one thing I'd change, from the decision record above: queue depth is currently something I check by hand during a load test, not something the observability layer surfaces as its own metric. Backpressure being visible at the queue boundary was the win; backpressure being visible without me going and looking for it is the part that's still outstanding.

Reproducing this

The burst test lives at workspace/tests/load/test_pipeline_burst.py on the Synapse-OSS repository's develop branch, the same branch the test-suite count elsewhere on this site is measured from. FloodGate, Dedup, and the bounded queue's cap and worker count live in the ingestion module that same test imports, in the same repository: configured in source, not written up in a separate doc.

The full case study, with the rest of Synapse's numbers, is at /work/synapse.