Flash5/flash/docs/http2/WRITER.md

The serialized frame writer

Audience: contributors. This is the design record and benchmark evidence for dev.relism.flash.http2.frame.Http2FrameWriter, the one component every HTTP/2 write in this codebase passes through. It was the central architectural risk: frames, HPACK and flow control are table-driven, but multiplexed streams require concurrent producers to share one socket without interleaving bytes or pinning carrier threads. The measured gate is recorded below.

The problem, precisely

Under HTTP/1.1, one virtual thread owns one connection's socket for the request/response it is currently serving; there is never a second writer. Under HTTP/2, N streams share one connection and their frames must interleave on the wire, so every write must pass through a serialization point that plain HTTP/1.1 never needed. A lock taken naively per frame — synchronized or an uncontended ReentrantLock.lock() — costs more per write than every allocation this codebase has ever saved elsewhere (EX-04 through EX-29), because it sits on the one path every response, of either protocol width, eventually goes through.

The design, three layers

Layer 1 — serialize outside the lock. By the time Http2FrameWriter.write(WriteIntent) is called, the caller (a stream, or a connection-level singleton such as a precompiled SETTINGS ACK) has already built its complete frame — header, HPACK block, payload — into a buffer it owns. The writer never serializes anything; it holds the lock only for the duration of one bulk sink.write(buffer, offset, length) call, never for a sequence of small writes. This is why EX-27 (collapsing HttpServer.writeResponse's ~10 small writes into one) is a prerequisite for HTTP/1.1 too; Http1ResponseWriter now follows the same bulk-write discipline.

Layer 2 — ReentrantLock, never synchronized. On Java 21, a virtual thread that blocks inside a synchronized block pins its carrier platform thread (JEP 491, which removes this, only lands in JDK 24+). Blocking on a ReentrantLock unmounts the virtual thread instead. This is the same fix EX-01 applies to WebSocketSession, generalized to the connection writer where it matters far more (N streams instead of one WebSocket session). ReentrantLock is load-bearing for a second reason synchronized cannot offer: tryLock().

Layer 3 — tryLock() fast path, intrusive MPSC fallback. The overwhelmingly common instant, even on a genuinely multiplexed connection, has exactly one stream wanting to write: a browser calling one API endpoint, a gRPC unary call. tryLock() on an uncontended lock is one successful CAS; the calling thread writes inline and releases — no handoff, no queue touched, no allocation, no context switch. Only when tryLock() fails — genuine contention, genuine multiplexing — does the intent get published through IntrusiveMpscQueue (one more CAS, still zero allocation: the WriteIntent itself is the queue node, via mpscNext()/setMpscNext) for the current lock holder to drain.

happy path (1 active writer):  tryLock → sink.write → unlock              ≈ 1 CAS
contended  (N active writers): tryLock fails → CAS enqueue → return
                               current holder drains the queue before unlocking

Why the fast path checks queue.hasWork(), not just tryLock()

Found by this phase's own stress test at N=64/256 — exactly the class of bug R10 exists to catch before it ships, not after. Writing an intent immediately, ahead of anything already queued, is only safe when nothing is already queued. Without the hasWork() guard:

1. Producer P calls write(a), then write(b). Both contend (someone else holds the lock) and both get queued — fire-and-forget from P's point of view.
2. The current holder is about to drain them but has not yet done so.
3. P's very next call, write(c), finds the lock free (the holder released it between P's calls) and — without the guard — would write c directly, landing it on the wire before a and b, which are still sitting in the queue.

write() therefore checks !queue.hasWork() && lock.tryLock() before taking the direct path: "bypass the queue" only happens when the queue is observed genuinely empty, i.e. everything any producer has ever offered has already been written. hasWork() never false-negatives (it would only ever wrongly report work that isn't there, which just costs an extra harmless tryLock() attempt), so this preserves per-producer ordering without adding a false rejection of the fast path.

Lost-wakeup avoidance

The classic hazard for a design like this: a producer offers its intent to the queue at the exact moment the current lock holder has just found the queue empty and is about to unlock. Without care, the item is stranded — offered, but nobody left to drain it, and the producer already returned believing the write is in flight.

Producer P                              Holder H (currently draining, about to unlock)
───────────                             ──────────────────────────────────────────────
                                         next = queue.poll()   // null: queue looks empty
queue.offer(intent)     ← races here →
if (lock.tryLock())                     lock.unlock()
    drive(null)          // P's own second chance: if P wins the tryLock() race
                          // immediately after H's unlock(), P itself becomes the new
                          // holder and drains — including its own just-offered intent.

Two cooperating mechanisms close this, and both are required — neither alone is sufficient:

1. The producer's own second chance. After a failed tryLock(), write() offers the intent then immediately attempts tryLock() again. If H has already unlocked by this point, P wins the second tryLock() and drains the queue itself (drive(null) — draining whatever is queued, which necessarily includes the intent P just offered, since offer() had already completed).
2. The holder's re-check-after-unlock loop, in drive(): after unlock(), re-read queue.hasWork(). If non-empty, attempt tryLock() again and drain, then unlock and re-check once more — looping, because this recheck cycle can itself race the same way a first pass can. If a second tryLock() in this loop fails, some other thread now holds the lock, and by the same argument that other holder's own re-check-after-unlock covers the item once it releases.

The correctness argument for why together these are sufficient is a happens-before chain through the queue's AtomicReference (IntrusiveMpscQueue.head, a getAndSet per offer) and the lock's own acquire/release ordering: every offer() happens-before some subsequent poll() that observes it (directly, or via the momentary-null self-correcting race documented on IntrusiveMpscQueue itself — see its class Javadoc), and every thread that successfully offers either (a) is itself about to attempt tryLock() and, on success, drains everything including its own offer, or (b) fails that tryLock(), meaning some other thread holds the lock at that instant and that thread's own unlock will trigger its own re-check-after-unlock loop. There is no interleaving in which an offered intent is neither drained by its own producer nor covered by some other thread's re-check loop.

A frame's bytes are never interleaved with another frame's bytes: every write of one intent is a single sink.write call issued while holding the lock, and the lock is not released between a WriteIntent's bytes — proven directly by Http2FrameWriterStressTest, which reassembles producer/sequence/marker-tagged frames from the sink's output and fails loudly on any torn, duplicated, reordered, or lost frame.

Write timeout

A blocking write is unavoidable when the kernel send buffer is full and the peer is not reading — whoever holds the lock is blocked in the syscall, holding up every other stream on the connection. This is bounded by Http2Limits.WRITE_TIMEOUT_MS (30 s), enforced by a single shared daemon thread (Http2FrameWriter.WriteTimeoutReaper) rather than Socket#setSoTimeout, which bounds reads, not writes.

The reaper deliberately does not ask each write to record a System.nanoTime() deadline — an early revision did, and this phase's own N=1 benchmark measured that single nanoTime() call (plus the extra volatile field it required) costing enough to put per-write overhead over the 50 ns-over-baseline gate budget. Instead, the reaper scans every registered writer every SCAN_INTERVAL_MS (50 ms) and counts consecutive scans a writer has been observed still blocked (writingThread non-null); a writer blocked for more than WRITE_TIMEOUT_MS / SCAN_INTERVAL_MS consecutive scans is interrupted. This trades a little precision — up to one scan interval of slop, already inherent to any background-reaper design — for removing all per-write timing cost from the path this document's gate criteria are strictest about.

Benchmark methodology

flash/src/jmh/java/dev/relism/flash/http2/frame/FrameWriterBenchmark.java (a JMH source root registered only under the jmh Maven profile — see DECISIONS.md, DEC-17, for why it does not live in src/test/java) compares four harnesses at threads ∈ {1, 2, 4, 8, 16, 64}:

• trylock_mpsc — the shipped Http2FrameWriter design.
• plain_lock — every write blocks on ReentrantLock.lock() unconditionally (candidate (a)).
• dedicated_thread — every write hands off to one dedicated platform thread via the same IntrusiveMpscQueue, parked/unparked, never busy-polled (candidate (c)).
• raw_unsynchronized — no coordination at all; not a candidate (concurrent writers would tear each other's frames), included only to answer "what does a write cost with zero coordination", which the N=1 gate criterion is defined relative to.

Each JMH "operation" is a full burst: threads virtual producer threads each write 4 000 frames of 512 bytes into a CountingSink that discards the bytes but atomically counts completed writes; runBurst blocks until the count reaches the expected total, so the timed interval always covers real completion, not mere submission (write() can return once an intent is merely queued on the contended path — timing only "how long until every write() call returned" would flatter whichever design most aggressively defers work). @Threads was not usable here: it requires a compile-time constant, not a value swept via @Param, and JMH's own thread pool is platform threads, not the virtual threads under test.

Two honest caveats, stated plainly rather than glossed over (R3):

1. The sink is an in-memory counter, not a real socket. "p999 latency ... on loopback" in the plan's gate wording implies real socket I/O; this harness measures writer-lock-contention latency in isolation from network variance, which is the right isolation for judging this component, but it means the recorded p999 numbers below are a lower bound on what a real loopback socket would show, not a direct stand-in for it. Frame size used is 512 B, not the plan's illustrative 1 KB — chosen to keep the burst's own array allocation small relative to JVM defaults; the writer's cost model does not depend on frame size (it copies nothing; see WriteIntent's Javadoc), so this does not affect the gate conclusions.
2. One JMH "op" is a whole burst (4 000 writes), not one write, because @OperationsPerInvocation requires a compile-time constant and cannot vary with the threads @Param. Every burst also pays fixed harness costs common to all four designs equally: one ExecutorService (a virtual-thread-per-task executor) created and torn down, one Future[] array, one long[threads][4000] latency-sample array, and one fresh BenchIntent object allocated per write (matching the stress test's own pattern, not the writer's actual production contract — a real stream is long-lived and reuses itself as its own WriteIntent). Because this cost is identical across designs, absolute gc.alloc.rate.norm numbers below are dominated by this shared harness cost (~33 443 B/op), not by the design under test; the number that actually answers the "0 B/op" gate criterion is the differential between a design and the raw_unsynchronized baseline, which isolates exactly the bytes that design itself adds.

Results

All runs: JDK 21.0.11 (Temurin), this development sandbox, JMH 1.37, -Fork per run noted below. Raw JMH output is not reproduced in full here; the numbers below are the reported means with their 99.9% CI half-widths.

N=1 — throughput and allocation (-f 4 -wi 5 -w 1s -i 12 -r 2s, throughput; separately

-f 2 -wi 3 -i 8, -prof gc)

design	ops/s (bursts/s)	derived ns/write	gc.alloc.rate.norm (B/op, per burst)
trylock_mpsc	2007.930 ± 60.623	124.5 ns	33 449.253 ± 13.383
raw_unsynchronized	3052.036 ± 52.515	81.9 ns	33 443.349 ± 1.572

• Overhead vs. raw unsynchronized: 124.5 − 81.9 = 42.6 ns (point estimate). Worst case within the 99.9% CI (slowest plausible trylock_mpsc, fastest plausible baseline): ≈ 47.9 ns. Both are under the 50 ns gate budget.
• Allocation delta: 33 449.253 − 33 443.349 = 5.9 B per 4 000-write burst ≈ 0.0015 B per write — within trylock_mpsc's own ±13.383 error band, i.e. not distinguishable from zero. Consistent with the design: the fast path is queue.hasWork() (a volatile read) plus ReentrantLock.tryLock()/unlock() (well-known non-allocating on the JDK's implementation) plus one bulk sink.write. Gate criterion: 0 B/op — PASS.

N=64 — throughput retention and tail latency (-f 2 -wi 3 -w 1s -i 5 -r 1s)

design	N=1 writes/s (per-thread)	N=64 writes/s (aggregate)	retention	p999 @ N=64
trylock_mpsc (shipped)	8 721 148	5 712 640	65.5 %	11.8–14.2 µs
plain_lock (candidate a)	10 469 956	6 082 048	58.1 %	1627–1952 µs
dedicated_thread (candidate c)	2 673 964	5 718 528	213.8 %†	1.5–6.7 µs
raw_unsynchronized (unsafe baseline)	11 353 924	20 764 160	n/a	n/a

† dedicated_thread's N=1 baseline is itself poor (every uncontended write still pays a full park/unpark handoff to the dedicated thread — there is no fast path for the "only one writer" case at all), so a >100% "retention" number reflects a bad denominator, not superlinear scaling. It is reported for completeness, not as a pass/fail signal — the gate criterion is defined relative to trylock_mpsc's own N=1 baseline, which is the design that shipped.

• trylock_mpsc throughput retention: 65.5 % ≥ the required 60 %. PASS.
• trylock_mpsc p999 latency: 11.8–14.2 µs, far under the 1 ms budget. PASS.
• (Not gate-relevant, but part of why (b) was chosen over (a) and (c), per the plan's task 6: plain_lock blows past the 1 ms p999 budget by ~1000× under load — unfair blocking causes tail pile-up exactly as expected from a design with no fast path and no fairness guarantee. dedicated_thread has the best tail latency of the three but a ~3.3× throughput penalty at N=1, because every write, even genuinely uncontended ones, pays a full thread handoff. Neither alternative is a better shipped default than trylock_mpsc.)

Stress test — correctness under concurrency, 1000 iterations per N

Run via an ad hoc reflective driver invoking Http2FrameWriterStressTest's private runStress method directly (the shipped test class runs reduced counts for a fast default mvn test; this is the full gate verification described in that class's own Javadoc), for N ∈ {1, 2, 8, 64, 256}, 1000 iterations each:

Scheduler	n=1	n=2	n=8	n=64	n=256	Total wall time
default parallelism	0 failures	0 failures	0 failures	0 failures	0 failures	≈ 9.1 s
-Djdk.virtualThreadScheduler.parallelism=1	0 failures	0 failures	0 failures	0 failures	0 failures	≈ 8.9 s

Every byte of every frame arrived, correctly ordered per-producer, with no tearing, duplication, or loss, in both configurations — 10 000 total stress runs, 0 failures.

Carrier pinning: the parallelism=1 run above was additionally run under -Djdk.tracePinnedThreads=full, which prints a stack trace to stderr for any virtual thread found blocked while pinning its carrier. Zero output — no pinning observed, consistent with the design's exclusive use of ReentrantLock (never synchronized) on every path that can block.

Gate criteria — final tally

#	Criterion	Result	Verdict
1	N=1: 0 B/op	0.0015 B/write differential vs. baseline, within noise	PASS
1	N=1: ≤50 ns overhead vs. raw unsynchronized	42.6 ns point estimate, ≤47.9 ns worst-case CI	PASS
2	N=64: throughput ≥60% of N=1 per-thread rate	65.5 %	PASS
2	N=64: p999 <1 ms (512 B frame, in-memory sink)	11.8–14.2 µs	PASS
3	No carrier pinning under -Djdk.tracePinnedThreads=full	none observed	PASS
4	Stress test green at every N, 1000 iterations, incl. parallelism=1	10 000/10 000	PASS

All four gate criteria are met. Http2FrameWriter ships as designed: a tryLock() fast path with an intrusive MPSC fallback. See DECISIONS.md for the retained alternatives and evidence.

What this design costs vs. what it saves

The honest framing (per R3, extended from HPACK's own to the writer): the writer's happy path costs one uncontended CAS (ReentrantLock.tryLock()) plus a volatile read (queue.hasWork()) plus the write syscall itself — on the order of tens of nanoseconds, measured above at ~42.6 ns over a raw unsynchronized write. What it buys is the only thing that makes HTTP/2 multiplexing possible on a codebase built around "one thread owns the socket": N concurrent streams can write frames to the same connection without a naive per-frame lock (which the plain_lock comparison above shows costs ~1000× more in tail latency once real contention appears), and without committing every connection to a dedicated writer thread's per-write handoff cost (which the dedicated_thread comparison shows costs ~3.3× throughput at the N=1 case that dominates real traffic). Forty-two nanoseconds is a price worth paying once, on the one path that gates multiplexed HTTP/2 correctness at all.