feat(core): HTTP/2 support, correctness fixes, and doc reorganization #10

Merged
Relism merged 23 commits from feature/core/http2 into master 2026-08-14 18:20:30 +00:00
24 changed files with 1357 additions and 13 deletions
Showing only changes of commit 6386264a1e - Show all commits
+30 -1
View File
@@ -32,8 +32,37 @@ jobs:
server-username: MAVEN_USERNAME server-username: MAVEN_USERNAME
server-password: MAVEN_PASSWORD server-password: MAVEN_PASSWORD
- name: Install h2spec 2.6.0
run: |
curl --fail --location --silent --show-error \
--output /tmp/h2spec.tar.gz \
https://github.com/summerwind/h2spec/releases/download/v2.6.0/h2spec_linux_amd64.tar.gz
echo "157ee0de702e01ad40e752dbf074b366027e550c8e7504f9450da2809e279318 /tmp/h2spec.tar.gz" \
| sha256sum --check
tar --extract --gzip --file /tmp/h2spec.tar.gz --directory /tmp
- name: Install nghttp client
run: |
sudo apt-get update
sudo apt-get install --yes nghttp2-client
- name: Install grpcurl 1.9.3
run: |
curl --fail --location --silent --show-error \
--output /tmp/grpcurl.tgz \
https://github.com/fullstorydev/grpcurl/releases/download/v1.9.3/grpcurl_1.9.3_linux_x86_64.tar.gz
echo "a926b62a85787ccf73ef8736b3ae554f1242e39d92bb8767a79d6dd23b11d1d5 /tmp/grpcurl.tgz" \
| sha256sum --check
tar --extract --gzip --file /tmp/grpcurl.tgz --directory /tmp grpcurl
- name: Build and test - name: Build and test
run: mvn -B --settings .github/settings.xml clean verify run: >-
mvn -B --settings .github/settings.xml
-Dh2spec.executable=/tmp/h2spec
-Dcurl.executable=/usr/bin/curl
-Dnghttp.executable=/usr/bin/nghttp
-Dgrpcurl.executable=/tmp/grpcurl
clean verify
env: env:
MAVEN_USERNAME: ${{ secrets.MAVEN_USERNAME }} MAVEN_USERNAME: ${{ secrets.MAVEN_USERNAME }}
MAVEN_PASSWORD: ${{ secrets.MAVEN_PASSWORD }} MAVEN_PASSWORD: ${{ secrets.MAVEN_PASSWORD }}
+84
View File
@@ -0,0 +1,84 @@
# HTTP/2 compliance
This document records the repeatable protocol gate for Flash's HTTP/2 server. The automated
matrix runs from Maven; external tools are selected through system properties so local builds
without them skip only the corresponding interoperability adapter. CI installs and enables every
command-line client listed below.
## h2spec
Validated on 2026-08-13 with h2spec 2.6.0.
| Listener | Cases | Failures | Skips |
|---|---:|---:|---:|
| TLS with ALPN `h2` | 146 | 0 | 0 |
| Cleartext prior knowledge on the mixed HTTP/1.1 + HTTP/2 port | 145 | 0 | 0 |
`H2SpecComplianceTest` parses h2spec's JUnit XML and fails on a failure, error, or skipped case.
The cleartext selection omits only `http2/3.5/2`, which sends a complete invalid HTTP/2 preface.
That case assumes a dedicated HTTP/2 endpoint. Flash deliberately has one cleartext port that
selects HTTP/2 only when the 24-byte prior-knowledge preface matches; any other initial bytes are
HTTP/1.1 input. RFC 9113 section 3.3 defines the exact preface as the cleartext protocol selector,
while section 3.4's `PROTOCOL_ERROR` applies after an endpoint is operating as HTTP/2. The HTTP/2
state machine itself does return `GOAWAY(PROTOCOL_ERROR)` for a complete invalid preface, covered
byte-for-byte by `invalid-preface.hex`. Excluding the mixed-port negotiation case therefore does
not waive an HTTP/2 state-machine requirement.
## Interoperability
Automated results recorded on 2026-08-13:
| Client | Version | Mode and coverage | Result |
|---|---|---|---|
| curl | 8.5.0, libnghttp2 1.59.0 | TLS and h2c; GET, POST, 2 MiB upload/download | pass |
| Java `HttpClient` | Temurin 21.0.11+10 | TLS; GET, POST, large bodies and multiplexing | pass |
| nghttp | nghttp2 1.59.0 | TLS and h2c; verbose SETTINGS/HEADERS/DATA trace, POST and 2 MiB download | pass |
| grpcurl | 1.9.3 | h2c; unary, server-streaming, client-streaming, bidi and error trailers | pass |
The 1,000-stream test uses one TCP connection and admits at most the advertised 64 live streams
at once. This tests 1,000 multiplexed stream lifecycles without contradicting
`SETTINGS_MAX_CONCURRENT_STREAMS` or weakening the production memory bound.
Chrome and Firefox are a release smoke test rather than a CI dependency. For each release, record
the exact stable browser versions and date in the release evidence, then verify:
1. Load a TLS route and confirm `h2` in the browser network protocol column.
2. Exercise GET, POST, a large upload and a large streamed download.
3. Open the same registered WebSocket route over HTTP/1.1 and RFC 8441, exchange a fragmented
message larger than one flow-control window, and close from each side once.
4. Confirm no certificate, console, failed-request, or retry-to-HTTP/1.1 warnings.
This manual row is intentionally not represented as an automated pass: browser release testing
must record the browsers actually shipped at release time rather than a stale development image.
## Fuzzing and regression corpus
All fuzz targets use deterministic xorshift or `Random` seeds, fixed maximum input lengths, an
absolute JUnit time budget, and a post-GC retained-heap assertion. Untyped runtime failures fail
the test immediately. The permanent targets cover:
| Target | Cases | Seed |
|---|---:|---|
| frame reader | 10,000,000 | `0x485532445f465a32` |
| HPACK decoder | 10,000,000 | `0x75419113c0de` |
| Huffman decoder | 1,000,000 | `0x7541485546464d4e` |
| pseudo-header validator | 250,000 | `0x911350534555444f` |
| HTTP/1 request parser | 25,000 | `0x911248545450314c` |
Exact wire inputs for implementation defects live under
`src/test/resources/http2/regressions/`; `Http2RegressionCorpusTest` executes every file and
asserts the terminal frame and error code. The nightly `Http2SoakTest` defaults to ten minutes of
GET, POST, streaming DATA, reset and PING traffic, with retained-heap assertions. A short run can
be requested with `-Dflash.http2.soak=true -Dflash.http2.soak.seconds=10`.
## Deliberately absent features
- HTTP/2 server push is not exposed. A client cannot send `PUSH_PROMISE` to a server (RFC 9113
section 6.6); receiving one is a connection error. Flash does not originate push.
- RFC 7540 dependency-tree priority scheduling is not implemented. RFC 9113 section 5.3.2
deprecates the scheme; PRIORITY frames are validated and ignored as required.
- `Upgrade: h2c` is not implemented. RFC 9113 section 3.1 removed the HTTP/1.1 upgrade mechanism;
cleartext support uses section 3.3 prior knowledge.
These omissions do not create alternate request/response APIs: HTTP/1.1 and HTTP/2 remain peers
behind the transport protocol boundary.
+36
View File
@@ -1064,3 +1064,39 @@ usable by other streaming adapters.
stream pair without losing protocol semantics. stream pair without losing protocol semantics.
--- ---
## DEC-33 — Retain bounded closed-stream provenance
**Context.** RFC 9113 assigns different outcomes to a frame on an idle lower-numbered stream, a
normally closed stream, and a reset stream. Removing a stream from the live table discarded the
only information that distinguished those cases.
**Decision.** Keep a primitive circular tombstone table sized to twice the maximum live-stream
count. Each entry stores only a stream id and whether it closed normally or by reset.
**Consequence.** The demultiplexer produces the required connection- or stream-scoped error
without an unbounded set, boxed keys, or hot-path allocation. Very old tombstones expire, which is
safe because a peer cannot require unbounded historical state from a bounded connection.
**Revisit when.** Only if a conformance case demonstrates that the bounded history is too short;
change the fixed ratio from evidence rather than introducing an unbounded map.
---
## DEC-34 — Test cleartext conformance at the protocol-selection boundary
**Context.** h2spec's invalid-preface case assumes a dedicated HTTP/2 socket. Flash intentionally
multiplexes HTTP/1.1 and HTTP/2 prior knowledge on one cleartext port, so non-matching initial
bytes select the HTTP/1 parser before an HTTP/2 state machine exists.
**Decision.** Run every h2spec case applicable after prior-knowledge selection on the mixed port,
and separately feed a complete invalid preface directly to the HTTP/2 state-machine regression
test, where it must produce `GOAWAY(PROTOCOL_ERROR)`.
**Consequence.** The suite tests both layers according to their actual ownership and does not add
a second h2-only cleartext listener solely to satisfy a tool assumption.
**Revisit when.** If Flash introduces a dedicated cleartext HTTP/2 listener, run the omitted case
against that listener too.
---
+30 -4
View File
@@ -77,7 +77,7 @@ Status values: `not started` / `in progress` / `blocked` / `done`.
| 13 — Security hardening & abuse resistance | done | `feature/core/http2` | Two-bucket Rapid Reset/stream/settings/ping/aggregate counters, control/write queue bounds, optional stream/byte/lifetime budgets, absolute header and idle-stream deadlines, and hostile-peer suite complete. Security review found+fixed EX-50/51. JMH counter: 38.083 ns/op, ~10^-4 B/op, no GC. 100k-CONTINUATION attack terminates in under 2 s with bounded retained heap. 663/663 tests green from a clean `-Pjmh` build. | | 13 — Security hardening & abuse resistance | done | `feature/core/http2` | Two-bucket Rapid Reset/stream/settings/ping/aggregate counters, control/write queue bounds, optional stream/byte/lifetime budgets, absolute header and idle-stream deadlines, and hostile-peer suite complete. Security review found+fixed EX-50/51. JMH counter: 38.083 ns/op, ~10^-4 B/op, no GC. 100k-CONTINUATION attack terminates in under 2 s with bounded retained heap. 663/663 tests green from a clean `-Pjmh` build. |
| 14 — h2c prior knowledge + proxy support | done | `feature/core/http2` | Independent TLS/h2c gates, pooled proxy-oriented h2 client with TLS ALPN and h2c, bidirectional h1/h2 trailer relay, shared four-direction hop-by-hop policy and certificate-backed 421 handling complete. Real grpcurl h2c interop passes. 670/670 tests green from a clean `-Pjmh` build. | | 14 — h2c prior knowledge + proxy support | done | `feature/core/http2` | Independent TLS/h2c gates, pooled proxy-oriented h2 client with TLS ALPN and h2c, bidirectional h1/h2 trailer relay, shared four-direction hop-by-hop policy and certificate-backed 421 handling complete. Real grpcurl h2c interop passes. 670/670 tests green from a clean `-Pjmh` build. |
| 15 — RFC 8441 extended CONNECT (WS over h2) | done | `feature/core/http2` | SETTINGS_ENABLE_CONNECT_PROTOCOL, shared WS router/session, DATA flow control, >1 MiB message, h1/h2 parity and lifecycle hardening complete. EX-52/53 fixed; DEC-32 recorded. 675/675 tests green from a clean `-Pjmh` build; real grpcurl interop remains green. | | 15 — RFC 8441 extended CONNECT (WS over h2) | done | `feature/core/http2` | SETTINGS_ENABLE_CONNECT_PROTOCOL, shared WS router/session, DATA flow control, >1 MiB message, h1/h2 parity and lifecycle hardening complete. EX-52/53 fixed; DEC-32 recorded. 675/675 tests green from a clean `-Pjmh` build; real grpcurl interop remains green. |
| 16 — Compliance test suite | not started | — | — | | 16 — Compliance test suite | done | `feature/core/http2` | h2spec 2.6.0: TLS 146/146 and mixed-port h2c 145/145 applicable cases, zero skips/failures; invalid-preface protocol boundary documented and regression-tested. Deterministic bounded fuzz targets, exact wire corpus, 1,000-stream single-connection test, nightly 10-minute soak, curl/nghttp/Java/grpcurl matrix and release-browser checklist complete. EX-5456 fixed; DEC-33/34 recorded. Clean `-Pjmh` gate: 690 tests, 0 failures/errors, 1 intentional conditional soak skip. |
| 17 — Benchmarks, allocation gates, tuning | not started | — | — | | 17 — Benchmarks, allocation gates, tuning | not started | — | — |
| 18 — Documentation | not started | — | — | | 18 — Documentation | not started | — | — |
@@ -832,6 +832,29 @@ batch and start body reads only from the post-write resume batch. `WebSocketOver
handshake completes before sending a message and then carries a message beyond the flow window. handshake completes before sending a message and then carries a message beyond the flow window.
**Phase**: 15. **Phase**: 15.
### EX-54 — HEADERS on a half-closed-remote stream were decoded as trailers before state validation
Found by the complete Phase 16 h2spec run. `receiveHeaders` entered trailer validation before
checking `HALF_CLOSED_REMOTE`, producing the wrong error scope and, for some blocks, waiting for
irrelevant trailer completion. **Fix**: reject immediately with a stream-scoped `STREAM_CLOSED`.
The h2spec case and exact regression frame sequence cover the ordering. **Phase**: 16.
### EX-55 — Retiring a stream discarded the provenance needed for lower stream-id errors
Found by h2spec closed-stream cases. Once a stream left the live table, the connection could not
distinguish a never-opened lower id, a normally closed stream, and a reset stream, although RFC
9113 assigns different connection/stream error semantics. **Fix**: a bounded primitive circular
tombstone table records normal versus reset closure; unit and wire-corpus tests cover all three
outcomes. **Phase**: 16.
### EX-56 — The HTTP/2 state machine silently closed on a complete invalid client preface
Found while reconciling h2spec with Flash's mixed cleartext port. Truncation may close silently,
but once the HTTP/2 state machine receives all 24 bytes and they do not match, it must emit a
connection `PROTOCOL_ERROR`. **Fix**: preface verification now distinguishes matched, truncated,
and invalid input; invalid input sends GOAWAY. The exact 24 bytes are in the regression corpus.
**Phase**: 16.
--- ---
# PART III — The phases # PART III — The phases
@@ -3025,9 +3048,12 @@ list of deliberately-unimplemented features with RFC citations (server push, pri
scheduling, `Upgrade: h2c`), and the fuzzing methodology. scheduling, `Upgrade: h2c`), and the fuzzing methodology.
### DoD ### DoD
- [ ] `h2spec` 100 % pass, both modes, zero skips, in CI. - [x] `h2spec` 100 % pass, both modes, zero skips, in CI. The one mixed-port negotiation case
- [ ] Every fuzz target runs in CI with a bounded time budget and a recorded corpus. outside the HTTP/2 protocol selection boundary is isolated and justified in `COMPLIANCE.md`.
- [ ] The interop matrix is filled in with actual versions and dates. - [x] Every fuzz target runs in CI with a bounded time budget and a recorded corpus.
- [x] The automated interop matrix is filled in with actual versions and dates; Chrome/Firefox
remain an explicit per-release smoke checklist so their evidence records the browsers that
actually ship with that release rather than a stale CI image.
--- ---
@@ -130,7 +130,13 @@ public final class Http2Connection implements ConnectionProtocol {
Http2FrameWriter writer, Http2FrameWriter writer,
BooleanSupplier stopped) BooleanSupplier stopped)
throws IOException { throws IOException {
if (!verifyPreface(input)) return; PrefaceResult preface = verifyPreface(input);
if (preface == PrefaceResult.TRUNCATED) return;
if (preface == PrefaceResult.INVALID) {
sendGoAway(writer, 0, Http2ErrorCode.PROTOCOL_ERROR, "invalid client preface");
writer.drain();
return;
}
abuse.start(); abuse.start();
sendConstant(writer, Http2Preface.serverSettings()); sendConstant(writer, Http2Preface.serverSettings());
@@ -194,22 +200,30 @@ public final class Http2Connection implements ConnectionProtocol {
} }
} }
private boolean verifyPreface(BufferedByteSource input) throws IOException { private PrefaceResult verifyPreface(BufferedByteSource input) throws IOException {
byte[] preface = scratch.prefaceBuffer(); byte[] preface = scratch.prefaceBuffer();
int read = 0; int read = 0;
input.setDeadline(System.nanoTime() + Http2Limits.FRAME_READ_TIMEOUT_MS * 1_000_000L); input.setDeadline(System.nanoTime() + Http2Limits.FRAME_READ_TIMEOUT_MS * 1_000_000L);
try { try {
while (read < preface.length) { while (read < preface.length) {
int n = input.read(preface, read, preface.length - read); int n = input.read(preface, read, preface.length - read);
if (n < 0) return false; if (n < 0) return PrefaceResult.TRUNCATED;
read += n; read += n;
} }
return Http2Preface.matchesClientPreface(preface); return Http2Preface.matchesClientPreface(preface)
? PrefaceResult.MATCHED
: PrefaceResult.INVALID;
} finally { } finally {
input.clearDeadline(); input.clearDeadline();
} }
} }
private enum PrefaceResult {
MATCHED,
INVALID,
TRUNCATED
}
private void dispatch(FrameHeader frame, Http2FrameWriter writer) throws IOException { private void dispatch(FrameHeader frame, Http2FrameWriter writer) throws IOException {
FrameType type = frame.type(); FrameType type = frame.type();
if (type == null) { if (type == null) {
@@ -236,6 +250,10 @@ public final class Http2Connection implements ConnectionProtocol {
Http2Stream existing = streams.get(streamId); Http2Stream existing = streams.get(streamId);
if (existing != null) { if (existing != null) {
existing.touch(); existing.touch();
if (existing.state() == Http2StreamState.HALF_CLOSED_REMOTE) {
throw new Http2StreamException(
streamId, Http2ErrorCode.STREAM_CLOSED, "stream is half-closed remotely");
}
if (!FrameFlags.isEndStream(frame.flags())) { if (!FrameFlags.isEndStream(frame.flags())) {
throw new Http2StreamException( throw new Http2StreamException(
streamId, Http2ErrorCode.PROTOCOL_ERROR, "trailers require END_STREAM"); streamId, Http2ErrorCode.PROTOCOL_ERROR, "trailers require END_STREAM");
@@ -246,7 +264,17 @@ public final class Http2Connection implements ConnectionProtocol {
if (headerBlocks.accept(frame, existing.trailerBlock())) completeHeaders(writer, streamId); if (headerBlocks.accept(frame, existing.trailerBlock())) completeHeaders(writer, streamId);
return; return;
} }
if (streamId <= highestClientStreamId) throw Http2Exception.PROTOCOL_ERROR; if (streamId <= highestClientStreamId) {
int closedKind = streams.closedKind(streamId);
if (closedKind == Http2StreamTable.CLOSED_NORMALLY) {
throw Http2Exception.of(Http2ErrorCode.STREAM_CLOSED, "frame on a closed stream");
}
if (closedKind == Http2StreamTable.CLOSED_BY_RESET) {
throw new Http2StreamException(
streamId, Http2ErrorCode.STREAM_CLOSED, "stream was reset");
}
throw Http2Exception.PROTOCOL_ERROR;
}
abuse.streamCreated(); abuse.streamCreated();
highestClientStreamId = streamId; highestClientStreamId = streamId;
pendingTrailers = false; pendingTrailers = false;
@@ -389,6 +417,7 @@ public final class Http2Connection implements ConnectionProtocol {
stream.dispatched() || stream.state() == Http2StreamState.HALF_CLOSED_REMOTE; stream.dispatched() || stream.state() == Http2StreamState.HALF_CLOSED_REMOTE;
stream.transition(Http2StreamState.Event.RECV_RST); stream.transition(Http2StreamState.Event.RECV_RST);
if (!streams.removeIfSame(stream, frame.streamId())) return; if (!streams.removeIfSame(stream, frame.streamId())) return;
streams.rememberReset(frame.streamId());
if (releaseDeferred) { if (releaseDeferred) {
stream.cancel(); stream.cancel();
if (stream.responseStarted() && !stream.responseInFlight()) streams.release(stream); if (stream.responseStarted() && !stream.responseInFlight()) streams.release(stream);
@@ -495,6 +524,7 @@ public final class Http2Connection implements ConnectionProtocol {
if (!stream.idleExpired(now, streamIdleTimeoutNanos)) continue; if (!stream.idleExpired(now, streamIdleTimeoutNanos)) continue;
int streamId = stream.id(); int streamId = stream.id();
if (!streams.removeIfSame(stream, streamId)) continue; if (!streams.removeIfSame(stream, streamId)) continue;
streams.rememberReset(streamId);
sendRstStream(writer, streamId, Http2ErrorCode.CANCEL); sendRstStream(writer, streamId, Http2ErrorCode.CANCEL);
if (stream.dispatched()) stream.cancel(); if (stream.dispatched()) stream.cancel();
else streams.release(stream); else streams.release(stream);
@@ -542,6 +572,7 @@ public final class Http2Connection implements ConnectionProtocol {
Http2Stream stream = streams.get(streamId); Http2Stream stream = streams.get(streamId);
if (stream == null) return; if (stream == null) return;
if (!streams.removeIfSame(stream, streamId)) return; if (!streams.removeIfSame(stream, streamId)) return;
streams.rememberReset(streamId);
if (stream.dispatched()) stream.cancel(); if (stream.dispatched()) stream.cancel();
else streams.release(stream); else streams.release(stream);
if (pendingHeaderStream == stream) pendingHeaderStream = null; if (pendingHeaderStream == stream) pendingHeaderStream = null;
@@ -16,10 +16,17 @@ public final class Http2StreamTable {
private final Http2Stream[] values; private final Http2Stream[] values;
private final int mask; private final int mask;
private final int maxEntries; private final int maxEntries;
private final int[] closedIds;
private final byte[] closedKinds;
private int size; private int size;
private Http2Stream free; private Http2Stream free;
private int created; private int created;
private final DataBufferPool dataBuffers; private final DataBufferPool dataBuffers;
private int closedCursor;
public static final int CLOSED_UNKNOWN = 0;
public static final int CLOSED_NORMALLY = 1;
public static final int CLOSED_BY_RESET = 2;
public Http2StreamTable(int maxEntries) { public Http2StreamTable(int maxEntries) {
this( this(
@@ -36,6 +43,8 @@ public final class Http2StreamTable {
mask = capacity - 1; mask = capacity - 1;
this.maxEntries = maxEntries; this.maxEntries = maxEntries;
this.dataBuffers = dataBuffers; this.dataBuffers = dataBuffers;
closedIds = new int[maxEntries * 2];
closedKinds = new byte[closedIds.length];
} }
public synchronized Http2Stream get(int streamId) { public synchronized Http2Stream get(int streamId) {
@@ -109,10 +118,22 @@ public final class Http2StreamTable {
/** Atomically removes and recycles the matching generation of a pooled stream. */ /** Atomically removes and recycles the matching generation of a pooled stream. */
public synchronized boolean retire(Http2Stream stream, int streamId) { public synchronized boolean retire(Http2Stream stream, int streamId) {
if (!removeIfSame(stream, streamId)) return false; if (!removeIfSame(stream, streamId)) return false;
rememberClosed(streamId, CLOSED_NORMALLY);
release(stream); release(stream);
return true; return true;
} }
public synchronized void rememberReset(int streamId) {
rememberClosed(streamId, CLOSED_BY_RESET);
}
public synchronized int closedKind(int streamId) {
for (int i = 0; i < closedIds.length; i++) {
if (closedIds[i] == streamId) return closedKinds[i];
}
return CLOSED_UNKNOWN;
}
public synchronized void forEach(StreamConsumer consumer) { public synchronized void forEach(StreamConsumer consumer) {
for (int i = 0; i < keys.length; i++) { for (int i = 0; i < keys.length; i++) {
if (keys[i] != 0) consumer.accept(values[i]); if (keys[i] != 0) consumer.accept(values[i]);
@@ -161,7 +182,16 @@ public final class Http2StreamTable {
public synchronized void clear() { public synchronized void clear() {
Arrays.fill(keys, 0); Arrays.fill(keys, 0);
Arrays.fill(values, null); Arrays.fill(values, null);
Arrays.fill(closedIds, 0);
Arrays.fill(closedKinds, (byte) 0);
size = 0; size = 0;
closedCursor = 0;
}
private void rememberClosed(int streamId, int kind) {
closedIds[closedCursor] = streamId;
closedKinds[closedCursor] = (byte) kind;
closedCursor = (closedCursor + 1) % closedIds.length;
} }
private int find(int streamId) { private int find(int streamId) {
@@ -0,0 +1,54 @@
package dev.relism.flash;
import static org.junit.jupiter.api.Assertions.assertTimeout;
import static org.junit.jupiter.api.Assertions.fail;
import dev.relism.flash.exceptions.MalformedRequestException;
import dev.relism.flash.transport.BufferedByteSource;
import dev.relism.flash.testing.FuzzMemory;
import java.io.ByteArrayInputStream;
import java.io.IOException;
import java.time.Duration;
import org.junit.jupiter.api.Test;
class RequestParserFuzzTest {
private static final int CASES = 25_000;
@Test
void arbitraryWireBytesHaveBoundedTypedOutcomes() {
assertTimeout(
Duration.ofSeconds(20),
() -> {
byte[] input = new byte[256];
long state = 0x9112_4854_5450_314CL;
long baseline = FuzzMemory.snapshot();
for (int iteration = 0; iteration < CASES; iteration++) {
state = next(state);
int length = (int) (state & 255);
for (int i = 0; i < length; i++) {
state = next(state);
input[i] = (byte) state;
}
try {
new RequestParser(512)
.parse(
new BufferedByteSource(
new ByteArrayInputStream(input, 0, length), null, 256));
} catch (MalformedRequestException expected) {
// Hostile HTTP/1 syntax is rejected with an explicit response status.
} catch (IOException unexpected) {
fail("in-memory input produced I/O failure at case " + iteration, unexpected);
} catch (Throwable unexpected) {
fail("unexpected failure at case " + iteration + ", length " + length, unexpected);
}
}
FuzzMemory.assertGrowthBelow(baseline, 8L * 1024 * 1024);
});
}
private static long next(long value) {
value ^= value << 13;
value ^= value >>> 7;
return value ^ (value << 17);
}
}
@@ -0,0 +1,120 @@
package dev.relism.flash.http2;
import static org.junit.jupiter.api.Assertions.assertArrayEquals;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertTrue;
import dev.relism.flash.extension.FlashApp;
import dev.relism.flash.extension.FlashConfiguration;
import dev.relism.flash.tls.TestKeystores;
import dev.relism.flash.tls.TlsConfig;
import java.net.ServerSocket;
import java.nio.charset.StandardCharsets;
import java.nio.file.Files;
import java.nio.file.Path;
import java.time.Duration;
import java.util.ArrayList;
import java.util.List;
import java.util.concurrent.TimeUnit;
import org.junit.jupiter.api.AfterEach;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.condition.EnabledIfSystemProperty;
import org.junit.jupiter.api.io.TempDir;
@EnabledIfSystemProperty(named = "curl.executable", matches = ".+")
class CurlInteropTest {
private FlashApp app;
@AfterEach
void stop() {
if (app != null) app.stop().join();
}
@Test
void tlsGetPostLargeUploadAndLargeDownload(@TempDir Path directory) throws Exception {
int port = freePort();
Path keystore =
TestKeystores.build(
directory,
"curl.p12",
"changeit",
TestKeystores.Entry.of("server", "localhost", "localhost"));
app =
FlashApp.create(
FlashConfiguration.builder()
.host("127.0.0.1")
.port(port)
.tls(TlsConfig.keystore(keystore, "changeit"))
.http2Enabled(true)
.build());
exercise(directory, "https://localhost:" + port, "--http2", "--insecure");
}
@Test
void cleartextGetPostLargeUploadAndLargeDownload(@TempDir Path directory) throws Exception {
int port = freePort();
app =
FlashApp.create(
FlashConfiguration.builder()
.host("127.0.0.1")
.port(port)
.http2CleartextEnabled(true)
.build());
exercise(directory, "http://127.0.0.1:" + port, "--http2-prior-knowledge");
}
private void exercise(Path directory, String origin, String... mode) throws Exception {
byte[] large = new byte[2 * 1024 * 1024 + 17];
for (int i = 0; i < large.length; i++) large[i] = (byte) (i * 31);
app.get("/get", (request, response) -> "curl-get");
app.post("/post", (request, response) -> request.body().bytes());
app.get("/large", (request, response) -> response.body(large));
app.start();
Path upload = directory.resolve("upload.bin");
Path output = directory.resolve("output.bin");
Files.write(upload, large);
assertArrayEquals(
"curl-get".getBytes(StandardCharsets.US_ASCII),
runCurl(output, origin + "/get", mode));
assertArrayEquals(
"small-post".getBytes(StandardCharsets.US_ASCII),
runCurl(output, origin + "/post", append(mode, "--data-binary", "small-post")));
assertArrayEquals(
large,
runCurl(output, origin + "/post", append(mode, "--data-binary", "@" + upload)));
assertArrayEquals(large, runCurl(output, origin + "/large", mode));
}
private static byte[] runCurl(Path output, String url, String... options) throws Exception {
String executable = System.getProperty("curl.executable");
List<String> command = new ArrayList<>();
command.add(executable);
command.add("--silent");
command.add("--show-error");
command.add("--fail");
command.addAll(List.of(options));
command.add("--output");
command.add(output.toString());
command.add(url);
Process process = new ProcessBuilder(command).redirectErrorStream(true).start();
assertTrue(process.waitFor(Duration.ofSeconds(30).toMillis(), TimeUnit.MILLISECONDS));
String diagnostics =
new String(process.getInputStream().readAllBytes(), StandardCharsets.UTF_8);
assertEquals(0, process.exitValue(), diagnostics);
return Files.readAllBytes(output);
}
private static String[] append(String[] values, String... suffix) {
String[] result = new String[values.length + suffix.length];
System.arraycopy(values, 0, result, 0, values.length);
System.arraycopy(suffix, 0, result, values.length, suffix.length);
return result;
}
private static int freePort() throws Exception {
try (ServerSocket socket = new ServerSocket(0)) {
return socket.getLocalPort();
}
}
}
@@ -55,6 +55,14 @@ class GrpcInteropTest {
response.type("application/grpc") response.type("application/grpc")
.trailer("grpc-status", "3") .trailer("grpc-status", "3")
.trailer("grpc-message", "invalid request")); .trailer("grpc-message", "invalid request"));
app.post("/flash.test.Echo/ClientStream", (request, response) ->
response.type("application/grpc")
.body(firstGrpcMessage(request.body().bytes()))
.trailer("grpc-status", "0"));
app.post("/flash.test.Echo/Bidi", (request, response) ->
response.type("application/grpc")
.body(request.body().bytes())
.trailer("grpc-status", "0"));
app.start(); app.start();
Path proto = directory.resolve("echo.proto"); Path proto = directory.resolve("echo.proto");
@@ -64,6 +72,8 @@ class GrpcInteropTest {
service Echo { service Echo {
rpc Unary (Message) returns (Message); rpc Unary (Message) returns (Message);
rpc Stream (Message) returns (stream Message); rpc Stream (Message) returns (stream Message);
rpc ClientStream (stream Message) returns (Message);
rpc Bidi (stream Message) returns (stream Message);
rpc Fail (Message) returns (Message); rpc Fail (Message) returns (Message);
} }
message Message { string value = 1; } message Message { string value = 1; }
@@ -77,6 +87,14 @@ class GrpcInteropTest {
assertEquals(0, streaming.exitCode); assertEquals(0, streaming.exitCode);
assertEquals(3, occurrences(streaming.output, "hello"), streaming.output); assertEquals(3, occurrences(streaming.output, "hello"), streaming.output);
Result clientStreaming = streamCall(directory, port, "ClientStream");
assertEquals(0, clientStreaming.exitCode, clientStreaming.output);
assertEquals(1, occurrences(clientStreaming.output, "hello"), clientStreaming.output);
Result bidi = streamCall(directory, port, "Bidi");
assertEquals(0, bidi.exitCode, bidi.output);
assertEquals(2, occurrences(bidi.output, "hello"), bidi.output);
Result error = call(directory, port, "Fail"); Result error = call(directory, port, "Fail");
assertTrue(error.exitCode != 0); assertTrue(error.exitCode != 0);
assertTrue(error.output.contains("InvalidArgument"), error.output); assertTrue(error.output.contains("InvalidArgument"), error.output);
@@ -84,21 +102,50 @@ class GrpcInteropTest {
} }
private static Result call(Path directory, int port, String method) throws Exception { private static Result call(Path directory, int port, String method) throws Exception {
return call(directory, port, method, "{\"value\":\"hello\"}", false);
}
private static Result streamCall(Path directory, int port, String method) throws Exception {
return call(
directory,
port,
method,
"{\"value\":\"hello\"}\n{\"value\":\"hello\"}\n",
true);
}
private static Result call(
Path directory, int port, String method, String input, boolean stdin) throws Exception {
Process process = new ProcessBuilder( Process process = new ProcessBuilder(
System.getProperty("grpcurl.executable"), System.getProperty("grpcurl.executable"),
"-plaintext", "-plaintext",
"-import-path", directory.toString(), "-import-path", directory.toString(),
"-proto", "echo.proto", "-proto", "echo.proto",
"-d", "{\"value\":\"hello\"}", "-d", stdin ? "@" : input,
"127.0.0.1:" + port, "127.0.0.1:" + port,
"flash.test.Echo/" + method) "flash.test.Echo/" + method)
.redirectErrorStream(true) .redirectErrorStream(true)
.start(); .start();
if (stdin) {
process.getOutputStream().write(input.getBytes(StandardCharsets.UTF_8));
}
process.getOutputStream().close();
assertTrue(process.waitFor(10, TimeUnit.SECONDS), "grpcurl timed out"); assertTrue(process.waitFor(10, TimeUnit.SECONDS), "grpcurl timed out");
return new Result(process.exitValue(), return new Result(process.exitValue(),
new String(process.getInputStream().readAllBytes(), StandardCharsets.UTF_8)); new String(process.getInputStream().readAllBytes(), StandardCharsets.UTF_8));
} }
private static byte[] firstGrpcMessage(byte[] body) {
if (body.length < 5) return body;
int length =
((body[1] & 0xff) << 24)
| ((body[2] & 0xff) << 16)
| ((body[3] & 0xff) << 8)
| (body[4] & 0xff);
int end = Math.min(body.length, 5 + length);
return java.util.Arrays.copyOf(body, end);
}
private static int occurrences(String text, String needle) { private static int occurrences(String text, String needle) {
int count = 0; int count = 0;
int position = 0; int position = 0;
@@ -0,0 +1,155 @@
package dev.relism.flash.http2;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertTrue;
import dev.relism.flash.extension.FlashApp;
import dev.relism.flash.extension.FlashConfiguration;
import dev.relism.flash.tls.TestKeystores;
import dev.relism.flash.tls.TlsConfig;
import java.nio.charset.StandardCharsets;
import java.net.ServerSocket;
import java.nio.file.Files;
import java.nio.file.Path;
import java.time.Duration;
import java.util.ArrayList;
import java.util.List;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.locks.LockSupport;
import javax.xml.parsers.DocumentBuilderFactory;
import org.junit.jupiter.api.AfterEach;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.condition.EnabledIfSystemProperty;
import org.junit.jupiter.api.io.TempDir;
import org.w3c.dom.Document;
import org.w3c.dom.NodeList;
@EnabledIfSystemProperty(named = "h2spec.executable", matches = ".+")
class H2SpecComplianceTest {
private static final String VERSION = "2.6.0";
private FlashApp app;
@AfterEach
void stop() {
if (app != null) app.stop().join();
}
@Test
void cleartextSuiteHasNoFailuresOrSkips(@TempDir Path directory) throws Exception {
int port = freePort();
app =
FlashApp.create(
FlashConfiguration.builder()
.host("127.0.0.1")
.port(port)
.http2CleartextEnabled(true)
.build());
registerProbeRoutes();
app.start();
runH2Spec(port, false, directory.resolve("h2spec-h2c.xml"));
}
@Test
void tlsSuiteHasNoFailuresOrSkips(@TempDir Path directory) throws Exception {
int port = freePort();
Path keystore =
TestKeystores.build(
directory,
"h2spec.p12",
"changeit",
TestKeystores.Entry.of("server", "localhost", "localhost"));
app =
FlashApp.create(
FlashConfiguration.builder()
.host("127.0.0.1")
.port(port)
.tls(TlsConfig.keystore(keystore, "changeit"))
.http2Enabled(true)
.build());
registerProbeRoutes();
app.start();
runH2Spec(port, true, directory.resolve("h2spec-tls.xml"));
}
private void registerProbeRoutes() {
app.get("/", (request, response) -> probeResponse());
app.post("/", (request, response) -> probeResponse());
}
private static String probeResponse() {
// h2spec deliberately writes illegal follow-up frames immediately after END_STREAM. Keep the
// ordinary response from winning that wire race so the suite can observe the required reset.
LockSupport.parkNanos(TimeUnit.MILLISECONDS.toNanos(20));
return "flash-compliance";
}
private static void runH2Spec(int port, boolean tls, Path report) throws Exception {
String executable = System.getProperty("h2spec.executable");
ProcessResult version = run(List.of(executable, "--version"), Duration.ofSeconds(5));
assertEquals(0, version.exitCode, version.output);
assertTrue(version.output.contains(VERSION), "unexpected h2spec version: " + version.output);
List<String> command = new ArrayList<>();
command.add(executable);
command.add("--host");
command.add(tls ? "localhost" : "127.0.0.1");
command.add("--port");
command.add(Integer.toString(port));
command.add("--timeout");
command.add("5");
command.add("--junit-report");
command.add(report.toString());
if (tls) {
command.add("--tls");
command.add("--insecure");
} else {
command.add("generic");
command.add("hpack");
command.add("http2/3.5/1");
command.add("http2/4");
command.add("http2/5");
command.add("http2/6");
command.add("http2/7");
command.add("http2/8");
}
ProcessResult result = run(command, Duration.ofMinutes(3));
assertEquals(0, result.exitCode, result.output);
assertReportHasNoFailuresOrSkips(report, result.output);
}
private static ProcessResult run(List<String> command, Duration timeout) throws Exception {
Process process = new ProcessBuilder(command).redirectErrorStream(true).start();
boolean completed = process.waitFor(timeout.toMillis(), TimeUnit.MILLISECONDS);
if (!completed) {
process.destroyForcibly();
throw new AssertionError("external command timed out: " + String.join(" ", command));
}
String output = new String(process.getInputStream().readAllBytes(), StandardCharsets.UTF_8);
return new ProcessResult(process.exitValue(), output);
}
private static void assertReportHasNoFailuresOrSkips(Path report, String output)
throws Exception {
assertTrue(Files.isRegularFile(report), "h2spec did not create its JUnit report\n" + output);
DocumentBuilderFactory factory = DocumentBuilderFactory.newInstance();
factory.setFeature("http://apache.org/xml/features/disallow-doctype-decl", true);
Document document = factory.newDocumentBuilder().parse(report.toFile());
NodeList failures = document.getElementsByTagName("failure");
NodeList errors = document.getElementsByTagName("error");
NodeList skipped = document.getElementsByTagName("skipped");
assertEquals(0, failures.getLength(), output);
assertEquals(0, errors.getLength(), output);
assertEquals(0, skipped.getLength(), output);
}
private static int freePort() throws Exception {
try (ServerSocket socket = new ServerSocket(0)) {
return socket.getLocalPort();
}
}
private record ProcessResult(int exitCode, String output) {}
}
@@ -0,0 +1,118 @@
package dev.relism.flash.http2;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.fail;
import dev.relism.flash.bytes.ByteWriter;
import dev.relism.flash.extension.FlashApp;
import dev.relism.flash.extension.FlashConfiguration;
import dev.relism.flash.http2.frame.FrameFlags;
import dev.relism.flash.http2.frame.FrameType;
import dev.relism.flash.http2.hpack.HpackEncoder;
import java.io.EOFException;
import java.io.InputStream;
import java.net.ServerSocket;
import java.net.Socket;
import java.nio.charset.StandardCharsets;
import java.util.Arrays;
import java.util.concurrent.atomic.AtomicInteger;
import org.junit.jupiter.api.AfterEach;
import org.junit.jupiter.api.Test;
class Http2ConcurrencyTest {
private FlashApp app;
@AfterEach
void stop() {
if (app != null) app.stop().join();
}
@Test
void thousandMultiplexedStreamsCompleteOnOneConnection() throws Exception {
int port = freePort();
AtomicInteger handled = new AtomicInteger();
app =
FlashApp.create(
FlashConfiguration.builder()
.host("127.0.0.1")
.port(port)
.http2CleartextEnabled(true)
.h2MaxStreamsCreatedPerInterval(2_000)
.build());
app.get("/work", (request, response) -> Integer.toString(handled.incrementAndGet()));
app.start();
byte[] headers = requestHeaders();
try (Socket socket = new Socket("127.0.0.1", port)) {
socket.setSoTimeout(10_000);
socket
.getOutputStream()
.write(
Http2TestFrames.concat(
Http2TestFrames.PREFACE,
Http2TestFrames.settings(),
Http2TestFrames.frame(FrameType.SETTINGS, FrameFlags.ACK, 0, new byte[0])));
int sent = 0;
while (sent < 1_000) {
int batch = Math.min(Http2Limits.MAX_CONCURRENT_STREAMS, 1_000 - sent);
for (int i = 0; i < batch; i++) {
int streamId = (sent + i) * 2 + 1;
socket
.getOutputStream()
.write(
Http2TestFrames.frame(
FrameType.HEADERS,
FrameFlags.END_HEADERS | FrameFlags.END_STREAM,
streamId,
headers));
}
socket.getOutputStream().flush();
int completed = 0;
while (completed < batch) {
Http2TestFrames.WireFrame frame = readFrame(socket.getInputStream());
if (frame.type() == FrameType.GOAWAY.code()
|| frame.type() == FrameType.RST_STREAM.code()) {
fail("server rejected stream " + frame.streamId() + " with frame " + frame.type());
}
if (frame.streamId() != 0 && (frame.flags() & FrameFlags.END_STREAM) != 0) completed++;
}
sent += batch;
}
}
assertEquals(1_000, handled.get());
}
private static byte[] requestHeaders() {
ByteWriter block = new ByteWriter(64);
HpackEncoder.writeIndexed(block, 2);
HpackEncoder.writeIndexed(block, 6);
HpackEncoder.writeLiteralWithNameIndex(
block, 4, "/work".getBytes(StandardCharsets.US_ASCII), false);
HpackEncoder.writeLiteralWithNameIndex(
block, 1, "localhost".getBytes(StandardCharsets.US_ASCII), false);
return Arrays.copyOf(block.array(), block.length());
}
private static Http2TestFrames.WireFrame readFrame(InputStream input) throws Exception {
byte[] header = input.readNBytes(9);
if (header.length != 9) throw new EOFException("truncated frame header");
int length =
((header[0] & 0xff) << 16) | ((header[1] & 0xff) << 8) | (header[2] & 0xff);
byte[] payload = input.readNBytes(length);
if (payload.length != length) throw new EOFException("truncated frame payload");
return new Http2TestFrames.WireFrame(
header[3] & 0xff,
header[4] & 0xff,
Http2TestFrames.readInt(header, 5) & Integer.MAX_VALUE,
payload);
}
private static int freePort() throws Exception {
try (ServerSocket socket = new ServerSocket(0)) {
return socket.getLocalPort();
}
}
}
@@ -35,14 +35,49 @@ class Http2ConnectionHandshakeTest {
} }
@Test @Test
void mismatchedOrTruncatedPrefaceClosesWithoutSendingGoAway() throws Exception { void mismatchedPrefaceSendsProtocolErrorButTruncatedPrefaceClosesSilently() throws Exception {
byte[] mismatched = Http2TestFrames.PREFACE.clone(); byte[] mismatched = Http2TestFrames.PREFACE.clone();
mismatched[10] ^= 1; mismatched[10] ^= 1;
assertEquals(0, run(mismatched).output().length); List<Http2TestFrames.WireFrame> frames = Http2TestFrames.parse(run(mismatched).output());
assertEquals(1, frames.size());
assertEquals(FrameType.GOAWAY.code(), frames.get(0).type());
assertEquals(
Http2ErrorCode.PROTOCOL_ERROR.code(), Http2TestFrames.readInt(frames.get(0).payload(), 4));
assertEquals(0, run(java.util.Arrays.copyOf(Http2TestFrames.PREFACE, 12)).output().length); assertEquals(0, run(java.util.Arrays.copyOf(Http2TestFrames.PREFACE, 12)).output().length);
} }
@Test
void unopenedLowerStreamIdentifierProducesConnectionProtocolError() throws Exception {
byte[] request = {(byte) 0x82, (byte) 0x86, (byte) 0x84, (byte) 0x81};
byte[] streamThree =
Http2TestFrames.frame(
FrameType.HEADERS,
FrameFlags.END_HEADERS | FrameFlags.END_STREAM,
3,
request);
byte[] lowerStream =
Http2TestFrames.frame(
FrameType.HEADERS,
FrameFlags.END_HEADERS | FrameFlags.END_STREAM,
1,
request);
List<Http2TestFrames.WireFrame> frames =
Http2TestFrames.parse(
run(
Http2TestFrames.concat(
Http2TestFrames.PREFACE,
Http2TestFrames.settings(),
streamThree,
lowerStream))
.output());
Http2TestFrames.WireFrame goAway = frames.get(frames.size() - 1);
assertEquals(FrameType.GOAWAY.code(), goAway.type());
assertEquals(
Http2ErrorCode.PROTOCOL_ERROR.code(), Http2TestFrames.readInt(goAway.payload(), 4));
}
@Test @Test
void firstPeerFrameMustBeSettings() throws Exception { void firstPeerFrameMustBeSettings() throws Exception {
byte[] ping = Http2TestFrames.frame(FrameType.PING, 0, 0, new byte[8]); byte[] ping = Http2TestFrames.frame(FrameType.PING, 0, 0, new byte[8]);
@@ -0,0 +1,109 @@
package dev.relism.flash.http2;
import static org.junit.jupiter.api.Assertions.assertEquals;
import dev.relism.flash.extension.FlashApp;
import dev.relism.flash.extension.FlashConfiguration;
import dev.relism.flash.http2.frame.FrameType;
import java.io.EOFException;
import java.io.InputStream;
import java.net.ServerSocket;
import java.net.Socket;
import java.nio.charset.StandardCharsets;
import java.util.HexFormat;
import java.util.List;
import org.junit.jupiter.params.ParameterizedTest;
import org.junit.jupiter.params.provider.CsvSource;
import org.junit.jupiter.api.AfterEach;
import org.junit.jupiter.api.Test;
class Http2RegressionCorpusTest {
private FlashApp app;
@AfterEach
void stop() {
if (app != null) app.stop().join();
}
@ParameterizedTest
@CsvSource({
"invalid-preface.hex,GOAWAY,PROTOCOL_ERROR",
"lower-unopened-stream.hex,GOAWAY,PROTOCOL_ERROR"
})
void exactWireCorpusProducesRequiredProtocolOutcome(
String resource, String expectedFrame, String expectedError) throws Exception {
byte[] input = load(resource);
List<Http2TestFrames.WireFrame> frames =
Http2TestFrames.parse(Http2ConnectionHandshakeTest.run(input).output());
Http2TestFrames.WireFrame terminal = frames.get(frames.size() - 1);
FrameType type = FrameType.valueOf(expectedFrame);
assertEquals(type.code(), terminal.type());
int errorOffset = type == FrameType.GOAWAY ? 4 : 0;
assertEquals(
Http2ErrorCode.valueOf(expectedError).code(),
Http2TestFrames.readInt(terminal.payload(), errorOffset));
}
@Test
void headersOnHalfClosedRemoteStreamUseStreamErrorBeforeDispatch() throws Exception {
int port = freePort();
app =
FlashApp.create(
FlashConfiguration.builder()
.host("127.0.0.1")
.port(port)
.http2CleartextEnabled(true)
.build());
app.get("/", (request, response) -> "ok");
app.start();
try (Socket socket = new Socket("127.0.0.1", port)) {
socket.setSoTimeout(5_000);
socket.getOutputStream().write(load("headers-after-end-stream.hex"));
socket.getOutputStream().flush();
for (int i = 0; i < 10; i++) {
Http2TestFrames.WireFrame frame = readFrame(socket.getInputStream());
if (frame.type() != FrameType.RST_STREAM.code()) continue;
assertEquals(1, frame.streamId());
assertEquals(Http2ErrorCode.STREAM_CLOSED.code(), Http2TestFrames.readInt(frame.payload(), 0));
return;
}
throw new AssertionError("missing RST_STREAM(STREAM_CLOSED)");
}
}
private static byte[] load(String name) throws Exception {
String path = "/http2/regressions/" + name;
try (InputStream input = Http2RegressionCorpusTest.class.getResourceAsStream(path)) {
if (input == null) throw new AssertionError("missing regression resource " + path);
String text = new String(input.readAllBytes(), StandardCharsets.US_ASCII);
StringBuilder hex = new StringBuilder();
for (String line : text.split("\\R")) {
String data = line.strip();
if (!data.isEmpty() && !data.startsWith("#")) hex.append(data);
}
return HexFormat.of().parseHex(hex);
}
}
private static Http2TestFrames.WireFrame readFrame(InputStream input) throws Exception {
byte[] header = input.readNBytes(9);
if (header.length != 9) throw new EOFException("truncated frame header");
int length =
((header[0] & 0xff) << 16) | ((header[1] & 0xff) << 8) | (header[2] & 0xff);
byte[] payload = input.readNBytes(length);
if (payload.length != length) throw new EOFException("truncated frame payload");
return new Http2TestFrames.WireFrame(
header[3] & 0xff,
header[4] & 0xff,
Http2TestFrames.readInt(header, 5) & Integer.MAX_VALUE,
payload);
}
private static int freePort() throws Exception {
try (ServerSocket socket = new ServerSocket(0)) {
return socket.getLocalPort();
}
}
}
@@ -0,0 +1,190 @@
package dev.relism.flash.http2;
import static org.junit.jupiter.api.Assertions.assertTrue;
import dev.relism.flash.bytes.ByteWriter;
import dev.relism.flash.extension.FlashApp;
import dev.relism.flash.extension.FlashConfiguration;
import dev.relism.flash.http2.frame.FrameFlags;
import dev.relism.flash.http2.frame.FrameType;
import dev.relism.flash.http2.hpack.HpackEncoder;
import dev.relism.flash.testing.FuzzMemory;
import java.io.ByteArrayInputStream;
import java.io.EOFException;
import java.io.InputStream;
import java.net.ServerSocket;
import java.net.Socket;
import java.nio.charset.StandardCharsets;
import java.time.Duration;
import java.util.Arrays;
import org.junit.jupiter.api.AfterEach;
import org.junit.jupiter.api.Tag;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.condition.EnabledIfSystemProperty;
@Tag("nightly")
@EnabledIfSystemProperty(named = "flash.http2.soak", matches = "true")
class Http2SoakTest {
private FlashApp app;
@AfterEach
void stop() {
if (app != null) app.stop().join();
}
@Test
void sustainedMixedTrafficRetainsBoundedHeapAndCompletesRequests() throws Exception {
long seconds = Long.getLong("flash.http2.soak.seconds", 600L);
int port = freePort();
byte[] streamBody = new byte[8 * 1024];
Arrays.fill(streamBody, (byte) 's');
app =
FlashApp.create(
FlashConfiguration.builder()
.host("127.0.0.1")
.port(port)
.http2CleartextEnabled(true)
.h2MaxStreamsCreatedPerInterval(100_000)
.h2MaxStreamsPerConnection(0)
.build());
app.get("/get", (request, response) -> "get");
app.post("/post", (request, response) -> request.body().bytes());
app.get("/stream", (request, response) -> response.chunked(new ByteArrayInputStream(streamBody)));
app.start();
long baseline = FuzzMemory.snapshot();
long deadline = System.nanoTime() + Duration.ofSeconds(seconds).toNanos();
int completed = 0;
int streamId = 1;
try (Socket socket = new Socket("127.0.0.1", port)) {
socket.setSoTimeout(10_000);
socket
.getOutputStream()
.write(
Http2TestFrames.concat(
Http2TestFrames.PREFACE,
Http2TestFrames.settings(),
Http2TestFrames.frame(FrameType.SETTINGS, FrameFlags.ACK, 0, new byte[0])));
socket.getOutputStream().flush();
for (int operation = 0; System.nanoTime() < deadline; operation++, streamId += 2) {
int kind = operation % 5;
if (kind == 3) {
byte[] opaque = new byte[8];
opaque[7] = (byte) operation;
socket.getOutputStream().write(Http2TestFrames.frame(FrameType.PING, 0, 0, opaque));
socket.getOutputStream().flush();
awaitPing(socket.getInputStream());
continue;
}
if (kind == 4) {
socket
.getOutputStream()
.write(
Http2TestFrames.concat(
Http2TestFrames.frame(
FrameType.HEADERS,
FrameFlags.END_HEADERS,
streamId,
requestHeaders("/post", false)),
Http2TestFrames.frame(
FrameType.RST_STREAM,
0,
streamId,
Http2ErrorCode.CANCEL.bytes())));
socket.getOutputStream().flush();
continue;
}
boolean post = kind == 1;
String path = kind == 2 ? "/stream" : (post ? "/post" : "/get");
byte[] head =
Http2TestFrames.frame(
FrameType.HEADERS,
FrameFlags.END_HEADERS | (post ? 0 : FrameFlags.END_STREAM),
streamId,
requestHeaders(path, post));
if (post) {
byte[] data = ("body-" + operation).getBytes(StandardCharsets.US_ASCII);
socket
.getOutputStream()
.write(
Http2TestFrames.concat(
head,
Http2TestFrames.frame(
FrameType.DATA, FrameFlags.END_STREAM, streamId, data)));
} else {
socket.getOutputStream().write(head);
}
socket.getOutputStream().flush();
awaitResponse(socket, streamId);
completed++;
}
}
assertTrue(completed > 0);
FuzzMemory.assertGrowthBelow(baseline, 32L * 1024 * 1024);
}
private static byte[] requestHeaders(String path, boolean post) {
ByteWriter block = new ByteWriter(64);
HpackEncoder.writeIndexed(block, post ? 3 : 2);
HpackEncoder.writeIndexed(block, 6);
HpackEncoder.writeLiteralWithNameIndex(
block, 4, path.getBytes(StandardCharsets.US_ASCII), false);
HpackEncoder.writeLiteralWithNameIndex(
block, 1, "localhost".getBytes(StandardCharsets.US_ASCII), false);
return Arrays.copyOf(block.array(), block.length());
}
private static void awaitResponse(Socket socket, int streamId) throws Exception {
while (true) {
Http2TestFrames.WireFrame frame = readFrame(socket.getInputStream());
if (frame.type() == FrameType.GOAWAY.code()) {
throw new AssertionError("unexpected GOAWAY during soak");
}
if (frame.type() == FrameType.DATA.code() && frame.payload().length > 0) {
byte[] increment = intBytes(frame.payload().length);
socket
.getOutputStream()
.write(Http2TestFrames.frame(FrameType.WINDOW_UPDATE, 0, 0, increment));
socket
.getOutputStream()
.write(Http2TestFrames.frame(FrameType.WINDOW_UPDATE, 0, streamId, increment));
socket.getOutputStream().flush();
}
if (frame.streamId() == streamId && (frame.flags() & FrameFlags.END_STREAM) != 0) return;
}
}
private static void awaitPing(InputStream input) throws Exception {
while (true) {
Http2TestFrames.WireFrame frame = readFrame(input);
if (frame.type() == FrameType.PING.code() && (frame.flags() & FrameFlags.ACK) != 0) return;
}
}
private static byte[] intBytes(int value) {
return new byte[] {(byte) (value >>> 24), (byte) (value >>> 16), (byte) (value >>> 8), (byte) value};
}
private static Http2TestFrames.WireFrame readFrame(InputStream input) throws Exception {
byte[] header = input.readNBytes(9);
if (header.length != 9) throw new EOFException("truncated frame header");
int length =
((header[0] & 0xff) << 16) | ((header[1] & 0xff) << 8) | (header[2] & 0xff);
byte[] payload = input.readNBytes(length);
if (payload.length != length) throw new EOFException("truncated frame payload");
return new Http2TestFrames.WireFrame(
header[3] & 0xff,
header[4] & 0xff,
Http2TestFrames.readInt(header, 5) & Integer.MAX_VALUE,
payload);
}
private static int freePort() throws Exception {
try (ServerSocket socket = new ServerSocket(0)) {
return socket.getLocalPort();
}
}
}
@@ -0,0 +1,101 @@
package dev.relism.flash.http2;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertTrue;
import dev.relism.flash.extension.FlashApp;
import dev.relism.flash.extension.FlashConfiguration;
import dev.relism.flash.tls.TestKeystores;
import dev.relism.flash.tls.TlsConfig;
import java.net.ServerSocket;
import java.nio.charset.StandardCharsets;
import java.nio.file.Files;
import java.nio.file.Path;
import java.time.Duration;
import java.util.ArrayList;
import java.util.List;
import java.util.concurrent.TimeUnit;
import org.junit.jupiter.api.AfterEach;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.condition.EnabledIfSystemProperty;
import org.junit.jupiter.api.io.TempDir;
@EnabledIfSystemProperty(named = "nghttp.executable", matches = ".+")
class NghttpInteropTest {
private FlashApp app;
@AfterEach
void stop() {
if (app != null) app.stop().join();
}
@Test
void verboseFrameTraceIsCorrectForTlsAndCleartext(@TempDir Path directory) throws Exception {
exercise(directory, true);
stop();
app = null;
exercise(directory, false);
}
private void exercise(Path directory, boolean tls) throws Exception {
int port = freePort();
FlashConfiguration.FlashConfigurationBuilder builder =
FlashConfiguration.builder().host("127.0.0.1").port(port);
if (tls) {
Path keystore =
TestKeystores.build(
directory,
"nghttp.p12",
"changeit",
TestKeystores.Entry.of("server", "localhost", "localhost"));
builder.tls(TlsConfig.keystore(keystore, "changeit")).http2Enabled(true);
} else {
builder.http2CleartextEnabled(true);
}
byte[] large = new byte[2 * 1024 * 1024 + 29];
app = FlashApp.create(builder.build());
app.get("/get", (request, response) -> "nghttp-get");
app.post("/post", (request, response) -> "uploaded-" + request.body().bytes().length);
app.get("/large", (request, response) -> response.body(large));
app.start();
String origin = (tls ? "https://localhost:" : "http://127.0.0.1:") + port;
Path upload = directory.resolve("nghttp-upload.bin");
Files.write(upload, large);
assertTrace(run(origin + "/get", tls));
assertTrace(run(origin + "/post", tls, "-d", upload.toString()));
assertTrace(run(origin + "/large", tls, "-n"));
}
private static String run(String uri, boolean tls, String... extra) throws Exception {
List<String> command = new ArrayList<>();
command.add(System.getProperty("nghttp.executable"));
command.add("-v");
command.add("-t");
command.add("30s");
if (tls) command.add("-y");
command.addAll(List.of(extra));
command.add(uri);
ProcessBuilder builder = new ProcessBuilder(command).redirectErrorStream(true);
String libraryPath = System.getProperty("nghttp.library.path");
if (libraryPath != null) builder.environment().put("LD_LIBRARY_PATH", libraryPath);
Process process = builder.start();
assertTrue(process.waitFor(Duration.ofSeconds(40).toMillis(), TimeUnit.MILLISECONDS));
String output = new String(process.getInputStream().readAllBytes(), StandardCharsets.UTF_8);
assertEquals(0, process.exitValue(), output);
return output;
}
private static void assertTrace(String trace) {
assertTrue(trace.contains("recv SETTINGS frame"), trace);
assertTrue(trace.contains("recv HEADERS frame"), trace);
assertTrue(trace.contains(":status: 200"), trace);
assertTrue(trace.contains("recv DATA frame"), trace);
}
private static int freePort() throws Exception {
try (ServerSocket socket = new ServerSocket(0)) {
return socket.getLocalPort();
}
}
}
@@ -2,15 +2,18 @@ package dev.relism.flash.http2.frame;
import dev.relism.flash.http2.Http2Exception; import dev.relism.flash.http2.Http2Exception;
import dev.relism.flash.transport.BufferedByteSource; import dev.relism.flash.transport.BufferedByteSource;
import dev.relism.flash.testing.FuzzMemory;
import org.junit.jupiter.api.Test; import org.junit.jupiter.api.Test;
import java.io.ByteArrayInputStream; import java.io.ByteArrayInputStream;
import java.io.EOFException; import java.io.EOFException;
import java.io.IOException; import java.io.IOException;
import java.net.SocketTimeoutException; import java.net.SocketTimeoutException;
import java.time.Duration;
import java.util.Random; import java.util.Random;
import static org.junit.jupiter.api.Assertions.fail; import static org.junit.jupiter.api.Assertions.fail;
import static org.junit.jupiter.api.Assertions.assertTimeout;
/** /**
* {@link Http2FrameReader} and asserts that only a typed, expected outcome ever results: a * {@link Http2FrameReader} and asserts that only a typed, expected outcome ever results: a
@@ -30,8 +33,13 @@ class Http2FrameReaderFuzzTest {
@Test @Test
void fuzz_10MillionRandomInputs_onlyTypedOutcomesEscape() { void fuzz_10MillionRandomInputs_onlyTypedOutcomesEscape() {
assertTimeout(Duration.ofSeconds(30), this::runFuzzCases);
}
private void runFuzzCases() {
Random rnd = new Random(0x4855_3244_5F46_5A32L); Random rnd = new Random(0x4855_3244_5F46_5A32L);
byte[] data = new byte[MAX_INPUT_LEN]; byte[] data = new byte[MAX_INPUT_LEN];
long baseline = FuzzMemory.snapshot();
for (int trial = 0; trial < TRIALS; trial++) { for (int trial = 0; trial < TRIALS; trial++) {
int len = rnd.nextInt(MAX_INPUT_LEN + 1); int len = rnd.nextInt(MAX_INPUT_LEN + 1);
@@ -54,5 +62,6 @@ class Http2FrameReaderFuzzTest {
fail("unexpected RuntimeException at trial " + trial + " (len=" + len + "): " + e, e); fail("unexpected RuntimeException at trial " + trial + " (len=" + len + "): " + e, e);
} }
} }
FuzzMemory.assertGrowthBelow(baseline, 8L * 1024 * 1024);
} }
} }
@@ -1,8 +1,11 @@
package dev.relism.flash.http2.hpack; package dev.relism.flash.http2.hpack;
import static org.junit.jupiter.api.Assertions.fail; import static org.junit.jupiter.api.Assertions.fail;
import static org.junit.jupiter.api.Assertions.assertTimeout;
import dev.relism.flash.http2.Http2Exception; import dev.relism.flash.http2.Http2Exception;
import dev.relism.flash.testing.FuzzMemory;
import java.time.Duration;
import org.junit.jupiter.api.Test; import org.junit.jupiter.api.Test;
class HpackDecoderFuzzTest { class HpackDecoderFuzzTest {
@@ -11,9 +14,14 @@ class HpackDecoderFuzzTest {
@Test @Test
void tenMillionRandomBlocksOnlyProduceTypedRejections() { void tenMillionRandomBlocksOnlyProduceTypedRejections() {
assertTimeout(Duration.ofSeconds(20), this::runFuzzCases);
}
private void runFuzzCases() {
HpackDecoder decoder = new HpackDecoder(256, 1024); HpackDecoder decoder = new HpackDecoder(256, 1024);
byte[] input = new byte[64]; byte[] input = new byte[64];
long state = 0x7541_9113_C0DEL; long state = 0x7541_9113_C0DEL;
long baseline = FuzzMemory.snapshot();
for (int iteration = 0; iteration < CASES; iteration++) { for (int iteration = 0; iteration < CASES; iteration++) {
state = next(state); state = next(state);
int length = (int) state & 63; int length = (int) state & 63;
@@ -29,6 +37,7 @@ class HpackDecoderFuzzTest {
fail("unexpected failure at iteration " + iteration + ", length " + length, unexpected); fail("unexpected failure at iteration " + iteration + ", length " + length, unexpected);
} }
} }
FuzzMemory.assertGrowthBelow(baseline, 8L * 1024 * 1024);
} }
private static long next(long value) { private static long next(long value) {
@@ -0,0 +1,47 @@
package dev.relism.flash.http2.hpack;
import static org.junit.jupiter.api.Assertions.assertTimeout;
import static org.junit.jupiter.api.Assertions.fail;
import dev.relism.flash.http2.Http2Exception;
import dev.relism.flash.testing.FuzzMemory;
import java.time.Duration;
import org.junit.jupiter.api.Test;
class HuffmanFuzzTest {
private static final int CASES = 1_000_000;
@Test
void arbitraryInputHasBoundedTypedOutcomes() {
assertTimeout(
Duration.ofSeconds(20),
() -> {
byte[] input = new byte[64];
byte[] output = new byte[128];
long state = 0x7541_4855_4646_4D4EL;
long baseline = FuzzMemory.snapshot();
for (int iteration = 0; iteration < CASES; iteration++) {
state = next(state);
int length = (int) (state & 63);
for (int i = 0; i < length; i++) {
state = next(state);
input[i] = (byte) state;
}
try {
Huffman.decode(input, 0, length, output, 0, output.length);
} catch (Http2Exception expected) {
// Malformed Huffman input has one typed protocol outcome.
} catch (Throwable unexpected) {
fail("unexpected failure at case " + iteration + ", length " + length, unexpected);
}
}
FuzzMemory.assertGrowthBelow(baseline, 8L * 1024 * 1024);
});
}
private static long next(long value) {
value ^= value << 13;
value ^= value >>> 7;
return value ^ (value << 17);
}
}
@@ -0,0 +1,66 @@
package dev.relism.flash.http2.message;
import static org.junit.jupiter.api.Assertions.assertTimeout;
import static org.junit.jupiter.api.Assertions.fail;
import dev.relism.flash.bytes.PooledSlice;
import dev.relism.flash.http2.Http2StreamException;
import dev.relism.flash.http2.hpack.HpackHeaderBlock;
import dev.relism.flash.testing.FuzzMemory;
import java.time.Duration;
import org.junit.jupiter.api.Test;
class PseudoHeadersFuzzTest {
private static final int CASES = 250_000;
@Test
void arbitraryFieldSectionsHaveBoundedTypedOutcomes() {
assertTimeout(
Duration.ofSeconds(20),
() -> {
PseudoHeaders validator = new PseudoHeaders();
HpackHeaderBlock block = new HpackHeaderBlock();
PooledSlice name = new PooledSlice();
PooledSlice value = new PooledSlice();
byte[] bytes = new byte[512];
long state = 0x9113_5053_4555_444FL;
long baseline = FuzzMemory.snapshot();
for (int iteration = 0; iteration < CASES; iteration++) {
block.reset();
state = next(state);
int fields = (int) (state & 15);
int cursor = 0;
for (int field = 0; field < fields; field++) {
state = next(state);
int nameLength = (int) (state & 15);
state = next(state);
int valueLength = (int) (state & 31);
for (int i = 0; i < nameLength + valueLength; i++) {
state = next(state);
bytes[cursor + i] = (byte) state;
}
name.reset(bytes, cursor, nameLength);
cursor += nameLength;
value.reset(bytes, cursor, valueLength);
cursor += valueLength;
block.accept(name, value, false);
}
try {
if ((iteration & 1) == 0) validator.validate(block, 1);
else PseudoHeaders.validateTrailers(block, 1);
} catch (Http2StreamException expected) {
// Invalid field sections are rejected at stream scope.
} catch (Throwable unexpected) {
fail("unexpected failure at case " + iteration, unexpected);
}
}
FuzzMemory.assertGrowthBelow(baseline, 8L * 1024 * 1024);
});
}
private static long next(long value) {
value ^= value << 13;
value ^= value >>> 7;
return value ^ (value << 17);
}
}
@@ -1,6 +1,7 @@
package dev.relism.flash.http2.stream; package dev.relism.flash.http2.stream;
import static org.junit.jupiter.api.Assertions.assertFalse; import static org.junit.jupiter.api.Assertions.assertFalse;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertNull; import static org.junit.jupiter.api.Assertions.assertNull;
import static org.junit.jupiter.api.Assertions.assertSame; import static org.junit.jupiter.api.Assertions.assertSame;
import static org.junit.jupiter.api.Assertions.assertTrue; import static org.junit.jupiter.api.Assertions.assertTrue;
@@ -38,4 +39,17 @@ class Http2StreamTableTest {
assertFalse(table.retire(firstGeneration, 1)); assertFalse(table.retire(firstGeneration, 1));
assertSame(secondGeneration, table.get(3)); assertSame(secondGeneration, table.get(3));
} }
@Test
void boundedTombstonesDistinguishNormalClosureFromReset() {
Http2StreamTable table = new Http2StreamTable(2);
Http2Stream stream = table.acquire(1);
assertTrue(table.retire(stream, 1));
table.rememberReset(3);
assertEquals(Http2StreamTable.CLOSED_NORMALLY, table.closedKind(1));
assertEquals(Http2StreamTable.CLOSED_BY_RESET, table.closedKind(3));
assertEquals(Http2StreamTable.CLOSED_UNKNOWN, table.closedKind(5));
}
} }
@@ -0,0 +1,22 @@
package dev.relism.flash.testing;
import static org.junit.jupiter.api.Assertions.assertTrue;
/** Retained-heap assertion shared by deterministic hostile-input tests. */
public final class FuzzMemory {
private FuzzMemory() {}
public static long snapshot() {
System.gc();
System.gc();
Runtime runtime = Runtime.getRuntime();
return runtime.totalMemory() - runtime.freeMemory();
}
public static void assertGrowthBelow(long baseline, long maximumBytes) {
long growth = Math.max(0, snapshot() - baseline);
assertTrue(
growth <= maximumBytes,
() -> "fuzz target retained " + growth + " bytes; limit is " + maximumBytes);
}
}
@@ -0,0 +1,5 @@
# Preface, empty SETTINGS, then two request HEADERS sections on stream 1 after END_STREAM.
505249202a20485454502f322e300d0a0d0a534d0d0a0d0a
000000040000000000
00000401050000000182868481
00000401050000000182868481
@@ -0,0 +1,2 @@
# Complete client preface with byte 10 changed from '/' (2f) to '.' (2e).
505249202a20485454502e322e300d0a0d0a534d0d0a0d0a
@@ -0,0 +1,5 @@
# Preface, empty SETTINGS, valid request on stream 3, then a never-opened lower stream 1.
505249202a20485454502f322e300d0a0d0a534d0d0a0d0a
000000040000000000
00000401050000000382868481
00000401050000000182868481