volto

This manual covers volto, a MASQUE proxy server written in Rust. One QUIC connection from the client carries TCP through classic CONNECT (RFC 9114 §4.4) and UDP through CONNECT-UDP (RFC 9298, HTTP Datagrams per RFC 9297), both dispatched by the :protocol pseudo-header. It terminates TLS itself, runs unattended on a small Linux host, reloads certificates and credentials on SIGHUP, and ships as a static binary. It is built to interoperate with Surge’s masque policy.

Install

One line on a fresh Ubuntu host downloads the newest release for the architecture, verifies it against SHA256SUMS, creates the system user, generates a self-signed certificate and a password, writes /etc/volto/config.toml, installs the systemd unit and starts it:

curl -fsSL https://raw.githubusercontent.com/vcarus/volto/main/script/deploy.sh |
  sudo bash -s -- --enable-timer --username yourname

It finishes by printing a ready-to-paste client policy line. Every flag, the certificate options and the tarball route are in the deployment chapter.

This manual

• Configuration — every key, its default, and what it costs to change.
• Deployment — building, certificates (ACME DNS-01 or self-signed plus pinning), releases and rollback, systemd, firewall, fd budget, reloads, relays, fail2ban.
• Architecture — how a request becomes a tunnel, the in-tree HTTP/3 layer, why quinn-proto is patched, and what the tests assert.

Reference

• API documentation — the crate’s rustdoc, rebuilt from main on every push.
• Source repository — the code, the issue tracker, and the tracked copy of these pages.
• Releases — static musl binaries for x86_64 and aarch64, with SHA256SUMS.
• Security policy — how to report a vulnerability privately.

Configuration

volto reads one TOML file, named with --config, or -c for short:

volto --config /etc/volto/config.toml
volto -c /etc/volto/config.toml

Only [server] is required; every other section and key has a default. Unknown keys are an error at startup rather than being silently ignored, so a typo fails loudly — in every table, not only in [server]. That also makes a config file forward-only, which matters when rolling a release back; see version compatibility. A commented reference file ships as script/config.example.toml.

--check-config answers whether a given binary can read a given file without starting anything; see checking a file.

SIGHUP re-reads the file. A file that fails to parse or validate is rejected whole and the running configuration keeps serving; see deployment.md.

[server]

[auth]

A username may not contain a colon (RFC 7617), and may not be longer than 32 bytes — the length a user-id is carried at in a log line, and therefore the length authentication failures are bucketed under, so a longer name could never have its failures cleared by its own success. Credentials are compared in constant time. Both Proxy-Authorization (preferred) and Authorization are accepted, because Surge’s manual does not say which it sends; a failed check is answered with 407 and Proxy-Authenticate: Basic.

[limits]

Notes that matter in practice

The fd budget is one number split in two. Every tunnel costs one file descriptor, so max_targets_per_conn and systemd’s LimitNOFILE are two halves of the same budget. What volto compares against RLIMIT_NOFILE at startup is max_connections × max_targets_per_conn plus FD_HEADROOM = 64 descriptors of headroom for the listening socket, the request streams and a certificate reload — 65600 for the defaults, which the shipped unit’s LimitNOFILE=131072 has room for. Raise either limit past that point and LimitNOFILE has to go up with it, or the startup warning fires.

A CONNECT-UDP session costs memory as well as a descriptor, and only the descriptors are checked at startup. Each session holds three buffers for its whole life: a 64 KiB receive buffer for the packets it reads off its target socket — it cannot be smaller, because a recv into a short buffer truncates the packet silently — an inbound datagram queue of 64 entries, each at most the 1472-byte max_udp_payload_size this server advertises, so about 92 KiB, and the capsule decoder on the request stream, which buffers one DATAGRAM capsule’s value until all of it has arrived and so tops out around 78 KiB. That is roughly 236 KiB per session, 59 MiB per connection at max_targets_per_conn = 256, and about 14.7 GiB across a server saturated at both defaults. Lowering either limit lowers it proportionally. It is a ceiling rather than a resting size: the queue is only full while a client sends faster than the proxy forwards, and the capsule buffer only fills for a client that leaves a capsule unfinished.

A TCP tunnel costs less, not nothing. Its relay buffer starts at RELAY_BUF_SIZE = 16 KiB, and settles on a single RELAY_BLOCK_SIZE = 64 KiB block once the tunnel has relayed anything: reads are cut from one block until too little of it is left to offer a full-sized window, and that is also why the first 16 KiB is let go after the first read. So the saturation product for TCP is max_connections × max_targets_per_conn × 64 KiB = 4 GiB at the defaults, beside the 9.8 GiB of the UDP one. What a tunnel holds beyond that one block is bounded by quinn’s per-connection send window (SEND_WINDOW = 10 MB): the pieces cut from a block share it, and each is held until the segment carrying it has been acknowledged, so the block outlives them all.

Every tunnel also holds its request’s header fields. volto advertises MAX_FIELD_SECTION_SIZE = 65536 in its SETTINGS, and a client is entitled to send a request that large; decoded, it costs roughly that much again — about 77 KiB measured — and it is kept for the whole life of the tunnel the request opened, not just until the target has been named. At the defaults that is another 19 MiB per connection and about 4.8 GiB across the server, on top of the tunnel figures above. Real clients send a few hundred bytes of headers, so this is a ceiling reached only by a client that chooses to fill the advertised limit; it is listed because nothing else in this section accounts for it.

connect_timeout is spent per request, not per connection. Without it a target that silently drops SYNs holds a tunnel slot and its file descriptor for as long as the operating system keeps retrying — around two minutes on Linux — so a handful of black-holed addresses during ordinary browsing can spend a connection’s whole max_targets_per_conn on tunnels that will never open. No attacker is needed for that. A reload carries a new value to connections accepted from then on, and each request those connections make gets the budget afresh; connections already open keep the value they were accepted with, like the rest of the per-connection policy. Set it to 0 only to hand the wait back to the operating system — and know what that hands back: a client that resets a request stream while its target is still being dialled does not cancel the dial, so with the budget off the slot stays spent until the kernel gives up. volto warns at startup when the budget is off.

connect_timeout bounds the answer, not the resolver. A lookup that runs out of budget is answered 504 immediately, but the getaddrinfo call behind it cannot be cancelled and keeps its thread until the system resolver gives up. The server bounds that separately: every connection has a name-lookup slot reserved for it that nothing else can take, plus a capped share of a server-wide allowance, so a client aiming at names that never resolve cannot stop anyone else’s names from resolving. Nothing is configurable there, and nothing changes on the wire — the refusals are the same 504 dns_timeout they always were.

max_connections also sizes the blocking thread pool, at startup only. The pool is given a thread for every reserved lookup slot the budget can hand out, plus the shared allowance and headroom; threads are created on demand and reaped when idle. A reload that raises max_connections takes effect for new connections but does not resize the pool, which needs a restart.

max_streams_bidi, max_idle_timeout, keep_alive_interval, initial_mtu, mtu_discovery, congestion_control and initial_rtt_ms are QUIC transport settings and apply to new connections only. A reload carries them to connections accepted from then on; connections already open keep what they negotiated at handshake time, because QUIC cannot renegotiate transport parameters. ip_family_preference is not a transport parameter, but it is snapshotted the same way: a connection resolves every target with the preference that was in force when it was accepted.

max_streams_bidi is the one of them that also moves within a connection’s life, when its first authenticated request raises the allowance from 16 to the configured value. That is not renegotiation — the raise travels as a MAX_STREAMS frame, which QUIC allows at any time and only ever upwards — and it makes the key no more reloadable than the rest: what a connection is raised to is the value that was in force when it was accepted, not whatever a later reload set. A reload during a connection’s unauthenticated window changes nothing about that connection either.

[server].listen is startup-only. A reload carrying a new value for it is accepted — the rest of the file still applies, because the usual sender of SIGHUP is a renewal hook that rewrites the whole file and refusing the reload over one key would be worse than ignoring it — but the socket does not move. The reload says so, naming both addresses, so an operator who did mean to move it is not left reading a successful reload that quietly did not apply:

WARN volto::quic: server.listen changed, but a reload cannot move the listening
socket; the server is still bound where it started. Restart to apply it.
bound=0.0.0.0:443 configured=0.0.0.0:8443

The two socket buffer keys are startup-only, not reloadable at all. They are applied to the UDP socket when it is created, and a reload does not rebind that socket — changing them needs a restart, the same as [server].listen. Each request is capped by the host: net.core.rmem_max / net.core.wmem_max on Linux, kern.ipc.maxsockbuf on macOS, and a host may fail the request outright rather than clamping it. volto warns at startup, naming the sysctl, whenever it got less than it asked for, and comes up on the operating system’s default either way. 0 asks for nothing and leaves that default in place. The reason these keys exist at all is that quinn never calls setsockopt itself, so a server that does not ask gets net.core.rmem_default — around 208 KiB — however high rmem_max has been raised; see UDP socket buffers.

keep_alive_interval is validated as strictly below half the idle timeout, not at most half. At exactly half, losing a single keep-alive packet is enough for the connection to time out. This pairing is what keeps a NAT mapping alive across an idle period; see running behind a UDP relay.

congestion_control should usually stay on BBR. Over a long, lossy path a loss-based controller (cubic, newreno) reads every dropped packet as congestion and collapses the window — downloads stall to near zero while a co-located TCP proxy, which the kernel runs on BBR, is unaffected. BBR models bandwidth and RTT instead. Switch to cubic only on a clean path, or as a fallback.

Path MTU discovery reports what it found in the connection close line. The INFO ... connection closed and WARN ... connection closed with error lines carry mtu=, the largest UDP payload the sender settled on for that path, and mtu_black_holes=, how many times quinn’s black-hole detector pushed it back to the floor during the connection, next to rtt_ms= and remote_now=. Both are reports, not knobs. A mtu= still at initial_mtu when a long-lived connection ends means the DPLPMTUD probes were never acknowledged, which is what a path that black-holes large packets looks like from here, and the case mtu_discovery = false exists for; anything above initial_mtu is discovery having done its job. The counter tells a fall-back apart from a path that never got there: the detector is a heuristic over loss bursts, and full-size packets lost to ordinary congestion during a bulk transfer look the same to it as a path that stopped carrying them, after which the connection sends packets at the 1200-byte floor for a one-minute cooldown before probing again. A non-zero count on a path where other connections settle above the floor is therefore that heuristic firing, not the path changing.

Those lines also report what the connection carried. Alongside rtt_ms= and mtu= they carry tunnels=, how many requests on that connection were granted a tunnel slot — TCP CONNECT and CONNECT-UDP draw on the same budget, and a request turned away before the slot (407, a malformed message, the tunnel limit itself) is not counted, while a destination the policy rejects and a target that could not be reached both are, since the slot is taken before the target is judged — and four transport counters. tx_bytes= and rx_bytes= are UDP-level byte counts: everything this server put on or took off the wire for that connection, QUIC and HTTP/3 framing, retransmissions, acknowledgements and padding included. They are neither tunnel payload, which is always smaller, nor bytes the peer acknowledged, since a packet is counted when it is sent whether or not it arrived; read them as how much this connection moved through the host, not as an accounting figure. sent_packets= and lost_packets= are reported together because a loss rate needs both, and a single count on its own says nothing about the path. dropped_datagrams= is this server’s own doing rather than the path’s: inbound HTTP Datagrams the connection’s router dropped on purpose — an unknown Context ID, a Quarter Stream ID no session claims, a session whose inbound queue was full, or a datagram cut short of its Context ID — each of which the RFCs require or permit to be silent where it happens, leaving this total as their only production-visible trace. All of them come from one snapshot taken as the connection ends, so they cost nothing while it is running.

initial_rtt_ms seeds the handshake retransmission timers. Until the first ACK arrives there is no RTT sample, and a lost handshake packet waits roughly three times this value before it is resent. The default of 333 comes from RFC 9002 and is deliberately conservative. On a known path, set 1.5–2× the RTT that volto’s connection logs report as rtt_ms — a measured ~90 ms path wants about 150 — which cuts the worst-case handshake stall from about a second to a few hundred milliseconds. The example configuration in script/ — and therefore every install derived from it — ships 150 for that reason; the compiled-in fallback used when the key is absent stays at 333. Keep the margin: a value below the real RTT makes the timer fire early and retransmit packets that were never lost.

The two MTU keys are shipped tuned as well. The example configuration in script/ ships initial_mtu = 1242 and mtu_upper_bound = 1464 as live keys, and the installer substitutes only the listen address, the certificate paths and the user — so every install derived from it runs above the compiled-in 1200 and 1452 that apply when the keys are absent. 1242 keeps the handshake inside a 1270-byte IPv4 packet: under the 1280 bytes any practical path carries, and below what Chromium (1250) and quic-go (1280) send everywhere. Over IPv6 the same packets are 1290 bytes, past that guarantee, so put it back to 1200 if clients reach the server over IPv6 — this is the key that is sent blind, and a size the path cannot carry kills the connection with nothing to fall back to. 1464 is what an IPv4 uplink behind a 1492-byte first-hop IP MTU (one PPPoE-sized deduction) leaves, and it is a ceiling for a search rather than a size that gets sent, so overshooting costs a probe and nothing else; 1472, clean Ethernet over IPv4, is the most volto accepts. Neither value is a measurement of your path: measure with ping -M do before raising either, and lower initial_mtu on the first sign that a handshake is not getting through.

ip_family_preference decides which half of a dual-stack target is tried first, and it is an operator’s call rather than the resolver’s. getaddrinfo sorts its answers by RFC 6724, which puts a global IPv6 address ahead of every IPv4 one whenever the host has a usable IPv6 route — the right answer for a host that is only a client of the internet, and the wrong one for a proxy whose IPv6 egress is tunnelled or worse peered than its native IPv4, which is a common shape on a VPS. volto therefore defaults to ipv4. A TCP tunnel would otherwise spend the whole IPv6 connect attempt before IPv4 is tried, and a CONNECT-UDP session would not recover at all: its socket is connected to the first address that has a route, and nothing later revisits that choice. Set ipv6 when the host’s IPv6 path is the better one — native IPv6 with tunnelled or NATed IPv4 — and system to hand the ordering back to the resolver, which on glibc can then be shaped through gai.conf. The ordering is a stable partition, so whatever RFC 6724 decided within a family still stands; a target that resolves to one family, an IP literal above all, is unaffected by any of the three.

[security]

• This whole section is snapshotted per connection, at accept. A reload applies it to connections accepted after it, and a connection already open keeps the rules it was accepted with for its whole life, tunnels opened on it later included, so a tightened allow_private_networks or a new entry in denied_ports reaches a client that is holding a connection open only when that connection ends. Use systemctl restart volto when the tightening has to apply to everything at once; the same reasoning and the same remedy as for credentials, worked through under Reloading in deployment.md.
• Addresses are normalized before matching, so neither ::ffff:127.0.0.1 nor ::127.0.0.1 gets past allow_private_networks = false.
• IPv6 transition addresses are judged by the IPv4 address they carry, because that is what a host routing them actually reaches: the well-known NAT64 prefix (64:ff9b::/96), 6to4 (2002::/16) and Teredo (2001::/32). So 64:ff9b::7f00:1 is refused as the 127.0.0.1 it is, while 64:ff9b::808:808 is reachable as 8.8.8.8. The local-use NAT64 prefix 64:ff9b:1::/48 is judged differently: its operator picks where the IPv4 address sits inside it and RFC 8215 §5 forbids a reader assuming a layout, so the whole prefix counts as private and follows allow_private_networks.
• Multicast, broadcast and the unspecified address are never dialled, regardless of that setting. They are amplification primitives, not destinations. What the client is told about the unspecified address is a separate question — see the note on blackholed names under [log].
• This host’s own addresses are never dialled either, at any port and regardless of that setting, because the reason is a different one: a tunnel to an address this machine carries reaches the machine’s own services with the proxy’s own source address, which is the privilege escalation RFC 9298 §7 names. Loopback is the exception, and it is the exception the RFC makes too: it lists “localhost” as a class of its own, and here that class is the private one above, off by default and opened by allow_private_networks. So a target is refused with 403 and destination_ip_prohibited when it resolves to an address of this host other than loopback, whether that address is public, on the LAN, or the one the server listens on.
• What that rule does not cover, said rather than left to be discovered. With allow_private_networks = true a client can still reach this server’s own listener through 127.0.0.1, exactly as it can reach every other loopback service the operator opened: loopback is the private class above, and that switch is what decides it. And behind a relay the address a client dials the proxy on is the relay’s, not one any interface here carries, so a target naming that address is judged like any other public address and it is the relay rather than this rule that stands in the way.
• UDP/53 must stay reachable. Surge’s UDP availability test is a DNS query through the tunnel, so denying port 53 makes Surge report the policy as broken. volto warns if 53 appears in denied_ports.
• unanswered_packet_budget stops a client using the proxy as a reflector or a port scanner (RFC 9298 §7). The first reply from the target lifts the limit for the rest of the session, so it only bites on one-way floods. The default is generous on purpose: handshakes that legitimately need several packets before the first reply must not break. The connection carries a total of its own, CONNECTION_UNANSWERED_MULTIPLIER = 8 times this value (512 packets at the default), spent by every session on it that sends into silence. A session whose target answers gives back what that session spent, and nothing else does, so opening a new session no longer restores the allowance, a long-lived connection of short answered sessions (DNS through the tunnel) spends none of it, and a session that finds the total spent is closed rather than left running and muted. 0 switches both halves off.
• max_auth_failures is not a rate limit and not a ban. It raises the cost of guessing from “one handshake, then unlimited attempts” to “one handshake per N attempts”, without any cross-connection state to keep or evict. What it counts is credential values rather than requests: every value that is tried and refused costs one failure, so a request carrying a guess under each of the two accepted field names spends two. A request carrying more than two credential values is answered 400 before any of them is tried, and costs nothing, because two is all that accepting both field names asks for. N is therefore a number of guesses whatever shape they arrive in. Pair it with fail2ban for actual banning — see deployment.md.

The SNI gate

expected_sni is the one key here that changes what an outsider sees rather than what a client may reach. With it set, the UDP port stops answering anybody who cannot name the host: the check runs on the datagram itself, under the QUIC layer, and a handshake that fails it is discarded rather than refused. There is no reply of any kind, so a scan of the address finds a closed port.

What it is for is a deployment whose address is not the name — volto behind a relay, where the port is on an IP that nothing else advertises. What it is not is traffic obfuscation: it says nothing about what a connection looks like once one is open, and it does not stop a probe that already knows the name.

Turning it on is a two-sided change. Set it to the name on the certificate, and make sure every client sends that name as SNI. Surge does when the policy line names the host; when the policy points at a relay’s IP instead, that is the sni= parameter (with server-cert-verify-name=), which such a deployment already needs. A client that sends the wrong name — or none — gets silence, and silence is indistinguishable from the server being down, so the two halves belong in the same change window.

Some details worth knowing before relying on it:

• A name is checked twice. The socket-level check reads the ClientHello out of the client’s first Initial packet, which is where a ClientHello normally fits whole. One that is split across several packets is deliberately let through — refusing a first flight before its extensions have arrived would make a large ClientHello unreachable — and is stopped a layer up instead, by a certificate resolver that declines to present a certificate for a name it does not know. That second refusal is a TLS alert rather than silence, which is the one case where a probe learns something.
• Reloadable. SIGHUP applies a new list to handshakes from then on; connections already open are unaffected, exactly like the rest of the file.
• Not a certificate selector. There is still one certificate and one key. Naming several hosts means volto answers to all of them with the same certificate, so they all have to be on it.
• A packet that names nobody can still draw an acknowledgement. The check refuses an Initial whose Destination Connection ID is under the eight bytes RFC 9000 §7.2 requires of a client’s first packet, since no later one is that shape either and quinn applies the same floor before decrypting; what it passes, it passes to quinn, which acknowledges an Initial carrying an ack-eliciting frame whether or not a name is in it. Sending one means building a QUIC Initial packet on purpose, which is a probe aimed at this server rather than a scan of the address.
• Stateless resets are not covered. A short-header packet for a connection this server does not hold can still draw one (RFC 9000 §10.3), because telling a live connection’s packets from a stranger’s needs state the gate does not have, and filtering on the address instead would break the connection migration a phone behind a relay’s NAT depends on. The residue is narrow: the packet has to be at least 22 bytes, start with the bits 01, and carry eight bytes that pass the endpoint’s own keyed connection-ID check. A scanner’s probe does not.
• Rejected names. A name that no ClientHello could carry is refused at startup rather than accepted and never matched — an empty string, a wildcard, a name over 253 bytes, anything outside the ASCII letters, digits, -, . and _ an A-label is made of. The failure mode this avoids is a gate that starts cleanly and silently refuses every client.

[log]

At debug, every inbound request is logged with its method, path, :protocol and header lines. Credential values are replaced with <scheme> <redacted N bytes>, but the header names are kept — that is what makes the log usable for confirming which authorization header a client actually sends.

A keylog file decrypts every session through the proxy, including sessions already recorded. Turn it off and delete the file when you are done.

Under the shipped unit it writes nowhere. ProtectSystem=strict with ReadOnlyPaths=/etc/volto and no ReadWritePaths= leaves the service no writable path but its own PrivateTmp=yes directory, and rustls reports the failure to open once and carries on, so the operator gets a server that started, a startup warning saying the keylog is on, and no file. Give it a directory and a path with a drop-in:

# /etc/systemd/system/volto.service.d/keylog.conf
[Service]
Environment=SSLKEYLOGFILE=/var/lib/volto/keylog
ReadWritePaths=/var/lib/volto

after sudo install -d -o volto -g volto -m 0700 /var/lib/volto, then systemctl daemon-reload and restart. Remove the drop-in and the directory together with keylog = false when the session is over: the file is every secret the proxy has negotiated since it was created.

Under systemd, volto prefixes each line with a syslog priority (<3> for ERROR, <4> for WARN, <6> for INFO, <7> for DEBUG and TRACE). journald parses that prefix, strips it, and files the record with the matching PRIORITY, so journalctl -u volto -p warning selects what it says it does instead of matching everything. The prefix appears only when systemd sets JOURNAL_STREAM, so running volto in a terminal prints the same lines it always did, and the shipped unit needs no extra setting (SyslogLevelPrefix= already defaults to true).

Every line at info or above carries a log_id field, eight lowercase letters and digits. It names one statement in the source, is assigned once and is never reused, and it stays the same when that line is reworded, so it is what a runbook or a journal filter should match on. The message text is not covered by the compatibility promise below; the id is the stable half of the pair.

One refusal is deliberately quieter than its neighbours, in the log and on the wire. A target whose every resolved address is 0.0.0.0 or :: is a name a filtering resolver has blackholed: that decision belongs to the resolver, not to volto, so it is logged at INFO and answered with a 200 whose stream is closed immediately — the client sees a tunnel that opened and died, which is what a blocked name looks like through a transport that has no way to explain itself. Answering 403 instead would invite the client to blame the proxy for an ad blocker’s decision. A target that resolves to loopback, private or mixed addresses is a refusal volto really did make: it stays a WARN and a 403 with Proxy-Status: …; error=destination_ip_prohibited, because that is what a probe for internal services looks like from here.

That warning, and the one a request refused for reaching max_targets_per_conn writes, are reported on a doubling schedule per connection: the 1st, 2nd, 4th, 8th and so on, each carrying refusals= with the running total, and the ones in between at DEBUG. Both are refusals a client can repeat as fast as it can open request streams, and journald’s rate limiting counts lines, so one line per request would let a peer decide how much of the journal is left to record anything else in — including the lines about it. A scan of every port on a host is 17 warnings rather than 65535, the first arrives as promptly as it ever did, and the last one says how large the scan was.

Checking a file without starting the server

volto --check-config --config /etc/volto/config.toml

reads the file, validates it and exits. A file that is good exits 0 with one line on stdout; a file that is not exits non-zero with the reason on stderr — the same error, word for word, that the service would print at startup, because it is the same code path. Settings that are legal but worth a word ([auth].users empty, log.keylog on) are printed to stderr as warnings and do not change the exit status: they describe a server that runs, not one that refuses to.

Nothing is bound, started or written, so this needs no privilege beyond reading the file it is given — which on a host set up by install-selfsigned.sh still means root or sudo -u volto, since /etc/volto is 0750 volto:volto.

What it covers is what startup does with the file before it becomes a server: TOML syntax, every table’s refusal of a key it does not know, and every range and cross-field rule (keep_alive_interval against max_idle_timeout, mtu_upper_bound against initial_mtu, and the rest) — including that server.cert and server.key exist and are files.

What it does not cover is what only the running service can answer: whether the listen address is free, whether the certificate and key parse and form a usable pair, and whether RLIMIT_NOFILE leaves room for max_connections × max_targets_per_conn. That last one is a property of the systemd unit rather than of the file, so answering it from a shell would answer a different question than the one that matters. A file that passes can still fail to serve for one of those reasons; a file that fails will not start at all.

The reason the flag exists is the section below: it is how script/deploy.sh asks a release it is about to install whether it can read the file this host already has, before it swaps the binary.

Collecting a support bundle

volto --diagnostics --config /etc/volto/config.toml

prints, to stdout, everything an issue about a host would otherwise be a series of questions about, and exits 0. In order: the version of the binary that printed it; the configuration file’s path and every table of it as this binary parsed it, after defaults, so [limits] and [security] are the values the server would actually run on rather than the subset the file happens to name; the warnings --check-config prints; the descriptor limits of the process that ran the command, soft and hard, since the hard limit is what says whether a soft one that is too low can be raised here at all or needs the unit changed, and on Linux the same pair for the running service, read from /proc; the four net.core UDP buffer sysctls named under UDP socket buffers, read from /proc/sys on Linux and reported as unavailable on any other platform; and uname -srm.

Passwords are redacted. Every [auth].users entry prints its password as <redacted>, by the same guard that keeps one out of the error a malformed file produces, so the output is safe to paste into an issue. Read it before pasting anyway: the rest of the configuration is there in full, host names and listen address included.

The descriptor limits are this command’s, not the service’s. Run from an SSH shell the two RLIMIT_NOFILE lines are that shell’s, which is how 1024 and 1048576 came to be recorded for two hosts running the service at 131072. The line under them says so, and on Linux the section then prints the running service’s own figures, read from /proc/<pid>/limits:

[file descriptors]
RLIMIT_NOFILE soft = 1024
RLIMIT_NOFILE hard = 1048576
these two are this process's own limits, not the service's (the service's are in /proc/<MainPID>/limits)
service pid 5312 Max open files soft = 131072
service pid 5312 Max open files hard = 131072

The service is found by walking /proc, so no unit name is assumed and nothing is asked of systemctl: a process matches when its comm is volto, or when its exe link resolves to the binary this command is running. The name alone is enough because the link is readable only with privilege, reads (deleted) once the deploy script has replaced the file under the running service, and is a different path when this command is run from a freshly unpacked tarball. Every match is printed with its own pid, so a second volto started by hand is visible rather than folded into one answer. A host with no such process prints one line saying so, and a file that cannot be read prints one line saying that; neither changes the exit status. On any other platform the section says these figures are Linux only. To read them by hand, systemctl show -p MainPID volto gives the pid for /proc/<MainPID>/limits.

Nothing is bound, connected or resolved, nothing is written, and no journal is read, so this asks for no more privilege than reading the configuration file does. The two flags refuse to be combined: --check-config and --diagnostics answer different questions and neither is the obvious winner, so a command line naming both is rejected rather than one of them ignored.

Version compatibility

An unknown key is refused, and the file is refused whole rather than the one key: nothing else in it takes effect, so this is a startup failure and not a warning. Two consequences, and they are not symmetric.

Upgrading is always safe. A key a later release adds takes its documented default when the file does not mention it, and nothing ever rewrites /etc/volto/config.toml — not script/deploy.sh on an update, not install-selfsigned.sh on a re-run. No key has ever been renamed or removed, so a file written for any earlier release still loads.

Rolling back is not. A file that names a key the older binary does not know stops it from starting at all, which is the one moment that costs the most: a rollback is only ever run when something is already wrong. The failure looks like this in the journal, and it names the file, the line and the column but not the key — a parse error redacts every quoted segment of the parser’s message, because the same message can quote a password, and a key is quoted the same way:

Error: failed to parse config file /etc/volto/config.toml at line 181, column 1: unknown field `<redacted>`

The line number is the thread to pull: comment that key out and start the service. To find out before the restart instead of after it, ask the binary you are about to install:

/path/to/old/volto --check-config --config /etc/volto/config.toml

which is what script/deploy.sh does on your behalf — but only for a release that knows the flag. Rolling back past the release that introduced it leaves you with the table below and the line number in the journal.

Everything else has been there since v0.1.0. mtu_upper_bound is the one that bites in practice, because the shipped example sets it and every install made by install-selfsigned.sh is derived from that example — so a host first installed at v0.4.5 or later carries it whether or not anyone chose it.

Why the rejection stays. Ignoring unknown keys would make rollback a non-event, and it would also make one typo dangerous rather than merely wrong. Every misspelled key falls back to a default; for [auth].users that default is the empty list, and an empty user list is an open proxy. A configuration that silently drops its credentials is a worse failure than a service that refuses to start, so the refusal stands and the rollback cost is paid in this section instead.

What is stable from v1.0.0

Configuration keys, their defaults and the command-line arguments are stable within 1.x. A key that exists keeps its name and its meaning, a documented default is not changed under a running deployment, and a key is removed only after at least one minor release has warned about it at startup. A new key keeps taking its documented default when the file does not mention it, which is what makes the upgrade half of this section a no-op.

Two things are deliberately outside that promise. Log line shapes are not an interface: fields are added, reworded and moved between levels as the operational picture changes, and a parser built on them is built on something that moves. Neither is the volto library API: the crate is a library so that the tests and the fuzz targets can reach the parsers and the bounds directly, and every item it exposes exists for them.

A minimal file

[server]
listen = "0.0.0.0:443"
cert   = "/etc/volto/fullchain.pem"
key    = "/etc/volto/privkey.pem"

[auth]
users = [{ username = "user1", password = "…" }]

Deployment

Target platform is a Linux host with systemd; the development host is macOS and the test suite is expected to pass on both.

A stock Ubuntu 24.04 image already carries everything the release path needs: curl, tar, sha256sum, openssl and systemd itself. ufw is not among them and neither script requires it — the installer notices its absence and says so instead of failing. Building on the host instead of downloading a release is what adds requirements, and they are named under building.

Building

Rust 1.95 or newer. Cargo.lock is committed, and quinn-proto is redirected by a [patch.crates-io] stanza to a commit carrying an MTU fix no release has yet (see architecture.md), so build with the lockfile and do not run cargo update:

cargo build --release --locked      # target/release/volto

On Debian or Ubuntu that needs build-essential and pkg-config. Compiling on the server itself is the simplest route; the dependency stack is pure Rust apart from the C code aws-lc-rs builds, so cross-compiling works as well:

rustup target add x86_64-unknown-linux-musl
cargo build --release --locked --target x86_64-unknown-linux-musl

Cross-compiling to musl needs a C toolchain for aws-lc-rs (musl-tools for the x86_64 target, a cross toolchain for aarch64). The non-FIPS build used here ships pre-generated bindings, so no cmake, bindgen or Go is involved – a C compiler is enough. The release workflow builds both targets with cross; the resulting static binaries are attached to each tagged release together with a SHA256SUMS file. Each archive unpacks to the binary, LICENSE, README.md, script/ and docs/ – so the installer runs straight out of it, and the pages both scripts name when they refuse to install are on the host rather than only on the web.

Certificates

Two paths. Pick one.

ACME with DNS-01

The right choice when you own a domain and want clients to need no extra configuration. When the domain’s A record points at a UDP relay rather than at the server, HTTP-01 and TLS-ALPN-01 cannot validate — they are answered at the address the record names — so DNS-01 is the only usable challenge:

sudo apt install -y certbot python3-certbot-dns-cloudflare   # or your provider's plugin
sudo certbot certonly \
  --dns-cloudflare --dns-cloudflare-credentials /root/.secrets/cloudflare.ini \
  --key-type ecdsa \
  -d example.com

--key-type ecdsa is not only a performance preference. Until a client’s address is validated, QUIC lets a server send at most three times the bytes it received (RFC 9000 §8.1, roughly a 3600-byte budget). An RSA chain can exceed that and cost the handshake an extra round trip; an ECDSA chain fits comfortably.

The symlinks under /etc/letsencrypt/live/ are readable by root only, so rather than pointing volto at them, have the renewal hook copy the files into /etc/volto and reload:

sudo tee /etc/letsencrypt/renewal-hooks/deploy/volto.sh >/dev/null <<'EOF'
#!/bin/sh
set -e
install -o volto -g volto -m 0644 /etc/letsencrypt/live/example.com/fullchain.pem /etc/volto/fullchain.pem
install -o volto -g volto -m 0640 /etc/letsencrypt/live/example.com/privkey.pem   /etc/volto/privkey.pem
systemctl reload volto
EOF
sudo chmod +x /etc/letsencrypt/renewal-hooks/deploy/volto.sh
sudo /etc/letsencrypt/renewal-hooks/deploy/volto.sh   # run it once by hand

Self-signed with fingerprint pinning

The right choice for a handful of your own devices when maintaining a domain and DNS-01 credentials is not worth it. The client verifies one specific certificate by its SHA-256 fingerprint instead of a chain.

Start from a release tarball; nothing has to be built on the host. The archives are named volto-<version>-<target>.tar.gz for the two published targets, x86_64-unknown-linux-musl and aarch64-unknown-linux-musl — which is what uname -m already answers on a Linux host:

version=1.0.2                  # whatever the releases page shows
name="volto-${version}-$(uname -m)-unknown-linux-musl"
base="https://github.com/vcarus/volto/releases/download/v${version}"

curl -fsSLO "$base/$name.tar.gz"
curl -fsSLO "$base/SHA256SUMS"
sha256sum --ignore-missing -c SHA256SUMS

tar xzf "$name.tar.gz"
cd "$name"
sudo script/install-selfsigned.sh --binary ./volto

--ignore-missing is what lets that check pass at all: SHA256SUMS lists the archive for both architectures and you downloaded one, so without it the other is reported as FAILED open or read and sha256sum exits 1. It still fails, as it should, when the archive it can read does not match.

From a checkout instead; --binary is unnecessary there, because its default is exactly what the build produces:

cargo build --release --locked
sudo script/install-selfsigned.sh

The installer creates the volto system user, installs the binary, generates an EC P-256 certificate valid for ten years, derives /etc/volto/config.toml from the shipped example with a random password, installs and starts the systemd unit, opens the port in ufw if it is active, and prints the fingerprint, the expiry date and a pasteable Surge policy line. It asks for the certificate name when neither --sni nor $SNI is set and a terminal is attached; everything else has a default:

sudo script/install-selfsigned.sh \
  --sni volto.internal \
  --port 443 \
  --username yourname \
  --password 'or let it generate one'

The binary it installs is ./target/release/volto unless -b, --binary PATH names another; that is how deploy.sh points a first install at the binary it unpacked from a release tarball rather than at a local build. Each option that takes a value can come from the environment instead — BINARY, SNI, PORT, USERNAME, PASSWORD — and --help lists the full set.

A username may not contain a colon (RFC 7617) and may not be longer than 32 bytes (see configuration.md), and neither a username nor a password may contain ", \, | or &: the first two cannot be written into the generated TOML string, and the other two are metacharacters of the substitution that writes it. The installer refuses them up front rather than installing something other than what was asked for. Everything else printable is accepted, * and . included, and a generated password never runs into this.

Before the generated config.toml is installed, the binary being installed is asked whether it can load it — the same volto --check-config question deploy.sh puts to a release on the update path. A no ends the run with the binary’s own message and nothing written, so the cause can be fixed and the installer re-run on a host it left untouched. A binary too old to know the flag is not asked, and the install goes ahead as it always did.

Re-running is safe: an existing config file, certificate or user is kept. --force regenerates the certificate only — it never rewrites config.toml, so hand edits survive. Regenerating changes the fingerprint, and every client then has to be updated.

Generating the certificate by hand instead:

sudo openssl req -x509 -newkey ec -pkeyopt ec_paramgen_curve:prime256v1 \
  -keyout /etc/volto/key.pem -out /etc/volto/cert.pem \
  -days 3650 -nodes \
  -subj "/CN=volto.internal" \
  -addext "subjectAltName=DNS:volto.internal" \
  -addext "basicConstraints=critical,CA:FALSE" \
  -addext "keyUsage=critical,digitalSignature,keyEncipherment" \
  -addext "extendedKeyUsage=serverAuth"

sudo chown volto:volto /etc/volto/cert.pem /etc/volto/key.pem
sudo chmod 0644 /etc/volto/cert.pem
sudo chmod 0640 /etc/volto/key.pem

openssl x509 -in /etc/volto/cert.pem -noout -fingerprint -sha256   # for the client
openssl x509 -in /etc/volto/cert.pem -noout -enddate               # note the expiry

The SAN is what gets matched — a bare CN has not been accepted for years — so it must be present even though the name is fictional. The name need not resolve: the client connects to an address and uses this only as the SNI and the name to match.

What pinning changes, and why each point matters:

• The fingerprint is the trust anchor. Generate it on the server, and carry it to each client yourself. If it arrives over a channel someone else can rewrite, pinning buys nothing — an attacker supplies their own fingerprint.
• The private key never leaves the machine. Not into backups, not into chat.
• There is no revocation. If the key may have leaked, the only remedy is regenerating (--force) and updating the fingerprint on every client by hand.
• Track the expiry date yourself. A pinning client usually skips validity and hostname checks, so an expired certificate may fail silently and only surface after a client update. The installer prints the date; put it in a calendar.
• Never use skip-cert-verify instead of pinning. It disables server identity entirely, and Basic credentials are sent on every request — a man in the middle collects the username and password on first use.

Deploying from releases

script/deploy.sh turns the release page into the deployment mechanism: it resolves the newest release (or the one given with --tag), downloads the tarball for the host’s architecture, verifies it against the release’s SHA256SUMS, and then does whichever of three things the host needs:

• No install yet (/etc/volto/config.toml absent): it runs the bundled install-selfsigned.sh for the full first-time setup, so the same flags and environment variables apply (--sni, --port, --username, --password).
• Older version installed: it asks the release it is about to install whether it can load this host’s /etc/volto/config.toml and stops if it cannot (see before the swap); otherwise it keeps the current binary at /usr/local/bin/volto.prev and the current unit at /etc/systemd/system/volto.service.prev, swaps in the new one, refreshes the systemd unit and restarts. If the service is not running a few seconds later, both are restored, systemd is reloaded, the service is restarted, and the script fails loudly. The unit is covered because it is refreshed on every update and carries the hardening directives an older host is most likely to reject.
• Already on that version, with the config and the unit in place: it exits without touching anything. The presence checks are part of the deal — deleting /etc/volto/config.toml and re-running is the supported way to regenerate it (the certificate and the system user survive, so the fingerprint does not change). What decides this is the version string and nothing else, so a binary that reports the wanted version is left in place whatever its bytes are; --force below is how it gets replaced.

--dry-run prints which of the three a run would pick and stops before doing anything about it. It needs an explicit --tag, since resolving “latest” is a network call, and it is how tests/it_deploy.rs exercises the decision without root, systemd or a download.

--force takes the download, verify, install and restart path even when the installed version already matches and the config and unit are both in place. It needs --tag for the same reason --dry-run does: a reinstall is about the bytes of one named release. Nothing else about the path changes, the rollback guard included, and neither /etc/volto/config.toml nor the certificate is rewritten. A dry run reports the same decision as dry-run: would reinstall v0.4.7. The case it is for is a release deleted and published again under the same tag: the convergence check compares version strings, so a host that already reports that version keeps the bytes of the first upload until a run is forced.

The script is also its own bootstrap. On a bare host, pipe it straight from the repository and let it do the downloading — everything it installs comes out of the checksum-verified release tarball, the piped copy only steers. With stdin being a pipe it never prompts, so give --sni (and friends) explicitly or accept their defaults:

curl -fsSL https://raw.githubusercontent.com/vcarus/volto/main/script/deploy.sh |
  sudo bash -s -- --enable-timer --sni volto.internal --port 443 \
       --username yourname --password 'or let it generate one'

Omitting --password is also fine, and usually better: the installer generates 18 random bytes — 144 bits, 24 characters — and prints the result in the final Surge policy line. --username is independent, so naming the user still leaves the password generated.

The no-op path is what makes it safe to run on a schedule:

sudo script/deploy.sh --enable-timer

installs the script as /usr/local/sbin/volto-deploy plus a systemd timer that re-runs it daily (OnCalendar=daily, randomized by up to an hour, Persistent= so a powered-off day is caught up). Every later deploy refreshes the installed copy of the script from the release it just verified, so the timer keeps pace with the repository. journalctl -u volto-deploy.service shows what each run did; a failed update leaves the timer unit in a failed state, which is the signal to go look.

Before the swap

On every update, and on a rollback above all, the script asks the binary it is about to install whether it can load the configuration this host already has:

volto --check-config --config /etc/volto/config.toml

If the answer is no, nothing is installed, the running service is not touched, and the run fails with the candidate’s own message — file, line and column — on stderr. Asking before the swap is the only time the answer is worth having: afterwards the service is already down, and the guard below has restored the release you were trying to leave.

Whether the candidate can be asked at all is decided by looking for the flag in its own --help, not by running it and reading how it fails, so a release from before the flag existed is never mistaken for a bad configuration. Such a release simply goes unchecked — which means a rollback past the release that introduced --check-config is exactly as unguarded as it always was, and gets the advisory below instead.

Rolling back

Rolling back is the same flow pinned to an older release:

sudo volto-deploy --tag v0.1.0

Add --force when the host already reports the version being pinned to. The convergence check compares version strings, so without it that run is a no-op and the binary on the host stays where it is.

Before pinning anything older than v0.4.5, comment mtu_upper_bound out of /etc/volto/config.toml: every install carries that key, no earlier release knows it, and a volto refuses the whole file rather than the one key it cannot read.

It is also the one flow nobody rehearses, so the script says the two things that bite on the way back before it does anything about them.

The config file goes back with nothing. The script never rewrites /etc/volto/config.toml or the certificate — both belong to install-selfsigned.sh’s first run and to you afterwards — and an older volto refuses a key it does not know, refusing the whole file rather than the key, so the service does not start at all. mtu_upper_bound is the one that bites in practice: it reached the shipped example in v0.4.5, and every install is derived from that example. The startup error names the file, the line and the column; comment that key out and start the service, or comment out everything introduced after the target release beforehand. See version compatibility for which key arrived when. On a rollback the check above catches this first, when the target release is new enough to be asked.

The script’s own guardrail does not help here, and reads backwards if you are not expecting it: when the newly installed binary is not running a few seconds later, the previous binary and the previous unit are both restored, which on a rollback is the release you were trying to leave.

--tag is not a pin. The script carries no version pin of its own; it converges on whatever the newest published release is, in either direction. So a rollback left alone is undone by the next timer tick, within a day. Hold it with

sudo systemctl disable --now volto-deploy.timer

or, if the release itself is the problem for every host, delete it from the releases page — that rolls every host back on its own next tick and needs no per-host action at all.

systemd

The shipped unit is script/masque.service. Manual installation:

sudo useradd --system --no-create-home --shell /usr/sbin/nologin volto
sudo install -m 0755 target/release/volto /usr/local/bin/volto
sudo install -d -o volto -g volto -m 0750 /etc/volto
sudo install -o volto -g volto -m 0640 script/config.example.toml /etc/volto/config.toml
sudo install -m 0644 script/masque.service /etc/systemd/system/volto.service
sudo systemctl daemon-reload
sudo systemctl enable volto

enable without --now on purpose: the service is not started yet. Then edit /etc/volto/config.toml: set the cert/key paths and set [auth].users. An empty user list means no authentication at all, and the password the example ships is a placeholder published in this repository and in every release tarball, so a server started before this edit accepts a credential anybody can read. The server warns about both at startup and refuses neither, so the order of these two steps is what keeps the port shut until the file is right.

Start it once that edit is made:

sudo systemctl start volto

The unit runs as a fixed system user rather than with DynamicUser=yes on purpose: the private key must be readable by this service and nothing else, which needs a stable owner to grant it to (chown volto:volto, mode 0640). AmbientCapabilities=CAP_NET_BIND_SERVICE is what allows binding a low port without root, and the rest of the unit is standard systemd hardening — ProtectSystem=strict with ReadOnlyPaths=/etc/volto, a @system-service syscall filter, no new privileges.

RUST_LOG overrides the configured log level without editing the config:

# /etc/systemd/system/volto.service.d/debug.conf
[Service]
Environment=RUST_LOG=volto=debug,quinn=info

Log lines carry a syslog priority when systemd is reading them, so journald’s own severity filter works rather than needing a text search:

journalctl -u volto -p warning --since -24h

Every line at info or above also carries a log_id field: eight lowercase letters and digits naming that one statement, fixed for the life of the line even if its wording changes. A runbook or an alert should match on the id rather than on the words, which a later release may rewrite. This selects the warning a connection that ended badly writes, and nothing else:

journalctl -u volto --since -24h | grep log_id=9pds6tk6

See [log] for the rest of the line format.

When a problem needs reporting rather than reading, volto --diagnostics --config /etc/volto/config.toml prints the version, the parsed configuration with passwords redacted, descriptor limits, the UDP buffer sysctls and the kernel, in one paste. The two RLIMIT_NOFILE lines are those of the process that ran the command, so from an SSH shell they are the shell’s and not the service’s. The bundle says so on the line under them, and on Linux it prints the service’s own Max open files values as well, read from /proc/<pid>/limits of every running volto process it finds. To read that by hand, systemctl show -p MainPID volto gives the pid. See collecting a support bundle.

Firewall

QUIC is UDP. This is the single most common reason for “it works locally but the client cannot connect”:

sudo ufw allow 443/udp       # only if ufw is installed and active
sudo ss -lunp | grep 443     # confirm volto is actually listening on UDP

A stock Ubuntu image has no ufw, so that first line is a command not found rather than a missing rule — the installer says as much in its own words, “no active ufw detected — open UDP 443 yourself if a firewall is in the way”. Open the port wherever this host actually filters, if it filters at all.

A cloud provider’s security group needs the same rule, on UDP.

File-descriptor budget

Each tunnel — TCP or UDP — costs one descriptor, and one client multiplexes many onto a single QUIC connection. The quota is per connection, so the number the process has to have descriptors for is the product, plus a fixed 64 for the listening socket, the request streams, stdio and the certificate a SIGHUP re-reads:

limits.max_connections × limits.max_targets_per_conn + 64

The shipped defaults make that 256 × 256 + 64 = 65600, and the shipped unit sets LimitNOFILE=131072, so a stock install has room over. The drop-in below is for operators who raise either limit past that point: at max_connections = 512 the number needed is 512 × 256 + 64 = 131136, past what the unit grants, and clients at their quotas can then consume every descriptor the process has, leaving none for the listening socket, for the certificate a SIGHUP re-reads, or for anything else. Fd exhaustion is not a crash here — a tunnel whose socket() fails is refused with a 500 and Proxy-Status: volto; error=proxy_internal_error, one request at a time, and the tunnels already running are untouched — but it is a degradation that hits every connection at once. That error type is what distinguishes it from an unreachable destination: it is RFC 9209’s “internal error unrelated to the origin”, so a burst of them in a client’s logs points at this host’s descriptor budget rather than at the targets, and it carries no next-hop because nothing was contacted. The same answer covers the other ways a host can run out — no kernel buffer, no ephemeral port left to bind.

volto compares that number against RLIMIT_NOFILE at startup and warns when it does not fit, which is worth heeding rather than silencing. Take the headroom in whichever direction suits the host: raise LimitNOFILE, or lower limits.max_connections, which is also the knob that bounds memory.

# /etc/systemd/system/volto.service.d/nofile.conf
[Service]
LimitNOFILE=262144

UDP socket buffers

On a high-bandwidth path the kernel’s UDP socket buffers are where packets are dropped first, and two different sysctls decide how big they are. Only one of them is a ceiling:

• net.core.rmem_default / wmem_default is what a socket gets when the application never asks — about 208 KiB on a stock Linux.
• net.core.rmem_max / wmem_max is the most an application may request. Raising it does nothing at all for a program that does not ask.

volto asks. limits.socket_recv_buffer and limits.socket_send_buffer are requested when the socket is created — DEFAULT_SOCKET_RECV_BUFFER = 2 MiB and DEFAULT_SOCKET_SEND_BUFFER = 2 MiB by default — so on this server the ceiling is the sysctl that matters:

sudo sysctl -w net.core.rmem_max=4194304
sudo sysctl -w net.core.wmem_max=4194304

Sized so the default request fits with room to spare; put the same lines in /etc/sysctl.d/ to survive a reboot. When the kernel caps the request instead, volto warns at startup and names the sysctl to raise, so a host that never had these touched says so in its own log rather than dropping packets quietly.

Two readings that look wrong and are not. Linux reports a granted buffer as double the size — the accounting includes per-packet overhead — so a satisfied 2 MiB request shows as rb 4194304 in ss -uanpm, and that same number is what the startup line prints as so_rcvbuf. And both keys take effect only when the socket is created: a reload does not rebind it, so changing them needs a restart. 0 hands the size back to the operating system.

Reloading

systemctl reload volto sends SIGHUP. volto re-reads the configuration file and applies it to connections accepted from then on: a renewed certificate, a changed user list, a raised or lowered limits.max_connections, changed transport parameters. Established connections keep the configuration they were accepted with — a tunnel’s rules must not change mid-transfer, and QUIC cannot renegotiate transport parameters anyway.

That matters when the change is meant to revoke access: a client that still holds an established connection keeps working on the old credentials, and keep-alives routinely hold one open across long idle periods. Use systemctl restart volto rather than reload when the old credentials must stop working — a restart closes those connections after the shutdown grace period (see Graceful shutdown) instead of leaving them running.

A reload is all-or-nothing. Parsing, validation and certificate loading all happen before anything is swapped in, so there is no state where a new certificate is paired with an old user list. If the file is broken, volto logs the error and keeps running on the previous configuration; it never exits. That property is the point: the process sending this signal is usually a renewal hook running unattended, and a typo must not become an outage.

Graceful shutdown

SIGTERM stops the endpoint from accepting new connections, sends GOAWAY on the established ones and waits for their tunnels to finish, up to server.shutdown_grace (default 5 s). The default is short on purpose: a client that keeps using a connection after GOAWAY instead of opening a new one — Surge does — has every new request fail until the drain ends, so a long grace period trades a longer outage for every new request against finishing the transfers already in flight. Raise it if long transfers matter more to you than a few seconds of failed requests at each restart. Keep systemd’s TimeoutStopSec comfortably above whatever you choose — the shipped unit uses 45 — so systemd does not send SIGKILL mid-drain. An hour is the most that can be configured: the grace period is the bound the drain is built around, and a value past that would only hand the ending back to SIGKILL.

The GOAWAY carries an identifier, and it is a promise in both directions. Requests below it were already accepted and are still served during the drain, tunnel and all, even if the client only finishes sending them after the signal. Requests at or past it are rejected with H3_REQUEST_REJECTED, which tells the client they were not processed and may be retried on another connection.

Sending the GOAWAY is itself bounded by limits.max_idle_timeout, because a peer decides when a write to it completes: one that grants no flow-control window on the control stream would otherwise hold that connection’s drain open for the whole grace period, and with it the process. A connection whose peer will not take the frame within that bound drains without one and is closed when its tunnels end, exactly as it would have been otherwise.

A SIGHUP that arrives once the drain has begun is refused and logged: the listener has been closed by then, and reopening it to accept handshakes that are seconds from being closed again would be worse than doing nothing. Reload before you stop, not during.

Running behind a UDP relay

volto needs no special configuration to sit behind a plain layer-4 UDP forwarder, because TLS terminates only at volto itself. The relay moves opaque UDP packets and holds no key material. Three things are worth knowing:

• Keep the relay’s UDP conntrack timeout above the keep-alive interval. On Linux nf_conntrack_udp_timeout defaults to 30 seconds. volto sends keep-alives every 20 seconds by default, which refreshes the mapping in time; if your relay expires entries faster, lower keep_alive_interval well under that value and lower max_idle_timeout with it (the keep-alive must stay below half the idle timeout, and volto refuses to start otherwise).

• Issue certificates with DNS-01 when the domain’s A record points at the relay — the other challenge types validate against that address and cannot reach volto; see ACME with DNS-01 above.

• Point the client at the relay’s address, and the certificate name at the domain. In Surge that is sni= plus server-cert-verify-name=:

  volto = masque, 203.0.113.10, 443, sni=example.com, server-cert-verify-name=example.com, username=user1, password=…
  

One consequence to plan for: every client then reaches volto from the relay’s address, so per-IP banning at the server cannot distinguish them — see fail2ban below.

A second one: the rule that refuses a tunnel to one of this host’s own addresses cannot see the relay’s, because no interface here carries it, so a target that names the relay is judged like any other public address. See [security].

fail2ban

A failed authentication logs one stable WARN line carrying the source address:

WARN ... authentication failed ... remote=203.0.113.7:5678 username="user1" reason="credentials rejected"

/etc/fail2ban/filter.d/volto.conf:

[Definition]
failregex = ^.*authentication failed.*remote=<HOST>:\d+.*$
ignoreregex =

/etc/fail2ban/jail.d/volto.conf:

[volto]
enabled  = true
backend  = systemd
journalmatch = _SYSTEMD_UNIT=volto.service
filter   = volto
maxretry = 10
findtime = 10m
bantime  = 1h
# QUIC is UDP: the action has to ban the UDP port.
action   = iptables[name=volto, port=443, protocol=udp]

Check first that the logged address actually distinguishes clients. Any NAT in the path rewrites it: behind a UDP relay every client appears with the relay’s address, and a server behind carrier-grade NAT can see all inbound traffic rewritten to one gateway address. Verify by connecting yourself and comparing remote= against your own public address. If they do not match, or if every connection shares one address, banning by IP bans everyone — drop fail2ban and rely on the connection-level limit instead, or run the ban on the machine closest to the internet that still sees real client addresses.

That connection-level limit needs no configuration and is unaffected by topology: after security.max_auth_failures failures (default 5) the whole QUIC connection is closed, so an attacker pays for a full QUIC and TLS handshake every N guesses. A failure is one credential value tried and refused rather than one refused request: a request may carry credentials under either of the two accepted field names, both are charged, and a request carrying more than two credential values is answered 400 without any of them being tried. So a guess costs a guess however many the attacker packs into one request.

Architecture

MASQUE is not one protocol but a set of IETF specifications that put proxy tunnels on top of HTTP/3. A classic HTTP proxy gives you one TCP tunnel per connection; here one QUIC connection multiplexes arbitrarily many TCP and UDP tunnels, and looks like ordinary HTTP/3 traffic from the outside.

CONNECT-UDP (RFC 9298)          UDP proxying semantics
classic CONNECT (RFC 9114 §4.4) TCP proxying semantics
HTTP Datagrams + Capsules (RFC 9297)
Extended CONNECT (RFC 9220)
HTTP/3 (RFC 9114) + QPACK (RFC 9204)
QUIC (RFC 9000) + DATAGRAM frames (RFC 9221)
TLS 1.3 (RFC 8446 / RFC 9001) over UDP

RFC 9298 stands on 9297 and 9220; 9297 stands on 9221. The TCP path needs none of them — only RFC 9114 itself.

The path of a request

quic.rs owns the quinn endpoint. Transport parameters come from [limits], so the stream cap, idle timeout, keep-alive, MTU settings, congestion controller and initial RTT are configuration rather than constants; they apply to connections accepted from then on, including across a SIGHUP reload. The stream cap is the one of them that does not reach the handshake as itself; what an unauthenticated connection is worth is further down this section.

The UDP socket underneath is bound here rather than by quinn::Endpoint::server, which is a bare std::net::UdpSocket::bind and never sets SO_RCVBUF/SO_SNDBUF — a server that does not ask keeps net.core.rmem_default whatever rmem_max says. limits.socket_recv_buffer / socket_send_buffer are requested at that moment and the granted sizes read back, so a capped request becomes a startup warning rather than silent packet loss. Being a property of the socket rather than of a connection, these two are the only [limits] keys a reload cannot change.

The SNI gate

gate.rs sits between that socket and quinn. It is an AsyncUdpSocket wrapper, so it sees every datagram before the QUIC layer does, and it is off unless [security] expected_sni names at least one host — an empty list, the shipped default, passes everything through untouched.

It exists because quinn’s endpoint is conservative but not silent. A long-header packet naming a version it does not speak draws a Version Negotiation packet; an Initial whose Destination Connection ID is shorter than the eight bytes RFC 9000 §7.2 requires draws a CONNECTION_CLOSE; a short-header packet for no live connection can draw a Stateless Reset. All three are correct answers to a QUIC peer, and all three are answers to a port scan. With the gate open, a datagram that would draw one is discarded first.

The judgement is made on the first packet in the datagram, which is enough for the whole of it: RFC 9000 §12.2 both lets a receiver “route based on the information in the first packet contained in a UDP datagram” and forbids a sender from coalescing packets with different connection IDs into one. A short header passes — the gate holds no connection state to judge it with. A long header naming a version other than 1 is refused. A long header that is not an Initial passes, because quinn drops those for an unknown connection without a word. An Initial in a datagram below 1200 bytes passes for the same reason: RFC 9000 §14.1 has the server discard it, and quinn does, silently. What is left is an Initial that could start a connection. Its Destination Connection ID is judged first, on the header alone and before any decryption: RFC 9000 §7.2 gives a client’s first Initial a Destination Connection ID of at least eight bytes, every later Initial of an admitted handshake is addressed by the eight bytes this endpoint chose, and so is the one a Retry supplies, so nothing legitimate is shorter. quinn applies the same floor ahead of its own decryption, in early_validate_first_packet. A shorter one is refused here, which is the CONNECTION_CLOSE above never sent. The packet is then opened: the keys for it are derived from the Destination Connection ID in the packet itself (RFC 9001 §5.2), so no connection state is needed to read it. Its CRYPTO frames are assembled from offset zero, the ClientHello is parsed, and the server_name extension (RFC 6066 §3) is compared with the configured list — ASCII case-insensitively, with a trailing root dot ignored, name for name and no wildcards.

A refused datagram is not removed from the receive batch, because one buffer can hold a whole GRO run described by a single stride. It is left in place with its Destination Connection ID length byte overwritten by a value larger than the 20 bytes RFC 9000 §17.2 allows, which quinn’s own packet decoder rejects before it reads the version, the type or anything else — and rejects by returning rather than by answering.

Two deviations from a SHOULD are deliberate and are the point of the feature. RFC 9000 §5.2.2 says a server “SHOULD send a Version Negotiation packet” for a large enough packet naming an unsupported version; with the gate open it sends nothing. RFC 6066 §3 says a server that does not recognise the name “SHOULD take one of two actions: either abort the handshake by sending a fatal-level unrecognized_name(112) alert or continue the handshake”; the socket-level check does neither, and the second check below aborts with a different alert.

That second check is in tls.rs. The gate reads only what arrives in the first Initial packet, and a ClientHello large enough to be split across several of them is passed through on purpose — refusing a first flight whose extensions have not arrived would make a large ClientHello unreachable. Such a handshake is stopped by a ResolvesServerCert that declines to present a certificate for a name that is not on the list, which rustls turns into a fatal access_denied alert. It is not a certificate selector: there is still one certificate and one key, and naming several hosts means answering to all of them with the same certificate.

Stateless resets are deliberately left uncovered — see D106 and the configuration reference for what that leaves visible and why filtering on the address instead would break connection migration. So is the acknowledgement quinn returns for an Initial the gate passes that carries an ack-eliciting frame and no name — a PING on its own, or a ClientHello the gate could not finish reading — because refusing those means refusing a large first flight, which this design will not do. None of this is traffic obfuscation, which stays a non-goal: the gate hides that a service is here from somebody who does not know the name to ask for, and says nothing about what a connection looks like once one is open.

Each accepted connection is handed to h3api::Connection::handshake, which must advertise both SETTINGS_ENABLE_CONNECT_PROTOCOL (0x08) and SETTINGS_H3_DATAGRAM (0x33). Surge checks for both and disconnects if either is missing; this is the first thing to suspect when a client drops immediately after a successful TLS handshake. The frame also carries SETTINGS_MAX_FIELD_SECTION_SIZE (MAX_FIELD_SECTION_SIZE = 64 KiB), both QPACK settings as zero, and one reserved “grease” identifier of RFC 9114 §7.2.4.1’s 0x1f * N + 0x21 form, which that section says endpoints SHOULD send so peers keep exercising the rule that unknown identifiers are ignored.

conn.rs runs the accept loop and dispatches each request stream on the :protocol pseudo-header, after authenticating it:

• absent, method CONNECT → tunnel/tcp.rs
• connect-udp → tunnel/udp.rs
• anything else → 501

Authentication deliberately runs before routing, so an unauthenticated client gets 407 rather than learning from a 501 which :protocol values this server implements — but after the connection-specific-field check described below, so a peer that sends one of those fields gets 400 whether or not it has credentials.

That 501 is a real one for every :protocol value, not only the ones this server has heard of. The token is carried through as the bytes that arrived, so connect-ip, webtransport and anything else are answered with the status RFC 9220 asks for and logged under the name the client actually sent, rather than being refused as malformed before anything can look at them.

The routing above and the per-connection context both live in tunnel/mod.rs, beside the target resolution the two tunnel kinds share. Two concerns that are neither are files of their own: tunnel/status.rs holds every refusal and the RFC 9209 Proxy-Status vocabulary refusals are written in, and tunnel/quota.rs the per-connection tunnel budget and the guards that spend it.

Every wait that stands between a packet arriving and a tunnel opening is bounded, whatever the connection’s authentication state, and two limits share the job. All but two of those waits are bounded by limits.max_idle_timeout: the QUIC/TLS handshake in quic.rs, the HTTP/3 handshake and the read of each peer unidirectional stream’s type in h3/connection.rs, the read of a request stream’s HEADERS in Resolver::resolve, and every response this server writes before a tunnel starts relaying — 400, 403, 407, 431, 501 and the 5xx family, the 200 that closes on the spot, and the 200 that opens a tunnel — in Stream::respond_within. The two exceptions are resolving the target’s name and connecting to it, which draw on limits.connect_timeout instead: the lookup gets that budget once, the connect gets it once for the whole list of addresses rather than per address, and 0 hands both waits back to the operating system. The idle-timeout bound is needed because the keep-alive PINGs this server sends are answered by the peer’s QUIC stack with no application ever involved, so the transport’s own idle timer never fires on a peer that is present but says nothing; the response writes need it because a peer that grants no flow-control window never takes even the fifty-odd bytes of a 407. A tunnel’s own 200 is bounded like the rest: the pumps that would otherwise end the wait do not exist until that write returns, and a lapsed one resets the stream and drops the target socket the request had already opened, with a reset rather than a FIN.

One further bound applies until a request on a connection has passed the credentials check: twice limits.max_idle_timeout for one request to authenticate. It is armed once at the handshake and never re-armed by a new stream, so what it bounds is the connection rather than a pause in it; twice, so that the transport’s own idle timeout stays the first thing to fire on a peer that has simply gone away. The clock is read on each pass as well as awaited, because awaiting it alone is not enough: a timeout only reports a lapsed deadline on a poll where the work underneath it was not already finished, so a peer with another request stream always queued — one opened and finished empty costs it nothing and the server nothing but a stream error — was never measured against the deadline at all. Without it a peer that finishes the QUIC handshake and then says nothing holds a max_connections slot for as long as it keeps its socket open. It is lifted for the life of the connection the moment one request authenticates, so a client reusing an idle connection between requests is untouched. A lapsed connection bound closes the connection with H3_NO_ERROR, which is not an error and is logged as the idle ending it is; a lapsed HEADERS bound resets that one stream with H3_REQUEST_INCOMPLETE and a lapsed response write resets it with H3_REQUEST_CANCELLED, leaving everything else on the connection running.

That bound is on time. The second one on an unauthenticated connection is on size, and it is a transport parameter: the handshake advertises INITIAL_BIDI_STREAMS = 16 concurrent bidirectional streams rather than the configured limits.max_streams_bidi, and the first request to pass the credentials check raises the connection to the configured value in one step. Before it, a peer that has proved nothing holds at most 16 request streams — 16 parked request tasks and 16 refusals to write, where 1024 of each were free — and its whole HEADERS buffering is bounded twice over by the same number, since 16 field sections at the MAX_FIELD_SECTION_SIZE = 64 KiB per-frame cap are exactly the HEADERS_BUFFER_BUDGET = 1 MiB budget below. Running into the allowance is backpressure rather than an error: the peer’s own stack holds the stream, and closing streams returns credit before authentication as after it. A client that fires more than 16 CONNECTs at once therefore waits for the raise, which happens before its first request has resolved a name or opened a socket, so what it waits is the round trip it was already waiting for that first answer. An operator who configures fewer than 16 gets what they configured throughout: the clamp only ever lowers.

That bound holds one connection; it does not bound how many of the limits.max_connections slots unauthenticated peers hold between them, and a peer that completes a handshake about once a second and never sends a credential would fill every one of them legitimately, each slot bounded, all of them replaced. So at the cap a new connection does not simply lose: the endpoint takes the slot of the oldest connection that has never authenticated and admits the newcomer, and only refuses when every live connection has authenticated. The evicted connection is closed with H3_NO_ERROR — nothing about it failed — and logged like the other idle endings, with reason=evicted; one whose QUIC handshake had not finished has already been accepted at the transport layer, so what its peer receives is a CONNECTION_CLOSE carrying APPLICATION_ERROR and an empty reason rather than the CONNECTION_REFUSED a refusal sends. A connection that has authenticated is never a candidate, and below the cap nothing of this runs. There is no sub-quota for unauthenticated connections because it would not help: a legitimate client is unauthenticated at accept time too, so it would be squeezed by exactly the pool it is trying to join.

Evicting on the word of an unverified source address would be worse than the problem: a spoofed Initial costs an attacker one datagram and no return path, so a flood of them would empty the roster and start a TLS server flight per packet. At the cap, a newcomer whose address QUIC has not yet validated is therefore answered with a Retry (RFC 9000 §8.1) — it takes no slot and no crypto — and only the newcomer that comes back with the token may evict anybody. The extra round trip is charged to an address the server holds no token from: quinn’s bloom feature is on, so the server sends NEW_TOKEN frames on every connection it validates (RFC 9000 §8.1.3) and a client that kept one comes back already validated and evicts without a Retry — first contact, a token past its two-week lifetime, or a token sealed before the last restart or SIGHUP are what pay. Below the cap no Retry is sent at all and the handshake is what it always was.

Target address selection

Both tunnel kinds resolve their target through one function in tunnel/mod.rs, which is also where the resolved list is ordered by limits.ip_family_preference — before the destination policy filters it and before either tunnel dials anything, so the two cannot disagree about which family goes first. The default is IPv4-first, deliberately unlike getaddrinfo, which orders by RFC 6724 and so puts global IPv6 ahead of IPv4 on any host with an IPv6 route. That ordering is an operator policy on a proxy rather than a resolver detail: the TCP path walks the list in order, so the non-preferred family costs a full connect attempt, and the CONNECT-UDP path has no failover at all — connecting a UDP socket only asks the kernel for a route, so the first address with one wins outright. The reorder is a stable partition, leaving RFC 6724’s ordering within each family intact; system opts out of it entirely.

The name lookup budget

Name resolution is the one place in this server where a request reaches a resource it cannot give back on demand. tokio::net::lookup_host runs getaddrinfo on the runtime’s blocking thread pool, and that call cannot be cancelled: when limits.connect_timeout expires the client is answered, but the thread stays in the resolver until the stub gives up on its own — five seconds per attempt on a stock resolv.conf, tens of seconds when the nameserver answers nothing at all. The pool is process-wide, so without a bound one client’s black-holed names park threads that every other connection’s lookups need, including names that would have resolved from /etc/hosts without a query.

So the pool is rationed in two tiers, in net.rs:

• One reserved lookup slot per connection, which nothing else can take. No number of hostile connections can starve a connection outright; at worst its lookups run one at a time — a lookup queued for the shared allowance keeps its claim on the reserved slot too, and takes it the moment it frees.
• A server-wide allowance on top of that, capped per connection, so ordinary use resolves a page’s worth of tunnels in parallel while no single connection can hold more than its share.

The permit is dropped by the blocking task rather than by the request that asked for it — a permit released when the client is answered would say a thread is free while it is still inside the resolver, which is exactly how a client could park the pool one connect_timeout at a time. An IP literal never reaches the resolver and never takes a permit.

The blocking pool is sized from limits.max_connections at startup for the same reason: it has a thread for every slot the budget can hand out, plus headroom. Threads are created on demand and reaped when idle, so this is a ceiling rather than an allocation. Because it is fixed when the process starts, a SIGHUP that raises max_connections does not resize it.

TCP tunnels

tunnel/tcp.rs resolves :authority explicitly — not implicitly through TcpStream::connect((host, port)), because the destination policy has to see the resolved address — answers 200, and then runs two independent pumps with full RFC 9114 §4.4 half-close semantics:

Collapsing this into “one side closes, everything closes” breaks every protocol that depends on half-close. The abnormal paths coordinate through a sticky watch teardown channel so that neither pump can strand the other.

While both directions are live, each pump is the other’s watchdog and nothing in a tunnel is on a timer: whatever one pump notices — a reset stream, a target that fails — raises a teardown, and that teardown is what ends the other’s wait. The first two rows above are the endings that take the watchdog away, because they end a direction without a teardown. So from the moment one direction has ended cleanly, each write in the surviving direction is bounded by [limits] udp_session_timeout — the same knob a CONNECT-UDP session’s idle bound comes from — and a write that does not complete within one of those cuts the tunnel: the request stream is reset (H3_REQUEST_CANCELLED when it is the client that stopped reading after its own FIN, H3_CONNECT_ERROR when it is the target that stopped reading after its own EOF) and the target socket is closed abortively — though on the client-FIN path the proxy’s clean FIN has already reached the target, and a kernel that does not reset from FIN_WAIT_2 (Linux) closes silently behind it, so only the resources are reclaimed there. Every write gets its own budget, so a client that keeps reading may take hours to drain a download after its FIN — but progress alone does not save a write that never finishes, which puts a floor of roughly one RELAY_BLOCK_SIZE = 64 KiB relay chunk per budget under how slowly a half-closed download may be drained (about one 1.4 KB piece per budget in the other direction). A tunnel whose two directions are both still open is never on a timer at all, however long a write parks.

The bound covers the parked write half of the problem and only that. A half-closed tunnel whose surviving direction is parked in a read — no bytes in either direction — is deliberately not put on a timer, because a target that has taken the client’s FIN may legitimately take minutes to answer and cutting it would break the half-closes this proxy exists to carry; what limits those is capacity rather than time, max_connections × max_targets_per_conn sockets. Without the write bound, a half-closed tunnel whose surviving peer stopped taking bytes held the target socket, its file descriptor and the tunnel slot until the QUIC connection ended — which this server’s own keep-alives can postpone indefinitely.

Not on a timer is not unwatched. Such a tunnel still ends the moment the client says it wants no more of it: the response direction watches for the client’s STOP_SENDING — the other half of what dropping a request stream sends — even while there is nothing to write, so a client that half-closed and then walked away releases the target socket and its tunnel slot at once rather than at the end of the QUIC connection. Only a client that is still waiting for an answer keeps a quiet half-closed tunnel alive.

The last row cancels both directions because RFC 9114 §4.4 asks for it: “If the stream is reset or reading is aborted by the client, a proxy SHOULD perform the same operation on the other direction in order to ensure that both directions of the stream are cancelled.” Without it a client that reset only its sending side had the response direction finished with a clean FIN, so a truncated response read as a complete one, and a client that sent STOP_SENDING had its own sending side stopped with code 0 rather than with an HTTP/3 code.

UDP tunnels

tunnel/udp.rs parses the RFC 9298 well-known path template. Surge has no URI template parameter, so it uses the default template the RFC defines for exactly that case:

https://$PROXY_HOST:$PROXY_PORT/.well-known/masque/udp/{target_host}/{target_port}/

Parsing is deliberately lenient: percent-decoding, an optional trailing slash, a port in 1..=65535 and an empty query. IPv6 literals arrive without brackets, with the colons escaped (2001:db8::42 → 2001%3Adb8%3A%3A42) per RFC 9298 §3; the bracketed form is also accepted.

Two ordering rules that are easy to get wrong:

• Answer 2xx immediately — do not wait for the target to send anything. UDP is connectionless and the proxy cannot know whether the target is reachable.
• But resolve DNS first. When target_host is a name, resolution MUST complete before the response; a failure is refused with 502 and Proxy-Status: volto; error=dns_error (RFC 9209). “Immediate” means “without waiting for the target”, not “without resolving”.

The target socket is connected, so packets from anywhere else are dropped by the kernel. Sessions have their own idle timeout (default DEFAULT_UDP_SESSION_TIMEOUT = 180 s; RFC 9298 §3.1 asks for at least 120), where idle means no packet crossed the proxy: the clock is re-armed by a payload reaching the target or the target answering, never by bytes alone, so a peer dripping fragments of a capsule that never completes is reclaimed on schedule. As on the connection bound above, that clock is read on each pass and not only awaited: a peer with another payload always queued — the ones the amplification cap drops re-arm nothing, and cost it only bandwidth — kept the session’s wait finishing on its own and the deadline was never measured against it. Closing the socket must also close the request stream.

Datagram routing

Each connection routes inbound datagrams to per-session channels from its own background task, in src/h3/connection.rs — the same task that reads the peer’s unidirectional streams, since both belong to the connection and both end with it. A peer may hold MAX_PEER_UNI_STREAMS = 16 of those open at once (a transport parameter rather than a configuration key) where HTTP/3 needs three, and each one’s type has to arrive within an idle timeout. Routing is per request stream, which is why it lives in the HTTP/3 layer: a session claims its Quarter Stream ID by asking its stream for a DatagramReceiver and holds the claim exactly as long as it holds that receiver, so a session that ends — however it ends — takes its routing entry with it. Sending is the other half and stays outside: a UDP session writes its datagrams straight onto the quinn::Connection.

Two fields decide where a packet goes, and getting either subtly wrong is silent:

• Quarter Stream ID = stream ID ÷ 4 (RFC 9297 §2.1), not the stream ID. Client-initiated bidirectional stream IDs are always multiples of four, so the low two bits carry no information and dividing saves encoding space. The old draft term “Flow Identifier” is gone, and there is no separate mapping table. A value no session owns is dropped, since a session can close with packets still in flight — but a value that cannot be a stream ID at all (above 2⁶⁰-1, or a datagram too short to parse one) closes the whole QUIC connection with H3_DATAGRAM_ERROR (0x33), which RFC 9297 §2.1 states as a MUST.
• Context ID, a varint at the head of the payload (RFC 9298 §5). 0 means a raw UDP payload; anything else is an extension. An unknown Context ID is dropped silently — never a connection error — and so is a truncated one, which no requirement covers.

Every such drop — an unknown Context ID, a truncated one, an unclaimed Quarter Stream ID, or a session’s inbound queue already full — is counted, and the total is reported as dropped_datagrams= on the connection’s closing log line, since the drops themselves are required to be silent.

Get either wrong and the symptom is “the handshake succeeds, the tunnel is established, and not one packet gets through”, with no error anywhere. The encoder and decoder are hand-rolled in src/datagram.rs — about thirty lines, which is why no dependency is carried for them.

Routing uses a bounded channel and try_send: a session that is not draining its queue loses packets, which is correct UDP behaviour, instead of blocking the routing task and starving every other session on the connection. A CONNECT (TCP) tunnel never claims a receiver, so it never appears in the table at all.

The capsule stream

The request stream of a CONNECT-UDP tunnel is not an empty stream. Its body is a sequence of capsules (RFC 9297 §3), each Type varint + Length varint + Value, handled by src/capsule.rs as an incremental TLV decoder:

• Requests and responses carry Capsule-Protocol: ?1; a response using capsules must not carry Content-Length, Content-Type or Transfer-Encoding.
• Unknown capsule types are skipped by reading the length and discarding the value edge-to-edge, with no accumulation — a peer declaring a 2⁶²-byte capsule costs no memory. A truncated capsule is a malformed message.
• The DATAGRAM capsule (type 0x00) is the reliable fallback channel for when QUIC datagrams are unavailable. It is a fallback, not a downgrade: a target packet too large for a QUIC datagram is dropped rather than re-routed through the capsule stream, which would break end-to-end unreliability and defeat the inner path-MTU discovery.

Treating the stream bytes as opaque is a worse trap than the Quarter Stream ID: the decoder desynchronizes on the first capsule that arrives.

RFC 9297 §2.1.1 also forbids sending QUIC datagrams before the peer has advertised SETTINGS_H3_DATAGRAM. The flag is one shared atomic rather than a value each session snapshots: the task that reads the peer’s control stream writes it and every session reads it once per packet, so a session opened before the SETTINGS arrived moves onto QUIC datagrams the moment they do. The module comment on src/h3/connection.rs records what the sampled version cost.

The HTTP/3 layer

HTTP/3 is implemented in src/h3 rather than taken from a crate. What it covers is what a CONNECT proxy needs, and it is stated in full:

• Framing (RFC 9114 §7) — an incremental decoder that hands DATA payload on in the chunks quinn delivered it in, buffers every other frame because none of them can be acted on piecewise, and skips an unknown frame type by its declared length without allocating for it. A frame’s type is judged before its length: the decoder knows which of the three kinds of stream it is reading — the peer’s control stream, a request stream, or a request stream whose CONNECT has been answered — and a frame type that may not appear there is a connection error at any size. The number of reserved or unknown frames a request stream may make the decoder skip before its request arrives is bounded too (RFC 9114 §10.5): grease and padding are legitimate (§7.2.8), but a flood of them — zero- length frames included, which no size bound would catch — is refused for H3_EXCESSIVE_LOAD. The limit is generous, per stream, and answered with a bare reset rather than a 431, since no field section was ever too large.
• QPACK (RFC 9204) against the static table, including Huffman decoding (RFC 7541 Appendix B).
• The control stream — SETTINGS, GOAWAY, and the rules about what may appear there, read by a task of its own for as long as the connection lasts. RFC 9114 §6.2.1 makes a control stream that is closed at any point a connection error, and that holds in both directions: a peer that finishes its own, and a peer that stops this endpoint’s with STOP_SENDING, are each answered with a CONNECTION_CLOSE carrying H3_CLOSED_CRITICAL_STREAM.
• Request validation — RFC 9114 §4.1.2, §4.3 and §4.4 plus RFC 8441 §4: the point at which a field section becomes a request or is refused as malformed.
• The message vocabulary — a status, a method, a request and a list of field lines, in src/h3/message.rs. These were the http crate’s until decision D78; carrying it meant every decoded field was validated and copied a second time into a HeaderName/HeaderValue, and the pseudo-headers were folded into a URI that the tunnels then took apart again. A field section is now decoded once, validated once, and carried in one type, and the crate is gone from the dependency tree.

What it leaves out, deliberately:

• The QPACK dynamic table. SETTINGS_QPACK_MAX_TABLE_CAPACITY and SETTINGS_QPACK_BLOCKED_STREAMS are both advertised as 0, which RFC 9204 §3.2.3 and §2.1.2 make binding on the peer’s encoder. A field line referencing a dynamic entry is therefore a protocol violation answered with QPACK_DECOMPRESSION_FAILED, not a gap in the implementation — and the eviction and head-of-line blocking that go with a dynamic table are gone with it. The peer’s encoder and decoder streams are still read, instruction by instruction: with no table, an insertion, a capacity above zero, a Section Acknowledgment or an Insert Count Increment is the connection error RFC 9204 §3.2.2, §4.3.1, §4.4.1 and §4.4.3 make it, and only a zero capacity and a Stream Cancellation pass.
• Huffman encoding. A response here is a status line and at most two short fields; compressing it would save a handful of bytes per tunnel. Decoding is implemented because a client’s request arrives Huffman-coded.
• Server push and WebTransport. A push stream from a client is a connection error either way (RFC 9114 §6.2.2), and webtransport is a :protocol this server answers 501 to. The push bookkeeping frames are still judged: every CANCEL_PUSH names a push this server never promised (RFC 9114 §7.2.3), and a MAX_PUSH_ID may not shrink (§7.2.7) — both H3_ID_ERROR.

A connection error is a quinn::Connection::close carrying the HTTP/3 code, which is exactly what RFC 9114 §8 defines one to be. Nothing has to be propagated between tasks: closing the connection makes every operation on it fail on its own, so only the reason is recorded on the way past — quinn overwrites its own stored reason with “closed locally” — and read back by the accept loop.

Which frame types may appear is decided first, and from the frame header alone. SETTINGS, GOAWAY, CANCEL_PUSH, MAX_PUSH_ID and PUSH_PROMISE on a request stream are a connection error of type H3_FRAME_UNEXPECTED whatever length they declare (RFC 9114 §7.2.3–§7.2.7, and §4.1 for GOAWAY, whose “wrong stream” rule §7.2.6 states for a client only). Once a CONNECT has been answered, RFC 9114 §4.4 narrows the stream to DATA and unknown types, so a HEADERS frame there is a connection error too. Ordering these ahead of the size checks is what keeps a 431 from being the answer to something that is not a field section, and what stops a frame the peer was never allowed to send from being charged for.

Field sections are bounded at MAX_FIELD_SECTION_SIZE = 64 KiB, advertised as SETTINGS_MAX_FIELD_SECTION_SIZE so a peer that respects SETTINGS never sends more, and enforced on receipt so one that does not gets no further. The frame layer refuses an oversized non-DATA frame from its declared length, before a byte is allocated for it.

That bound is per frame, and a peer may open as many streams as its transport parameters allow. What one connection may hold in frames it has announced but not finished is therefore bounded separately, at HEADERS_BUFFER_BUDGET = 1 MiB summed over its request streams: a frame is charged for the length it announced the moment the decoder commits to buffering it, and the charge is returned when the frame completes or when the stream carrying it ends. The request that would take a connection past that is refused with 431 and stopped with H3_EXCESSIVE_LOAD, which is the same answer a single oversized field section gets; the connection and its tunnels are untouched. A CONNECT request with credentials is a few hundred bytes, so every request stream the transport allows could be mid-HEADERS at once and still be using a tenth of the budget — what reaches it is sixteen field sections of the largest advertised size in flight together, and the seventeenth of those is what is refused. Sixteen streams at once is also the whole of what an unauthenticated connection may hold, so a peer that has proved nothing can meet this budget and never overshoot it: the 431 belongs to a connection past the credentials check, and against such a connection’s 1024 streams it is the only thing standing. Only frames that are legal where they arrived are counted, since the type verdict comes first: a peer cannot hold the budget with PUSH_PROMISE frames on request streams, nor with field sections announced on tunnels that may carry none. The peer’s control stream is not counted in it either: there is one of it per connection, it buffers one frame at a time, and nothing on it can be refused stream by stream.

One choice here is worth stating, though it is within the RFC: a malformed request is answered with a bare 400 and a clean stream close rather than a reset wherever the fault is visible at the tunnel layer – a CONNECT-UDP request carrying Content-Length or Content-Type (RFC 9297 §3.2), or any request carrying one of the connection-specific fields RFC 9114 §4.2 forbids (Proxy-Connection, Keep-Alive, Transfer-Encoding, Upgrade). RFC 9114 §4.1.2 lets a server “send an HTTP response indicating the error prior to closing or resetting the stream”, and it_tcp and it_udp pin the 400. The Connection field itself is refused at decode time with H3_MESSAGE_ERROR, earlier still. The four-field check runs before the credentials check rather than after it, unlike the routing it precedes: a message carrying one is malformed whoever sent it, and judging it afterwards answered that MUST with a 407 to an unauthenticated peer and a 400 to an authenticated one.

A peer that closes its QPACK encoder or decoder stream is treated as H3_CLOSED_CRITICAL_STREAM, the connection error RFC 9204 §4.2 requires, and both endings count: a clean finish and a reset alike. The one exemption is a connection that has already ended, where the closure is the connection going away rather than a fault the peer committed. That is the exemption the control stream makes for the same race.

HTTP Datagrams are hand-rolled in src/datagram.rs. That started as a way around h3-datagram 0.0.2, which tagged every datagram with Quarter Stream ID 0 and so misrouted every session after the first on a connection; it is simply ours now — about thirty lines of varint work with no dependency behind them.

Why quinn-proto is patched (temporary)

Cargo.toml carries a [patch.crates-io] stanza pointing quinn-proto at a commit on upstream’s 0.11.x branch. quinn-proto’s MTU black-hole detector treats an ordinary congestion loss burst during a bulk transfer as proof that the path has stopped carrying the current MTU, so a download makes the connection fall back to the 1200-byte floor — and on the 0.11.x branch every further loss burst at the floor re-triggers the detector and pushes the next probe out again, so the MTU stays there for as long as the transfer lasts. That branch was also missing two comparison fixes that upstream had landed on main only.

The pinned commit, 48455d3, is the quinn-proto-0.11.17 tag plus five fixes (and two unrelated backports the branch picked up in between: ACK frames bundled into DATAGRAM/STREAM packets, and a minimum-RTT path statistic). Three are in quinn-proto/src/connection/mtud.rs, brought to 0.11.x by quinn-rs/quinn#2799:

• “Fix some comparisons in the black hole detector” and “Relax MTU discovery state assertion”, backported from quinn-rs/quinn#2400.
• “proto: treat an equal-size delivery as evidence against a preceding loss burst”. The actual bug is #2791 and the fix #2792, merged to main.

The fourth is in quinn-proto/src/connection/datagrams.rs: the outgoing-datagram eviction loop subtracted an evicted datagram’s size from the buffered-bytes counter twice, so sustained eviction underflows the counter and ends in a panic inside send_datagram while the connection lock is held. The bug is #2805 and the fix #2806, landed on 0.11.x.

The fifth is in quinn-proto/src/endpoint.rs: a datagram carrying an unsupported QUIC version elicited a Version Negotiation packet whatever its size, where RFC 9000 Section 5.2.2 says servers MUST drop the small ones. A spoofed 15-byte probe therefore got a larger reply, an amplification vector aimed squarely at the port scanner the SNI gate exists to defeat. The fix is #2822 on main, backported to 0.11.x by #2823.

No 0.11.x release carries these yet, which is why the stanza pins a commit rather than a version. Because that commit’s version is the upstream tag, anything true of quinn-proto 0.11.17 is true here except those five fixes.

Exit condition: drop the stanza as soon as a quinn-proto 0.11.x release includes the fix, then cargo update -p quinn-proto and remove the CI audit workaround described below. Two things do not work while it is in place:

• Dependabot does not touch git dependencies, so it cannot bump this one. The pinned commit is moved by hand whenever quinn-proto ships a security or hardening release, and moving it is forced if a quinn release raises its quinn-proto requirement above 0.11.17.
• cargo-audit skips lockfile entries whose source is not the default registry, so the patched entry would fall out of RustSec coverage. The audit job in .github/workflows/ci.yml works around this by scanning a copy of Cargo.lock with that one source line rewritten back to the registry — which is sound precisely because the version is the upstream tag.

The h3api facade

src/h3api.rs is the facade over src/h3: the only module allowed to name a type from it, so that everything else — conn, quic, the tunnels — sees bytes and quinn types plus the handful of names it exports. The facade began as insulation from a 0.0.x dependency’s API churn and was kept on its own terms: it is the list of what a proxy actually asks of HTTP/3, and it is short enough to read in one sitting.

The message types — Request, Status, Fields — are re-exported through it rather than wrapped in something else. The rule the facade enforces is about where a type is named, not about hiding it: nothing outside src/h3 reaches into that module, so this one file still says everything the rest of the crate may assume about HTTP/3.

One gotcha it cannot hide: h3api::Connection::handshake consumes the quinn::Connection, so clone it first if you also need to send datagrams on it. Receiving them needs no clone — the HTTP/3 connection routes them itself.

Testing

Integration tests assert on-wire behaviour, not internal state: status codes, response headers, and in it_settings the actual bytes the server writes on its control stream. That is what makes a dependency bump reviewable — the tests describe the protocol, so they still mean something after the implementation underneath changes.

Two tests are load-bearing beyond their names:

• The multi-session concurrent CONNECT-UDP test is the regression baseline for Quarter-Stream-ID-class bugs. Three sessions on one connection, each verified to receive its own traffic. Never weaken it.
• it_migration rebinds the client endpoint mid-tunnel and asserts that existing tunnels survive the address change and that new ones can still be opened, which pins QUIC connection migration against an accidental migration(false) or an upstream regression. What makes migration work behind NAT at all is quinn’s non-zero-length server connection IDs (8 bytes by default) — the property RFC 9308 §9 strongly recommends for exactly this reason — so any future endpoint tuning must not shorten them to zero.

it_stress keeps a heavy tier behind #[ignore] (500 concurrent tunnels, 10000 setup/teardown rounds) and a light tier in the default run that catches slot leaks by shrinking the quota until a leak turns into an observable 503.

These pages are checked too. it_docs reads docs/ as text and asserts three things a machine can settle: that the configuration keys documented here and the ones script/config.example.toml ships are one set, and that the documented set still parses as a Config; that every constant a page quotes in the `IDENT` = value notation holds the value the crate compiles; and that every docs/<page>.md#<anchor> reference in src/, in tests/ and in these pages themselves names a heading that exists. What no gate can check is whether a sentence is true, so that half stays a review question.

Test infrastructure lives in tests/common/: an in-process server plus self-signed certificates from rcgen, so no test needs a fixture on disk, and the HTTP/3 client in tests/common/h3client.rs that drives it.

One thing that suite cannot do is disagree with itself: that client is built on the same codec as the server, so a misreading of the framing or of QPACK is one both ends share. The interop CI job closes that gap by starting a real volto process and driving it with two clients that share no code, no QUIC stack and no reading of the RFCs with it or with each other. Go’s masque-go on quic-go speaks CONNECT-UDP over the RFC 9298 default URI template, with authentication on and the server’s certificate trusted rather than skipped; it asserts multi-round echo on one session, three concurrent sessions on a single QUIC connection receiving only their own traffic, Proxy-Status on a refusal, and a 407 when credentials are omitted. Python’s aioquic covers what masque-go cannot: the plain CONNECT (TCP) tunnel — round trips, two concurrent tunnels, and the RFC 9114 §4.4 half-close — plus a datagram carrying an unknown Context ID, which RFC 9298 requires to be dropped silently and which only a client that writes its own Context ID prefix can send. The clients live in tests/interop/ (a Go module) and tests/interop/aioquic/ (a Python script), so cargo test neither sees nor needs them.

A third independent implementation judges the field-section encoding in particular, and its verdicts are checked in rather than re-run: the differential oracle in tests/interop/difforacle/ puts generated and mutated field sections through this server’s QPACK and Huffman decoders and through ls-qpack (via pylsqpack), comparing both the accept/reject verdict and the decoded fields. It needs a pinned Python environment and so runs on demand like the fuzz targets, but what it settled lives in tests/it_diff_oracle.rs and is re-checked on every cargo test: the five inputs the two implementations answer differently, none of them a fault here, and ls-qpack’s own copy of the RFC 9204 static table, entry by entry. That table is the one thing the in-tree tests cannot check for themselves, since src/h3/qpack.rs transcribes RFC 9204 Appendix A by hand and its unit test can only compare that transcription with itself.

Every hand-written parser that reads bytes a peer chose also has a coverage-guided fuzz target under fuzz/, run on demand on a nightly toolchain in a Linux container rather than in CI – which only type-checks them, since fuzz/ is a workspace of its own and its targets reach items that are public only because they ask for them, so a rename in src/ would otherwise break the fuzzers silently between runs. fuzz/README.md maps each target to the parser it covers and has the invocation. Beside the five codecs — the frame state machine, QPACK, Huffman, the capsule parser and the datagram codec — that means the request validator that reaches RFC 9114 §4.1.2’s malformed verdict, the two parsers that turn a client-named target into a host and a port, and the credentials field, which is the one attacker-shaped byte string read before anything has authenticated. The property tests in tests/it_props.rs and tests/it_fuzz.rs state the same kinds of invariants on stable Rust, so they are what CI runs on every push.