bismigel

IPv6 packet loss, not slow DNS: why Jellyfin buffered off the LAN

Buffering from outside the LAN is the kind of report that invites guessing. The usual suspects are the transcode, the disk, or the network. This time the report came with a detail that did not fit any of them: the site felt slow “only at DNS”, and from a machine on 192.168.2.0/24 everything was fast.

That detail was the clue. DNS was not slow. One address family was losing packets, and everything else followed from that.

The report

The user was streaming from a remote network. Playback started, stalled after a few seconds, restarted, and stalled again. From inside the LAN the same file played normally.

The first instinct is to blame the server. The server was idle:

  • node CPU 41% of 4 cores, load 2.56, iowait 3.7%;
  • disk utilisation 1.9% on NVMe and 1.45% on the media disk;
  • Jellyfin using 0.51 cores and 783 MiB of a 4 GiB limit;
  • about 1.8 Mbit/s on the wire, which is exactly one stream.

Nothing on the box was the constraint. The Jellyfin log showed the real shape of the failure: ffmpeg started and was stopped a dozen times in six minutes, and every stop was clean (FFmpeg exited with code 0). The server was not crashing. The client was giving up and asking again.

Measure both families

The remote client was dual-stack. I asked for the same request over each family on that machine:

bash
curl -4 -sk -o /dev/null -w "v4 dns=%{time_namelookup} tcp=%{time_connect} ttfb=%{time_starttransfer}\n" https://jellyfin.bismigel.dev/System/Info/Public
curl -6 -sk -o /dev/null -w "v6 dns=%{time_namelookup} tcp=%{time_connect} ttfb=%{time_starttransfer}\n" https://jellyfin.bismigel.dev/System/Info/Public

One pair said IPv6 was slow. That was not the whole story, so we ran it five more times:

runv4 dnsv4 tcpv4 ttfbv6 dnsv6 tcpv6 ttfb
10.0130.0230.0940.0151.0441.246
20.0150.0250.1850.0150.0250.240
30.0160.0260.0960.0120.0220.169
40.0120.0220.1340.0110.0210.302
50.0120.0230.0785.0165.0275.215
60.0120.0220.1510.0110.0220.192

Look at runs 2, 3, 4 and 6: IPv6 is as fast as IPv4. Twenty milliseconds to connect, both families. Then run 1 and run 5 are one and five seconds.

That is not a slow path. That is loss with retransmission:

  • tcp=1.044 is a lost SYN, retried on the kernel backoff;
  • dns=5.016 is a DNS query that was sent, dropped, and retried on the resolver’s five second timer. Nothing had been sent to Cloudflare yet.

My first framing was wrong. I called it a constant 45x penalty after one run. Six runs turned it into intermittent loss, which is a different failure and a worse one for video, because it breaks in bursts instead of being uniformly slow.

Why it looked like slow DNS

Every hostname in the zone is a proxied CNAME to the tunnel. With Cloudflare IPv6 compatibility on, the zone answers both families for all of them:

text
A     172.64.80.1                              TTL 282
AAAA  2606:4700:130:436c:6f75:6466:6c61:7265   TTL 282

A dual-stack client will prefer AAAA. Browsers handle a bad IPv6 path with Happy Eyeballs: they race both families and drop the loser after about 250 ms. Most native apps do not. Jellyfin’s web client is a browser, but its TV and mobile clients are not, and neither are the small tools people test with.

So the client sat in the timeout, the user saw a spinner, and the natural report is “DNS is slow”. For one second or five seconds at a time, that is exactly what it was.

Why loss becomes buffering

TCP throughput falls roughly as 1 / (RTT * sqrt(loss)). One percent loss is not a small tax; it is close to an order of magnitude on a long path.

Jellyfin’s HLS player measures the throughput it gets and picks a bitrate from it. On the lossy path it measured about 1 Mbit/s and locked into a 720p transcode:

text
02:11:28  -b:v 1116000  ... scale_vaapi=w=1280:h=720 ...

Every stall also killed the HLS session. The next request started a new ffmpeg job, which is the restart storm in the log. The server was doing exactly what it was told; the client could not pull the segments fast enough to keep the session alive.

The shape of the path

text
dual-stack client
      |
      |  1. DNS: A and AAAA, both answered (proxied CNAME to the tunnel)
      v
Cloudflare anycast     172.64.80.1        /  2606:4700:130:436c:...
      |                      |                        |
      |  2. client prefers AAAA  ->  loss here -> 1 s connect, 5 s DNS retry
      |                      |                        |
      |  3. tunnel connector (cloudflared on .81 and .82, IPv4/QUIC)
      v
ingress-nginx :32080  ->  Jellyfin pod on .81  ->  /mnt/media

The tunnel is not part of the problem. The connector talks to Cloudflare over IPv4 either way. What changes is which family the client uses to reach the edge.

The fix at the edge, not on the client

There is no per-hostname switch for this. While IPv6 compatibility is on, Cloudflare serves AAAA for every proxied record in the zone, and a proxied record cannot be made A-only. The options were:

  1. fix every client network that has bad IPv6, forever;
  2. turn the zone’s IPv6 off, once.

For this lab the second one costs nothing. Everything public is a Cloudflare anycast front door in front of a tunnel. No service here needs IPv6, and any client that worked was already falling back to IPv4.

The OpenTofu change is four lines, next to the other zone settings:

hcl
resource "cloudflare_zone_setting" "ipv6" {
  zone_id    = var.zone_id
  setting_id = "ipv6"
  value      = "off"
}

That is PR bakuran-infra #69 (private repository). The change went in through tofu apply, not through the dashboard, so the reason lives in the repo next to the setting.

One thing the documentation does not say clearly: Cloudflare’s docs present turning IPv6 compatibility off as an Enterprise option, and suggest Pseudo IPv4 for everyone else. On this account the API accepted ipv6 = off and the AAAA answers stopped. I verified the result instead of trusting the plan:

bash
curl -s -H 'accept: application/dns-json' \
  'https://cloudflare-dns.com/dns-query?name=jellyfin.bismigel.dev&type=AAAA'
# returns no Answer section after the change

What it bought

The client had an AAAA cached for the TTL (282 seconds) and an open HTTP/2 connection, so nothing changed at once. When the page was reloaded, the session re-authenticated and the source address changed from 2a02:6ea0:... to 185.98.171.223. Two things happened in the same minute:

metricbeforeafter
source familyIPv6IPv4
target bitrate11160009753335
downscale720pnone
transcoder restartsevery 10 to 30 secondsone per user seek

The bitrate went up 8.7x, and the w=1280:h=720 downscale disappeared from the ffmpeg command. The stream was not only smoother, it was higher quality. The last restart in the log was a seek to 00:18:46, which is a user action, not a failure.

That is the part I did not expect. I thought removing the lossy path would stop the buffering. It also let the player re-measure the link and take 9.75 Mbit/s instead of 1.1 Mbit/s, because the loss was what made it choose the low ladder in the first place.

The second limit underneath it

With the path fixed, the next test was a 4K file: Hush (2016), 2160p HDR HEVC. This time the ffmpeg line said something else:

text
02:20:03  -b:v 11616000  scale_vaapi=w=2560:h=1072
          procamp_vaapi, tonemap_vaapi=format=nv12:p=bt709:t=bt709:m=bt709
02:20:05  MediaInfoHelper: RemoteClientBitrateLimit: 12000000 ...

The stream was being tone-mapped from HDR to SDR and re-encoded at 11.6 Mbit/s, not because of the network this time but because of a server default: the Internet streaming bitrate limit of 12 Mbit/s. It applies to every client Jellyfin considers remote, and it caps direct play as well as transcoding. The client could ask for 40 Mbit/s and still get 11.6.

That value is not a Kubernetes field. It lives in config/system.xml on the jellyfin-config PVC, which is why nobody sees it in the repo. It is removed by a one-shot Job that calls the API and verifies the result: bakuran-gitops PR #151.

After it ran, the same file started at 31.7 Mbit/s, still tone-mapping but with no downscale:

text
02:37:06  -b:v 31673882  procamp_vaapi, tonemap_vaapi ...

Two lessons from that: a transport fix can expose a policy limit underneath it, and a setting that lives on a PVC is invisible until it bites.

What actually limits it now

With both caps gone, the honest ceiling list for this lab, measured rather than assumed:

limitmeasured valuewhat it means
node NIC1 Gbps each (node_network_speed_bytes 125000000)~940 Mbit/s per node, about 23 streams at 40 Mbit/s
WAN upload3 Gbpswider than the NIC, so never the binding constraint
CPU, 4K HDR tone-map at 31.7 Mbit/s0.80 cores per stream~4 concurrent before 4 cores saturate
CPU, same at 11.6 Mbit/s downscaled0.58 cores per streamthe cheaper shape before the cap was removed
CPU, direct playnear zero (codec:v copy)remux only, limit becomes disk and NIC

The order matters. Direct play is cheap and the NIC decides. A 4K HDR title that needs tone-mapping is expensive and the CPU decides, long before any cable does. Four people watching the same HDR film now costs 3.2 of the 4 cores, so the fifth one queues. At the old 11.6 Mbit/s target the same four cost 2.3.

That is the real trade in raising the cap: better picture per stream, fewer streams before the box is full.

What I would do differently

  • Measure both address families before theorising. One pair of numbers looked like a constant penalty; six runs showed loss, which is a different diagnosis and a different fix.
  • Treat a 5 second DNS time as a dropped query, not a slow resolver. The number is a timer, not a measurement of distance.
  • Remember that the server can be idle while the user is unhappy. CPU, disk and memory were all clean during the worst of it.
  • Check the cache before declaring a fix live. The client kept using IPv6 for minutes after the zone stopped publishing AAAA, because both the answer and the connection were already open.
  • Keep going one layer down. Fixing the path made the transcodes cheap, which is what made the 12 Mbit/s policy cap visible in the first place.
  • Measure again after raising a cap. The CPU per stream went from 0.58 to 0.80 cores when the same 4K file stopped being downscaled, and that changes how many viewers fit.

Links