Adaptive Bitrate Streaming & MSE

2026-08-11 · 9 min read

A <video src="movie.mp4"> tag works fine for a short clip. It falls apart for real video products. A 4K movie file is several gigabytes — a user on a 3G connection can't stream it without constant buffering. A user on fiber doesn't want to watch a blurry 480p version. A single file can't adapt to either.

Adaptive bitrate streaming (ABR) solves this by encoding video at multiple quality levels and letting the player switch between them in real time based on the viewer's current network conditions. The player is always trying to answer one question: what's the highest quality I can sustain right now without the video stalling?

How HLS and DASH work

HLS (HTTP Live Streaming, developed by Apple) and DASH (Dynamic Adaptive Streaming over HTTP, an open standard) both follow the same model. The video is pre-processed into a set of files that the player fetches on demand.

Step 1: Encode at multiple quality levels.

The source video is encoded into several versions at different resolutions and bitrates:

240p  —  400 Kbps
480p  —  1.5 Mbps
720p  —  3 Mbps
1080p —  8 Mbps
4K    —  25 Mbps

Step 2: Split each version into short segments.

Each quality level is chopped into segments of 2–10 seconds each. A 2-hour movie at 5 quality levels split into 6-second segments produces roughly 6,000 segment files.

Segments come in two formats. The older format is MPEG-2 Transport Stream (.ts), widely supported and still used for legacy compatibility. Modern HLS and all of DASH use Fragmented MP4 (.m4s) — the same container format as a regular .mp4 file, but structured so that each segment is independently decodable. fMP4 enables better seeking and byte-range requests.

720p/segment_001.m4s   (seconds 0–6)
720p/segment_002.m4s   (seconds 6–12)
720p/segment_003.m4s   (seconds 12–18)
...

Step 3: Generate a manifest file.

A manifest (.m3u8 for HLS, .mpd for DASH) is a text file that lists all quality levels and the URLs for every segment within each level. The player fetches the manifest first to understand what's available.

# HLS manifest (simplified)
#EXTM3U
#EXT-X-STREAM-INF:BANDWIDTH=400000,RESOLUTION=426x240
240p/index.m3u8
#EXT-X-STREAM-INF:BANDWIDTH=1500000,RESOLUTION=854x480
480p/index.m3u8
#EXT-X-STREAM-INF:BANDWIDTH=3000000,RESOLUTION=1280x720
720p/index.m3u8
#EXT-X-STREAM-INF:BANDWIDTH=8000000,RESOLUTION=1920x1080
1080p/index.m3u8

The player fetches the manifest, picks a starting quality level, then downloads segments one at a time. Each segment is a regular HTTP request — the CDN serves it like any other file.

Live vs VOD manifests. For pre-recorded content (VOD), the manifest is a static file — all segments are listed upfront and the manifest ends with #EXT-X-ENDLIST. For a live stream, the manifest has no end tag and keeps changing: as new segments are produced every few seconds, they're appended to the manifest. The player polls the manifest URL every 1–3 seconds to discover new segments. When the broadcast ends, #EXT-X-ENDLIST is added and the player stops polling.

Media Source Extensions

The browser's <video> element doesn't know how to fetch and stitch together a stream of segments — it expects a single URL pointing to a complete file. Media Source Extensions (MSE) is the browser API that bridges this gap. It lets JavaScript feed raw video data into a <video> element piece by piece.

const video = document.querySelector('video');
const mediaSource = new MediaSource();

// Attach the MediaSource to the video element
video.srcObject = mediaSource; // modern preferred approach
// older code uses: video.src = URL.createObjectURL(mediaSource)

mediaSource.addEventListener('sourceopen', () => {
  // Tell the browser what codec to expect
  const sourceBuffer = mediaSource.addSourceBuffer(
    'video/mp4; codecs="avc1.42E01E,mp4a.40.2"'
  );

  // Fetch the first segment and hand it to the browser
  fetchSegment('/720p/segment_001.mp4').then(data => {
    sourceBuffer.appendBuffer(data);
  });
});

SourceBuffer is the conduit between your JavaScript and the browser's video decoder. You fetch a segment (a regular fetch() call), get back an ArrayBuffer of raw video bytes, and pass it to appendBuffer(). The browser decodes and renders it. The player loop just keeps fetching the next segment before the current one finishes playing.

// Simplified player loop
async function fillBuffer(sourceBuffer, currentSegmentIndex) {
  while (shouldKeepBuffering(sourceBuffer)) {
    const quality = chooseQuality();
    const url = segmentUrl(quality, currentSegmentIndex);
    const data = await fetchSegment(url);

    await new Promise(resolve => {
      sourceBuffer.appendBuffer(data);
      sourceBuffer.addEventListener('updateend', resolve, { once: true });
    });

    currentSegmentIndex++;
  }
}

sourceBuffer.updating is true while the browser is processing the last appendBuffer call. You must wait for updateend before appending the next segment — the buffer can only handle one append at a time.

The ABR algorithm

The ABR (Adaptive Bitrate) algorithm runs before each segment fetch and answers: which quality level should I use for the next segment?

The two inputs are bandwidth and buffer health.

Measuring bandwidth. Track how fast recent segments downloaded using an exponential moving average — a weighted average that gives more weight to recent measurements than old ones, so the estimate responds quickly to network changes without overreacting to a single slow segment.

class BandwidthEstimator {
  #estimate = 0;
  #alpha = 0.7; // weight for recent sample vs. historical estimate

  update(bytesDownloaded, durationMs) {
    const sample = (bytesDownloaded * 8) / (durationMs / 1000); // bits per second
    this.#estimate = this.#alpha * sample + (1 - this.#alpha) * this.#estimate;
  }

  get bitsPerSecond() {
    return this.#estimate;
  }
}

Buffer health. The player maintains a buffer of upcoming video. video.buffered returns the ranges of video already downloaded. Buffer health is how many seconds of video are queued ahead of the current playback position.

function bufferAhead(video) {
  const buffered = video.buffered;
  for (let i = 0; i < buffered.length; i++) {
    if (buffered.start(i) <= video.currentTime && buffered.end(i) > video.currentTime) {
      return buffered.end(i) - video.currentTime; // seconds buffered ahead
    }
  }
  return 0;
}

Choosing quality. Pick the highest quality level whose bitrate the current bandwidth can sustain, but apply asymmetric switching rules:

Switch up slowly — upgrade by at most one quality level per segment decision, and only when bandwidth is comfortably above the next level's bitrate. Avoids yo-yoing on a connection that's borderline.
Switch down immediately — if bandwidth drops below the current level's bitrate, drop quality right away. A stall is worse than a visible quality drop.

function chooseQuality(bandwidthBps, bufferAheadSeconds, qualities) {
  const BUFFER_THRESHOLD = 15; // seconds — don't upgrade if buffer is thin
  const SAFETY_FACTOR = 0.8;   // use 80% of measured bandwidth for headroom

  const usableBandwidth = bandwidthBps * SAFETY_FACTOR;

  // Find the highest quality the bandwidth can support
  const affordable = qualities.filter(q => q.bitrate <= usableBandwidth);
  if (affordable.length === 0) return qualities[0]; // lowest quality

  const best = affordable[affordable.length - 1];

  // Don't upgrade if the buffer is thin — prioritize stability
  if (bufferAheadSeconds < BUFFER_THRESHOLD) {
    return currentQuality; // stay where we are
  }

  return best;
}

Buffer-based ABR. The rate-based approach (measuring download speed) works well when bandwidth is stable, but it's sensitive to short network spikes — a momentary dip causes an unnecessary quality drop. An alternative is to make quality decisions based purely on buffer occupancy: if the buffer is healthy, hold or upgrade; if it's draining, downgrade. This is the idea behind BOLA (Buffer Occupancy-based Lyapunov Algorithm), the default ABR algorithm in dash.js and an option in Shaka Player. Buffer-based approaches are more stable on bursty networks because the buffer absorbs short dips rather than reacting to them immediately. The tradeoff is slower reaction when the network genuinely and permanently degrades.

Startup optimization

The naive approach — wait for the page to fully load, then start fetching segments — produces a noticeable delay before the video starts playing. Two optimizations help:

Start fetching early. Begin the manifest request and the first segment fetch in parallel with page hydration, before the React tree mounts or the player UI renders. The first segment can be in the buffer by the time the <video> element exists.

Start at low quality. Fetch the first segment at the lowest quality level regardless of estimated bandwidth. The goal at startup is to get something playing immediately. Once the buffer fills, the ABR algorithm upgrades quality naturally.

// Start fetching before React mounts
const prefetch = {
  manifest: fetch('/stream/manifest.m3u8'),
  firstSegment: fetch('/stream/240p/segment_001.mp4'),
};

// Later, when the player component mounts:
function VideoPlayer({ streamUrl }) {
  useEffect(() => {
    initPlayer(prefetch); // reuse the already-in-flight requests
  }, []);
}

Buffer management

SourceBuffer has a size limit that varies by browser and can shrink further under memory pressure. Approximate limits for video buffers:

Browser	Video buffer	Audio buffer
Chrome	~150 MB	~12 MB
Firefox	~100 MB	~15 MB
Safari	~290 MB	~14 MB
Chromecast	~30 MB	~2 MB

There's no API to query remaining capacity, and Chrome's limit can drop below its nominal value when the device is under memory pressure. The right approach is to evict proactively — remove segments already watched — and catch QuotaExceededError reactively as a fallback. The fix when a QuotaExceededError occurs is to remove more data and retry the append.

function evictOldSegments(sourceBuffer, video) {
  const KEEP_BEHIND = 60; // seconds to keep behind current time
  const evictUpTo = video.currentTime - KEEP_BEHIND;

  if (evictUpTo > 0 && sourceBuffer.buffered.length > 0) {
    const bufferedStart = sourceBuffer.buffered.start(0);
    if (bufferedStart < evictUpTo) {
      sourceBuffer.remove(bufferedStart, evictUpTo);
    }
  }
}

Call this before each appendBuffer — free old data before adding new data. The player only needs a forward-looking buffer; there's no reason to keep segments the viewer already watched.

Where this appears

Video streaming — Netflix, YouTube, Disney+, and every major streaming platform use HLS or DASH with MSE. The player (whether the browser's built-in one or a custom JavaScript player like Shaka Player or hls.js) implements the ABR loop.

Live streaming delivery — Twitch and YouTube Live encode the stream into segments in real time as the broadcast happens. The manifest file updates as new segments become available; the player fetches the manifest periodically to discover them.

Audio streaming — Spotify uses a simpler version of the same idea for audio. Tracks are split into chunks fetched progressively. There's no quality switching (audio bitrate is low enough that network adaptation is less critical), but the buffer management and progressive fetch pattern are the same.

Embedded video — Video in social feeds (Twitter/X, Instagram, TikTok) uses MSE for the same reason: start playback instantly with a low-quality segment, upgrade as the viewer watches.