Skip to main content

Streaming Video with Media Source Extensions

A page that plays a long video (a recorded class, a deposition, a screen capture) without ever loading the whole file into memory. The video stays on the server as a fragmented MP4 (fMP4); a Python generator reads one fragment at a time and streams it to the browser, which appends it to a SourceBuffer.

note

video.srcObject does not accept arbitrary MP4/H.264 bytes. To play bytes that came from a Python function inside a <video> element, you need Media Source Extensions (MediaSource + SourceBuffer).

The pieces:

build_index.py            # run once, locally — maps every fragment to a byte range
page_player.py            # the Page: index endpoint + one-fragment-at-a-time generator
static/js/player.js       # MSE player: append, buffer ahead, evict, seek

1. Convert the video to fragmented MP4​

A regular MP4 has one contiguous mdat, so there is no safe place to cut it. An fMP4 is a sequence of independent fragments:

[ftyp][moov]  [moof][mdat] [moof][mdat] ...
└─ init ───┘  └─ fragment ┘└─ fragment ┘

ftyp + moov form the init segment, which must reach the browser before any fragment. Remuxing does not re-encode — it only rewrites the container:

ffmpeg -i input.mp4 \
  -map 0:v:0 -map 0:a:0 \
  -c copy \
  -movflags +frag_keyframe+empty_moov+default_base_moof \
  lesson-1.mp4

Fragments are cut at the keyframes that already exist in the source, so their durations vary. If you need short, predictable fragments (2–4s), re-encode with a fixed GOP:

ffmpeg -i input.mp4 \
  -c:v libx264 -profile:v high -level 4.0 \
  -g 120 -keyint_min 120 -sc_threshold 0 \
  -c:a aac -b:a 128k \
  -movflags +frag_keyframe+empty_moov+default_base_moof \
  lesson-1.mp4

At 30 fps, -g 120 gives a keyframe every 4 seconds.

2. Index the fragments​

The browser needs to know where each fragment lives before it can ask for one. This script walks the MP4 box structure and writes a JSON index next to the video — run it locally, once per video:

# build_index.py
import json
import struct
import subprocess
import sys
from pathlib import Path

CONTAINERS = {b"moov", b"trak", b"mdia", b"minf", b"stbl", b"moof", b"traf"}


def iter_boxes(f, start, end):
    """Yield (type, offset, size, payload_offset) for every box in [start, end)."""
    offset = start
    while offset < end:
        f.seek(offset)
        header = f.read(8)
        if len(header) < 8:
            return
        size, kind = struct.unpack(">I4s", header)
        payload = offset + 8
        if size == 1:  # 64-bit size follows the header
            size = struct.unpack(">Q", f.read(8))[0]
            payload = offset + 16
        elif size == 0:  # box runs to the end of the file
            size = end - offset
        yield kind, offset, size, payload
        offset += size


def find(f, kind, start, end):
    """Depth-first search for boxes of a type, without descending into matches."""
    for box, offset, size, payload in iter_boxes(f, start, end):
        if box == kind:
            yield offset, size, payload
        elif box in CONTAINERS:
            yield from find(f, kind, payload, offset + size)


def read_uint(f, offset, length):
    f.seek(offset)
    return int.from_bytes(f.read(length), "big")


def video_track(f, moov_offset, moov_size, moov_payload):
    """(track_id, timescale) of the first video track."""
    for offset, size, payload in find(f, b"trak", moov_payload, moov_offset + moov_size):
        end = offset + size
        hdlr = next(find(f, b"hdlr", payload, end), None)
        if hdlr is None:
            continue
        f.seek(hdlr[2] + 8)  # version/flags + pre_defined
        if f.read(4) != b"vide":
            continue
        _, _, tkhd = next(find(f, b"tkhd", payload, end))
        _, _, mdhd = next(find(f, b"mdhd", payload, end))
        # 64-bit timestamps (version 1) push the field 8 bytes further
        track_id = read_uint(f, tkhd + (20 if read_uint(f, tkhd, 1) == 1 else 12), 4)
        timescale = read_uint(f, mdhd + (20 if read_uint(f, mdhd, 1) == 1 else 12), 4)
        return track_id, timescale
    raise RuntimeError("No video track found in the moov box.")


def fragment_start(f, offset, size, payload, track_id, timescale):
    """Start time of a moof, read from the video track's tfdt box."""
    for traf_offset, traf_size, traf_payload in find(f, b"traf", payload, offset + size):
        end = traf_offset + traf_size
        _, _, tfhd = next(find(f, b"tfhd", traf_payload, end))
        if read_uint(f, tfhd + 4, 4) != track_id:
            continue
        tfdt = next(find(f, b"tfdt", traf_payload, end), None)
        if tfdt is None:
            raise RuntimeError("Fragment without tfdt — re-run ffmpeg with +default_base_moof.")
        version = read_uint(f, tfdt[2], 1)
        return read_uint(f, tfdt[2] + 4, 8 if version == 1 else 4) / timescale
    raise RuntimeError(f"No video fragment inside the moof at byte {offset}.")


def build_index(video: Path) -> dict:
    duration = float(
        subprocess.run(
            ["ffprobe", "-v", "error", "-show_entries", "format=duration",
             "-of", "csv=p=0", str(video)],
            capture_output=True, text=True, check=True,
        ).stdout
    )

    fragments = []
    with video.open("rb") as f:
        boxes = list(iter_boxes(f, 0, video.stat().st_size))
        moov = next(box for box in boxes if box[0] == b"moov")
        track_id, timescale = video_track(f, moov[1], moov[2], moov[3])

        init_size = None
        pending = None
        for kind, offset, size, payload in boxes:
            if kind == b"moof":
                if init_size is None:
                    init_size = offset  # everything before the first moof
                pending = {
                    "index": len(fragments),
                    "offset": offset,
                    "size": size,
                    "start": round(
                        fragment_start(f, offset, size, payload, track_id, timescale), 3
                    ),
                }
                fragments.append(pending)
            elif kind == b"mdat" and pending is not None:
                pending["size"] += size  # a fragment is moof + mdat
                pending = None

    return {
        "duration": duration,
        "init": {"offset": 0, "size": init_size},
        "fragments": fragments,
    }


if __name__ == "__main__":
    video = Path(sys.argv[1])
    index_path = video.with_suffix(".index.json")
    index = build_index(video)
    index_path.write_text(json.dumps(index))
    print(f"{len(index['fragments'])} fragments indexed in {index_path}")

The result is a flat map from fragment number to byte range and start time:

{
  "duration": 3858.4,
  "init": { "offset": 0, "size": 1259 },
  "fragments": [
    { "index": 0, "offset": 1259, "size": 583237, "start": 0.0 },
    { "index": 1, "offset": 584496, "size": 504509, "start": 8.333 }
  ]
}

Never assume fragments have equal duration — always locate a timestamp through start.

Upload lesson-1.mp4 and lesson-1.index.json to the persistent directory (see File uploads) so both live under get_persistent_dir().

3. Page backend — read one segment at a time​

The browser drives the streaming: it asks for the init segment first, then for one fragment at a time. Python only ever reads the requested byte range, so memory stays flat no matter how long the video is.

# page_player.py
import base64
import json

from abstra.files import get_persistent_dir
from abstra.pages import register_function, register_static

VIDEO = get_persistent_dir() / "streaming" / "lesson-1.mp4"
INDEX = VIDEO.with_suffix(".index.json")

CHUNK_SIZE = 256 * 1024


def read_index() -> dict:
    if not VIDEO.is_file() or not INDEX.is_file():
        raise FileNotFoundError(f"Missing {VIDEO.name} or {INDEX.name} in Files.")
    return json.loads(INDEX.read_text())


@register_function
def get_stream_index():
    """Byte ranges and total duration — no media bytes."""
    return read_index()


@register_function
def stream_fragment(fragment_index: int):
    """Stream one fMP4 segment. Use -1 for the init segment."""
    index = read_index()

    if fragment_index == -1:
        segment = index["init"]
    else:
        fragments = index["fragments"]
        if not 0 <= fragment_index < len(fragments):
            raise ValueError(f"Invalid fragment: {fragment_index}")
        segment = fragments[fragment_index]

    remaining = segment["size"]
    with VIDEO.open("rb") as f:
        f.seek(segment["offset"])
        while remaining > 0:
            data = f.read(min(CHUNK_SIZE, remaining))
            if not data:
                raise RuntimeError("Incomplete read — the index does not match the file.")
            remaining -= len(data)
            yield {"data": base64.b64encode(data).decode("ascii")}
Why Base64?

Page generators stream NDJSON, so raw MP4 bytes have to be encoded to survive JSON. Base64 is simple and reliable, but adds roughly 33% to the transferred size — see Going to production below.

4. Player — feed the SourceBuffer​

Keep the player in its own file and serve it with register_static. It appends the init segment once, then keeps ~30s buffered ahead of playback and drops what is far behind, so memory stays bounded on a two-hour video.

// static/js/player.js
const MIME = 'video/mp4; codecs="avc1.640028, mp4a.40.2"';
const AHEAD = 30;   // seconds to keep buffered ahead of playback
const BEHIND = 60;  // seconds to keep behind before evicting

const video = document.getElementById("video");
const startButton = document.getElementById("start");
const status = document.getElementById("status");

let mediaSource;
let sourceBuffer;
let index;
let next = 0;
let pumping = false;

function once(target, event) {
  return new Promise((resolve, reject) => {
    const cleanup = () => {
      target.removeEventListener(event, done);
      target.removeEventListener("error", fail);
    };
    const done = () => { cleanup(); resolve(); };
    const fail = () => { cleanup(); reject(new Error(`${event} failed`)); };
    target.addEventListener(event, done, { once: true });
    target.addEventListener("error", fail, { once: true });
  });
}

function base64ToBytes(value) {
  const binary = atob(value);
  const bytes = new Uint8Array(binary.length);
  for (let i = 0; i < binary.length; i += 1) bytes[i] = binary.charCodeAt(i);
  return bytes;
}

// One call can arrive as several NDJSON chunks — reassemble the segment.
async function fetchSegment(fragmentIndex) {
  const parts = [];
  let total = 0;
  for await (const chunk of stream_fragment(fragmentIndex)) {
    const bytes = base64ToBytes(chunk.data);
    parts.push(bytes);
    total += bytes.length;
  }
  const segment = new Uint8Array(total);
  let at = 0;
  for (const part of parts) { segment.set(part, at); at += part.length; }
  return segment;
}

async function append(bytes) {
  sourceBuffer.appendBuffer(bytes);
  await once(sourceBuffer, "updateend");
}

// How much playable video sits ahead of the playhead, in the current range.
function bufferedAhead() {
  const ranges = sourceBuffer.buffered;
  for (let i = 0; i < ranges.length; i += 1) {
    if (ranges.start(i) <= video.currentTime + 0.1 && video.currentTime < ranges.end(i)) {
      return ranges.end(i) - video.currentTime;
    }
  }
  return 0;
}

async function evictBehind() {
  const ranges = sourceBuffer.buffered;
  const cutoff = video.currentTime - BEHIND;
  if (!ranges.length || cutoff <= ranges.start(0)) return;
  sourceBuffer.remove(ranges.start(0), Math.min(cutoff, ranges.end(0)));
  await once(sourceBuffer, "updateend");
}

async function pump() {
  if (pumping) return;
  pumping = true;
  try {
    while (next < index.fragments.length && bufferedAhead() < AHEAD) {
      await evictBehind();
      await append(await fetchSegment(next));
      next += 1;
      status.textContent = `${next}/${index.fragments.length} fragments`;
    }
    if (next >= index.fragments.length && mediaSource.readyState === "open") {
      mediaSource.endOfStream();
    }
  } finally {
    pumping = false;
  }
}

// Binary search: fragments have different durations, so only `start` is reliable.
function fragmentAt(time) {
  let low = 0;
  let high = index.fragments.length - 1;
  let found = 0;
  while (low <= high) {
    const middle = Math.floor((low + high) / 2);
    if (index.fragments[middle].start <= time) {
      found = middle;
      low = middle + 1;
    } else {
      high = middle - 1;
    }
  }
  return found;
}

async function start() {
  if (!window.MediaSource || !MediaSource.isTypeSupported(MIME)) {
    throw new Error(`Unsupported codec: ${MIME}`);
  }

  index = await get_stream_index();
  mediaSource = new MediaSource();
  video.src = URL.createObjectURL(mediaSource);
  await once(mediaSource, "sourceopen");

  sourceBuffer = mediaSource.addSourceBuffer(MIME);
  mediaSource.duration = index.duration; // full seek bar before anything is loaded

  await append(await fetchSegment(-1)); // the init segment always comes first
  await pump();
}

video.addEventListener("timeupdate", () => pump().catch(console.error));
video.addEventListener("waiting", () => pump().catch(console.error));
video.addEventListener("seeking", () => {
  next = fragmentAt(video.currentTime);
  pump().catch(console.error);
});

startButton.addEventListener("click", () => {
  startButton.disabled = true;
  start().catch((error) => {
    startButton.disabled = false;
    status.textContent = error.message;
  });
});

5. Render the page​

@register_function
def __render__():
    player_url = register_static("static/js/player.js")
    return f"""
<script src="https://cdn.tailwindcss.com"></script>
<style>body {{ margin:0; font-family:'Inter',system-ui,sans-serif; background:#0f172a; }}</style>
<div class="min-h-screen flex items-center justify-center p-8">
  <div class="w-full max-w-3xl">
    <video id="video" controls playsinline class="w-full rounded-2xl bg-black shadow-xl"></video>
    <div class="flex items-center justify-between mt-4">
      <button type="button" id="start"
        class="bg-indigo-600 hover:bg-indigo-700 disabled:opacity-40 text-white text-sm font-medium px-5 py-2.5 rounded-lg transition-all">
        Load video
      </button>
      <span id="status" class="text-sm text-slate-400 font-mono"></span>
    </div>
  </div>
</div>
<script type="module" src="{player_url}"></script>
"""
Codec string

The MIME must describe what is actually inside the file — avc1.640028, mp4a.40.2 is H.264 High @ L4.0 with AAC-LC. Check yours with ffprobe -show_streams, and always gate playback on MediaSource.isTypeSupported() instead of assuming.

Use type="button" on the button: without it, the browser treats it as a submit button and reloads the page.

Key patterns: Generator yields byte ranges instead of whole files (constant memory), register_static for the player, init segment appended before any fragment, await once(sourceBuffer, 'updateend') after every appendBuffer/remove, sliding buffer window with eviction, binary search over start for seeking, mediaSource.duration for a full seek bar up front.

Validation checklist​

  • The fMP4 starts with ftyp + moov, and every fragment is moof + mdat.
  • The index has init, duration, and one entry per fragment.
  • The codec string passed MediaSource.isTypeSupported().
  • The init segment is appended before any moof/mdat.
  • Every appendBuffer() and remove() waits for updateend.
  • The player keeps a bounded window and evicts what is behind.
  • Seeking resolves the fragment through start, never through a fixed duration.
  • Media files live under get_persistent_dir(), not next to the code.

Going to production​

Base64 over NDJSON costs ~33% extra bandwidth and CPU on both ends. For large videos with many concurrent viewers, serve the fMP4 from an origin that honors HTTP Range requests — create_public_link() from abstra.files gives you such a URL — and keep the exact same player, replacing fetchSegment with a ranged fetch:

async function fetchSegment(fragmentIndex) {
  const s = fragmentIndex === -1 ? index.init : index.fragments[fragmentIndex];
  const response = await fetch(VIDEO_URL, {
    headers: { Range: `bytes=${s.offset}-${s.offset + s.size - 1}` },
  });
  return new Uint8Array(await response.arrayBuffer());
}

Note that a public link is unauthenticated — anyone with the URL can download the video. Keep the Python generator when access must go through the page's own auth.