DK Dark Knight 2026-08-31 5 min read

HFlow: Robotics Data Pipeline SDK That 404s on First Run

HFlow hit the front page of Hacker News this morning as a YC S26 launch: an open-source SDK for robotics data pipelines — the boring, brutal plumbing between raw sensor recordings and training-ready episodes. I cloned it, ran the advertised quickstart, and it died in the first second with a 404. Then it worked beautifully. Both facts are the story.

What the HFlow Robotics Data Pipeline SDK Does

HFlow is a four-stage pipeline SDK from Hebbian Robotics (founders Brandon Ong and Kingston Kuan): collection → ingestion (transform → QC gate → enrich) → curation (DuckDB SQL over a Parquet catalog) → delivery (canonical MCAP plus a version-pinned manifest). Episodes are MCAP — the container format Foxglove and Rerun read natively — and the processing steps are plain Python functions you write and own. Scheduled runs compile to Airflow 3 DAGs.

The problem is real. Robotics teams train on recordings that mix video, joint states, actions and metadata, and quality control is where the rot starts: frozen cameras, timestamp drift, duplicate episodes quietly poisoning the corpus. HFlow's answer is its most opinionated design decision: evidence, not verdicts. Checks emit measurements — black-frame percent, period-violation percent, keyframe gaps — and pass/fail policy lives later, in SQL, at curation time. Verdicts rot; measurements don't.

The repo is 13 days old (created Aug 18) and already carries 195 stars and 116 forks. v0.2.4 landed yesterday; the last commit was pushed 90 minutes before I cloned it.

graph LR
  A[Raw recordings
landing bucket] --> B[hflow ingest]
  B --> C[Transform →
canonical MCAP]
  C --> D[QC checks →
measurements]
  D --> E[Enrichment +
artifacts]
  E --> F[Parquet catalog +
DuckDB SQL]
  F --> G[Version-pinned
manifest]
  G --> H[Trainer /
dataset buyer]

First Run: the 404 the Docs Don't Mention

The advertised path is four commands: git clone, uv sync --locked, uv run python examples/quickstart.py. The sync was clean — the project hardens its supply chain with exclude-newer = "5 days" in its uv config, refusing PyPI packages younger than five days, a policy most mature teams don't have. Then the quickstart died at its first video operation:

hflow.ffmpeg._binary.PinnedDownloadError: failed to download the pinned ffmpeg build
from https://github.com/BtbN/FFmpeg-Builds/releases/download/autobuild-2026-08-16-13-00/
ffmpeg-n8.1.2-44-g7c533d0f86-linux64-gpl-8.1.tar.xz: HTTP Error 404: Not Found

I re-checked the URL from outside the tool. GitHub really does return 404. BtbN's autobuild releases only keep recent builds — the release list now starts at Aug 28 — and HFlow pinned the Aug 16 one. That's pin rot: the build was pruned upstream, so the failure is deterministic for every Linux user on first run.

What makes it sting: HFlow's Linux policy is to never use a PATH ffmpeg — pinned downloads only, so measurements stay comparable across runs. Fair philosophy. But my box already had ffmpeg 7.1.5 sitting at /usr/bin/ffmpeg, and HFlow refused to touch it until I set two environment variables, exactly as the error message says:

HFLOW_FFMPEG=/usr/bin/ffmpeg HFLOW_FFPROBE=/usr/bin/ffprobe uv run python examples/quickstart.py

With the Workaround: the Whole Pipeline in 6 Seconds

Exit 0, six seconds wall time. HFlow synthesized a 7.99-second fake episode — two cameras (overhead + wrist), joint states, 1,040 messages across 3 topics — ran it through the QC gate in-process with no Docker and no Airflow, and wrote 2.0 MB of outputs: a canonical MCAP stamped with provenance (schema_version=1, pipeline_version=9cd5d34f513d, the ffmpeg instrument), per-step evidence, and contact-sheet artifacts.

The measurements are exactly what the docs promise:

hflow doctor then validated the canonical episode: conforming: no findings.

Where Zero Infrastructure Stops: Curation Needs Docker

Here's the gap the green run hides: the pipeline's last act — DuckDB SQL over the Parquet catalog, building the version-pinned manifest — never happened. hflow curate --dry-run "SELECT episode_id, uri, status FROM episodes" refuses cleanly: data/catalog is not a catalog root. The catalog is created by the Compose runtime (hflow up, ~2 GB of container images) or a server deploy. The zero-infra path ends at QC evidence, not at a curated dataset.

To their credit, they say this out loud: docs/ARCHITECTURE.md carries an implementation-status matrix labeling every mechanism implemented / simplified / deferred / out of scope — including their own scale ceiling: byte-balanced batch scheduling yes, the joint optimizer no; per-step GPU routing deferred; distributed corpus cache out of scope. I can't remember the last launch repo this honest.

The sharp irony: curation is where the ROI lives — policy-as-SQL over evidence is genuinely the right model — but the 6-second demo that sells the tool can't show it.

Bottom Line: Sharp Engineering, Broken First Step

HFlow is the most disciplined 13-day-old repo I've tested: evidence-not-verdicts, versioned steps, standard formats at every boundary, a supply-chain policy most mature teams lack. But a launch quickstart that dies on a pruned upstream build — while a working ffmpeg sits on the same machine, deliberately ignored — is exactly the first impression that loses evaluation teams. It's a one-line fix for users (env vars) and a two-line fix for the repo (mirror the pin so it can't rot). Worth the four minutes, because underneath the 404 there's a pipeline system that robotics is actually going to use.

Related reading