Cloudflare Streamline: Building Custom, Long-Running Video Pipelines on Workers, Containers, and Edge Delivery
Discover how to create custom video processing pipelines using Cloudflare’s Streamline, combining Workers, Containers, and Durable Objects for efficient…
Cloudflare has introduced Streamline, a developer playground that shows how to assemble bespoke real-time and on-demand video processing pipelines by combining Workers for control logic, Containers for long-running media workloads, and Durable Objects for session orchestration. The practical pitch: generate dynamic overlays on a live stream, burn in subtitles, or produce alternate video outputs—then publish the result immediately to Cloudflare’s streaming infrastructure—without forcing every request to stay open for the entire processing duration.
For infrastructure teams and hosting operators, Streamline is notable less for the demo use cases and more for the architecture pattern it validates: how to run CPU- and memory-stable media engines on the edge, coordinate stateful sessions safely, and deliver low-latency previews using browser-friendly transports. It also makes security mechanics—authentication, secret handling, and resource bounding—part of the core design rather than an afterthought.
Key Developments
- New developer system and playground that demonstrates custom video manipulation pipelines built on Cloudflare Developer Platform primitives.
- Modular pipeline runtime: a media engine container performs media input/output and processing, while Workers provide orchestration and monitoring.
- Session lifecycle designed for durability: pipelines continue processing even if the controlling Worker disconnects, with enforced maximum durations to prevent runaway sessions.
- Multi-protocol integration including RTMPS ingestion/output, HLS ingestion from hosted assets, and WebSocket-based preview delivery using fMP4 fragments.
- Security model embedded in the workflow: access control for session control, isolation between principals, and protection of upstream RTMPS keys and preview publishing credentials.
- Open source components released on Cloudflare’s GitHub, including the media engine and a reference Worker application with an Astro frontend.
Technical Analysis: How Streamline Fits into Modern Video Infrastructure
Streamline’s architecture is built around a recurring operational reality in media processing: real-time pipelines often run for minutes or hours, require predictable compute for encoding/decoding steps, and must not depend on the lifetime of an individual HTTP request. Traditional serverless request/response patterns are a mismatch; long-lived, stateful workloads need their own lifecycle.
1) A two-component design: control plane vs. data plane
Streamline separates concerns into:
- Media Engine (Container-hosted): handles ingestion, processing, and output. It can pull RTMPS streams from Stream Live inputs, ingest HLS manifests/segments from hosted video, accept webcam-style input from a controlling app, and publish RTMPS output to another Stream Live input.
- Controlling Application (Workers + Durable Objects): coordinates session creation, configuration, observation, and termination. Durable Objects act as the stateful orchestrator—tracking session context and relaying preview data.
This split mirrors the standard cloud-native pattern of a control plane (orchestration and state management) and a data plane (media processing). The difference is that it is implemented with edge primitives intended for operational safety and bounded resources.
2) Session orchestration and resilience
A key detail is Streamline’s session continuity. After a session starts, the controlling Worker can disconnect and later reconnect via a session-resume flow—while the Container keeps running until explicitly stopped or until a maximum duration is reached.
Streamline also addresses container idling behavior. Since Containers can automatically sleep when idle, the design overrides activity expiry behavior so the media pipeline remains active during processing, ensuring that the pipeline isn’t accidentally suspended due to missing control traffic.
3) Protocols chosen for where the platform needs to be fast
- RTMPS for end-to-end live ingest and egress, aligning with how many live broadcasting workflows integrate with streaming platforms.
- HLS for video-on-demand ingestion, leveraging the ecosystem of manifests/segments and enabling subtitle extraction and re-rendering workflows.
- WebSockets for preview delivery with low latency. The system publishes fMP4 fragments from the processing runtime to a Durable Object relay, which then pushes fragments to the client via a WebSocket endpoint.
From an operator standpoint, this implies a practical development workflow: build and iterate on pipeline configurations while previewing changes quickly—without waiting for full production publishing cycles.
4) Media operations as composable building blocks
The configuration model uses an ordered pipeline of operations (e.g., filtering, overlays, subtitle burning, and output encoding). While the underlying engine (currently using FFmpeg) can change, the platform’s exposed API abstracts pipeline intent into stable primitives.
Security and reliability: what
Frequently Asked Questions
How does Streamline avoid keeping every request open while video processing runs for minutes or hours?
Streamline separates orchestration from the media workload. Workers (plus Durable Objects) handle session control and monitoring, while the media engine runs inside a Container for the long-running part. That means the pipeline can continue even if the controlling Worker request ends, avoiding the typical serverless problem where an HTTP request lifetime limits processing.
If the controlling Worker disconnects, what keeps the video pipeline running and how can it be resumed?
Session lifecycle is designed for resilience. Once a session starts, the container-hosted media pipeline keeps processing until it’s explicitly stopped or reaches an enforced maximum duration. The controlling side can reconnect later through a session-resume flow, with Durable Objects maintaining the session context so orchestration can pick up again.
Why are Containers and Durable Objects used instead of implementing everything directly in Workers?
Workers are great for coordination and fast edge logic, but long-running media encoding/decoding is a different operational profile. Containers provide CPU- and memory-stable execution for the media engine, while Durable Objects provide stateful session orchestration. This combination matches the “control plane vs. data plane” reality of video infrastructure.
What happens to the media pipeline if there isn’t constant control traffic—does Container idling stop processing?
Streamline accounts for container idling behavior. Because Containers can sleep when idle, the design overrides activity expiry so the media pipeline isn’t accidentally suspended just because the control plane isn’t continuously “talking” to it. The pipeline stays active during the processing window to keep encoding and transformation steps reliable.
How does Streamline handle security for session control and media credentials within the workflow?
Security is embedded into the pipeline design rather than added later. Streamline includes access control for session management, isolation between principals, and protection of sensitive credentials such as upstream RTMPS keys and preview-publishing credentials. This helps reduce the risk of leaking secrets across sessions and ensures only authorized control can start, modify, or stop processing.
Which streaming protocols does Streamline support, and how are low-latency previews delivered to the browser?
Streamline supports multiple protocol paths: RTMPS for live ingest and output, HLS ingestion from hosted assets, and WebSocket-based preview delivery using fMP4 fragments. That combination lets you publish a processed live output while still giving browser-friendly, low-latency feedback during the session.