Streamexa
一套以 Go 打造的系統,能偵測、分析並下載網頁中的影音串流 (HLS / DASH)。
專案簡介
許多網站以 JavaScript 動態載入、播放器混淆等方式遞送影音,使得串流不易被直接取得。Streamexa 透過自動化瀏覽器分析網頁,辨識其中的影音串流與中繼資料,並以分散式的方式管理下載工作。
這個專案我最感興趣的部分在於它的工程架構:如何用單一 binary 涵蓋伺服器、worker 與 CLI 三種角色,如何把瀏覽器分析做進 Go 程序內,以及如何設計一套安全、可隔離的外掛系統。
架構亮點
- 單一 binary,三種模式 — 同一支
streamexa可切換為server(Web UI + REST API + gRPC 派工)、worker(連線伺服器執行分析與下載)、cli(終端機一次性下載器)。 - In-process CDP 分析器 — 以 go-rod 透過 Chrome DevTools Protocol 驅動 Chromium,分析與分類邏輯(廣告偵測、m3u8 解析、直播判定)以原生 Go 編譯進 binary,不需另外的子程序或 JavaScript 分析器。
- 分散式下載 worker — 多個 worker 可透過 gRPC 連上伺服器並行處理任務,搭配 per-domain 節流與斷路器。
- Vue 3 Web UI — 以 WebSocket 即時更新任務進度、ETA 與狀態,支援亮/暗主題。
- 瀏覽器擴充套件 — 一鍵把目前頁面加入下載佇列 (Chrome / Firefox, Manifest V3)。
外掛系統
第三方可以撰寫 Go 外掛來處理特定網站的擷取流程。外掛系統以 HashiCorp 的
go-plugin(gRPC)為基礎,並提供對外的 plugin SDK:
- 依外掛目錄中的
streamexa-plugin.json自動探索與 URL 比對。 -
白名單式的 action API(
Click、WaitForSelector、PlayVideos、Snapshot、GetResponseBody、Fetch…),外掛無法取得原生 CDP,快照中的認證資訊也會被遮蔽。 - 每個外掛在獨立的程序與瀏覽器工作階段中執行,單一外掛崩潰或逾時不影響其他外掛。
技術棧
開發狀態
主動開發中(v0.1.x 系列),核心分析、外掛系統與分散式下載皆已可運作。
連結
（原始碼 / Demo 連結待補）
Streamexa
A Go system that detects, analyzes, and downloads video streams (HLS / DASH) from web pages.
Overview
Many sites deliver video through JavaScript-heavy loading and obfuscated players, which makes the underlying streams hard to obtain directly. Streamexa drives a real browser to analyze a page, identifies its video streams and metadata, and manages downloads through a distributed set of workers.
The part I find most interesting is the engineering: fitting the server, worker, and CLI roles into a single binary, moving browser analysis in-process, and designing a plugin system that stays safe and well isolated.
Architecture
- Single binary, three modes — one
streamexabinary switches betweenserver(web UI + REST API + gRPC dispatch),worker(connects to the server to analyze and download), andcli(a one-shot terminal downloader). - In-process CDP analyzer — go-rod drives Chromium over the Chrome DevTools Protocol; analysis and classification (ad detection, m3u8 parsing, live-stream detection) are native Go compiled into the binary — no separate subprocess or JavaScript analyzer.
- Distributed download workers — multiple workers connect over gRPC and process tasks concurrently, with per-domain throttling and a circuit breaker.
- Vue 3 web UI — real-time task progress, ETA, and status over WebSocket, with light/dark themes.
- Browser extension — one click to add the current page to the download queue (Chrome / Firefox, Manifest V3).
Plugin System
Third parties can write Go plugins to handle site-specific extraction. The plugin system is
built on HashiCorp's go-plugin (gRPC) and ships a public plugin SDK:
- Automatic discovery and URL matching via a
streamexa-plugin.jsonmanifest. -
A whitelisted action API (
Click,WaitForSelector,PlayVideos,Snapshot,GetResponseBody,Fetch…) — plugins get no raw CDP access, and credentials are redacted from snapshots. - Each plugin runs in its own process and browser session; a crash or timeout in one plugin doesn't affect the others.
Tech Stack
Status
Under active development (the v0.1.x series); core analysis, the plugin system, and distributed downloads all work today.
Links
(Source / demo links to come.)