AeroFetch/ROADMAP-PINCHFLAT.md

AeroFetch Roadmap — media-manager mode (Pinchflat parity)

The Seal-parity roadmap took AeroFetch from "paste a URL → download" to a full-featured yt-dlp frontend (Phases A–E, complete). This document is a second, parallel track: turning AeroFetch from a one-shot downloader into a media manager in the spirit of Pinchflat — download entire channels, keep them organized into Channel / Playlist / Title folders, and (eventually) keep them in sync as the channel posts new videos.

Nothing here removes or replaces existing behaviour. The single-video / hand-picked-playlist flow (probe → format pick → queue → history) stays exactly as-is and remains the default. The media-manager features are additive: a new "Library" section feeds the same download queue, concurrency cap, history, error log, cookies, and post-processing options already built.

Sources: Pinchflat GitHub · Pinchflat architecture (DeepWiki) · Pinchflat FAQ — indexing.

---

Implementation status — Phases F–K shipped (2026-06-23)

All six phases are implemented and rebuild-gated (npm run typecheck + npm run test

• npm run build clean after each). The unit suite grew from 76 → 106 tests (new pure logic in indexerCore.ts + the folder/sidecar args). UI phases were additionally verified in the Vite browser preview.

Phase	What shipped	Verification
F	classifySource/buildMediaItems/stableSourceId (indexerCore.ts); channel-walk + persist (indexer.ts, sources.ts); IPC + preload	typecheck · 88 tests · build
G	sanitizeDirSegment + collectionOutputTemplate (buildArgs.ts); CollectionContext threaded through buildCommand	typecheck · 97 tests · build
H	Library sidebar tab + LibraryView.tsx + store/sources.ts; grouped tree, "Download N pending" → existing queue, live status pills	preview-verified
I	mergeItemsPreservingState (re-index keeps downloaded state, drops vanished, counts new); persist-on-complete via markDownloaded	preview-verified (state survives a queue clear)
J	Watched sources + RSS fast-check (sync.ts), per-source Watch toggle, "Check for new", autoDownloadNew; Task Scheduler (schedule.ts) + --sync launch	UI preview-verified; OS-level scheduling/RSS fenced — needs a real-install smoke test (same caveat as the aerofetch:// wiring in ROADMAP.md)
K	.info.json / thumbnail / .description sidecars via DownloadOptions + DownloadOptionsForm "Sidecar files" group	typecheck · 106 tests · build + preview-verified

Deferred sub-items (noted, not yet built): the power-user raw output-path template setting (G bullet 3 — current layout is the fixed Channel/Playlist/NNN - Title); an automatic incremental feeder so the live queue self-refills (H — today "Download pending" enqueues a capped batch of MAX_ENQUEUE_BATCH); the retention / quality-upgrade re-download (I stretch); a true headless quit-when-done for the scheduled --sync run (J — today it launches the normal window unobtrusively and syncs on startup); and Kodi/Jellyfin .nfo generation (K — the three native yt-dlp sidecars ship; .nfo needs a custom post-step).

---

The core idea borrowed from Pinchflat

Pinchflat's insight is to separate indexing from downloading, with a persisted index in between:

1. Index a Source (a channel or playlist) — enumerate all its videos with yt-dlp --flat-playlist and persist one lightweight record per video, regardless of any filters. Changing what you want to download never forces a re-index.
2. Filter that persisted list to decide what to actually fetch.
3. Download each chosen video as its own per-item yt-dlp run, placed into folders by an output-path template, with --download-archive + a per-item downloaded flag preventing re-downloads.

This is why "download an entire channel" doesn't have to flood the queue with thousands of live cards (the failure mode of naïve per-video enumeration) and doesn't need a custom playlist-progress parser (the failure mode of native yt-dlp playlist expansion in one process): the channel's full video list lives in a store; the live queue only holds what's actively downloading. AeroFetch's per-video progress, concurrency, cancel, retry, and history all keep working unchanged.

Data model — Pinchflat → AeroFetch

Pinchflat (Elixir + SQLite + Oban)	AeroFetch (Electron + JSON store + existing queue)
Source (watched channel/playlist)	Source record in a new sources.json (mirrors history.ts)
Media Item (one discovered video)	MediaItem record (id, title, playlist, index, source id, downloaded)
Media Profile (download preset)	Maps onto existing DownloadOptions + CommandTemplate + a new output-path template
Oban media_collection_indexing queue	An async index job in main (the probe, persisted)
Oban media_fetching queue	The existing renderer download queue + concurrency cap
Oban fast_indexing (RSS)	Phase J — YouTube RSS re-sync
--download-archive + item state	The existing --download-archive setting + MediaItem.downloaded

Storage decision: stay with plain-JSON stores (the pattern Phase D deliberately kept over better-sqlite3 to avoid a native-module build step). A single channel is up to a few thousand MediaItems — still fine to hold and filter client-side. Revisit better-sqlite3 only if a user indexes many large channels and the JSON files get unwieldy (noted as a Phase H risk).

---

Phase F — Channel & playlist indexing (foundation) ✅ COMPLETE

Make AeroFetch able to enumerate and remember a whole channel without downloading anything yet. This is the prerequisite for every later phase.

• Channel-depth probe. Extend src/main/probe.ts so a channel URL (/@handle, /channel/<id>, /c/<name>, /user/<name>) resolves to a new kind: 'channel' ProbeResult. Today buildPlaylist reads data.entries one level deep and ignores the nesting a channel returns (the channel's tabs/playlists). Walk channel → playlists → videos: probe …/playlists (flat) for the playlist list, plus a synthetic "Uploads" playlist from …/videos for videos that belong to no playlist. Lazy-probe each playlist's video list on demand rather than all up front.
• Persisted index. New src/main/sources.ts (plain JSON, mirrors src/main/history.ts) storing Source + MediaItem records. New IPC types in src/shared/ipc.ts: ts interface Source { id: string url: string kind: 'channel' | 'playlist' title: string channel?: string addedAt: number lastIndexedAt?: number } interface MediaItem { id: string // yt-dlp video id — the dedup key sourceId: string title: string playlistTitle?: string // for the folder path; 'Uploads' fallback playlistIndex?: number // 1-based, for NNN numbering durationLabel?: string downloaded: boolean downloadedAt?: number filePath?: string }
• Dedup on index. A video appearing in multiple playlists collapses to one MediaItem (keyed by video id); first playlist seen wins the folder assignment (surface this choice in the UI so the user can reassign). The "Uploads" synthetic playlist catches anything in no real playlist.
• Index job. Async indexing in main (not blocking the UI), pushing progress over a new IPC channel (index:progress) the way download.ts pushes download events — "indexed N of M playlists." Reuse cleanError and the same assertHttpUrl guard.

Phase G — Folder organization & output-path templates ✅ COMPLETE

Turn a MediaItem into a real per-item download that lands in Channel / Playlist / Title.

• Per-item output subpath. StartDownloadOptions.outputDir is already plumbed through buildCommand (src/main/download.ts). Add an outputSubdir (or reuse outputDir with a joined subpath) so each MediaItem downloads into <root>/<Channel>/<Playlist>/<NNN> - <Title>.ext. The NNN index comes from MediaItem.playlistIndex (computed at index time) — not %(playlist_index)s, which is empty under the per-item --no-playlist path that AeroFetch keeps using.
• Directory sanitizer. A small helper in buildArgs.ts that strips Windows-illegal directory chars (< > : " / \ | ? *, trailing dots/spaces, reserved names like CON). Critical because --restrict-filenames only sanitizes the filename yt-dlp generates, not the directory segments AeroFetch constructs from channel/playlist names.
• Output-path setting. A new Settings → Downloads field for the folder layout, defaulting to Channel / Playlist / Title, with the raw yt-dlp -o template exposed for power users (parallels how Phase C exposed raw extra args). Keeps the existing flat filenameTemplate as the default for non-collection downloads.
• "Media Profile" = named download preset (optional sugar). Pinchflat bundles quality + subs + output template into a reusable Profile. AeroFetch already persists DownloadOptions and named CommandTemplates; a Profile is just a named bundle of { DownloadOptions, outputTemplate, extraArgs } a Source can point at. Worth it once multiple sources want different rules; skip for v1 (use the global defaults).

Phase H — Library view (the media-manager UI) ✅ COMPLETE

The new surface where channels live. Everything below feeds the existing download queue.

• "Library" sidebar section (src/renderer/src/components/Sidebar.tsx) alongside Downloads / History / Settings. Lists added Sources.
• Source detail — playlist/video tree. A collapsible Channel → Playlist → Video tree showing each MediaItem's state (indexed · pending · downloading · downloaded · error), reusing the checkbox-selection pattern already in DownloadBar.tsx's playlist panel. Per-playlist select-all, live counts.
• "Download all pending" → existing queue, batched. Enqueue selected MediaItems via the existing addFromUrl path, a batch at a time (e.g. 50) so the queue/store never holds the entire channel at once — the live queue stays small while the Library view is the source of truth for the full list. Completion flips MediaItem.downloaded = true and records to history exactly as today.
• New useSources store (src/renderer/src/store/sources.ts, mirrors store/downloads.ts / store/history.ts) plus its IPC bridge. Browser-preview seed data so the view is demoable without Electron (same convention as downloads.ts's PREVIEW).
• Risk to watch: rendering a 2,000-row tree. Virtualize the list (or cap + paginate) and, if JSON-store load times bite, revisit the Phase-D better-sqlite3 decision.

Phase I — Incremental sync & dedup ✅ COMPLETE

Re-running a channel should grab only what's new — the everyday media-manager loop.

• Re-index = diff. Re-indexing a Source compares freshly enumerated video ids against the persisted MediaItems and marks only the new ones; existing records (and their downloaded state) are preserved. "X new since last sync" badge.
• "Download new only." One action that enqueues just the un-downloaded MediaItems, backed by the existing --download-archive setting (Settings.downloadArchive, getDownloadArchivePath()) as a second, yt-dlp-level dedup guard so even a stale index can't re-download.
• Retention / quality-upgrade (stretch, from Pinchflat). Optional re-download to upgrade quality/metadata, and optional pruning of old items. Defer — niche for a desktop app.

Phase J — Subscriptions & scheduled auto-download ✅ COMPLETE (OS wiring needs smoke test)

Pinchflat's headline feature: a Source you watch, that downloads new uploads on its own. AeroFetch already has the Windows pieces for this elsewhere in this workspace (Task Scheduler + single-instance Mutex + a headless run mode — see the DashMail/Weather Radar patterns).

• Watched sources. A Source.watched flag + an "auto-download new" toggle per source.
• Fast indexing via RSS. YouTube exposes a per-channel Atom feed (https://www.youtube.com/feeds/videos.xml?channel_id=<id>) — a cheap check for new uploads without a full yt-dlp scan. Caveat: the feed only carries the latest ~15 videos, so RSS is for staying current, not initial indexing (which stays a full Phase-F scan). Pinchflat draws the same line ("fast indexing is not recommended for playlists / initial scans").
• Scheduled headless run. A --sync CLI entry that indexes all watched sources and enqueues new items, wired to Windows Task Scheduler (reuse the single-instance-lock already added in Phase E so a scheduled run hands off to a running window instead of double-launching). Notify on new downloads via the existing Notification path.

Phase K — Media-server output (optional stretch) ✅ COMPLETE

Make AeroFetch's output drop-in for Jellyfin/Plex/Kodi the way Pinchflat does — relevant because organized channel folders are exactly what those servers ingest.

• Sidecar metadata. --write-info-json, --write-thumbnail, --write-description, and optional Kodi/Jellyfin .nfo generation. Most of this is already expressible through Phase C custom-command args; a checkbox group in the output settings makes it first-class.

---

What we reuse vs. build

Reuse unchanged: the download queue + concurrency cap + cancel/retry (store/downloads.ts), history + error log, cookies + proxy + rate-limit + aria2c (AccessOptions), all post-processing DownloadOptions, --download-archive, native notifications, the single-instance lock, and the JSON-store pattern.

Build new: channel-depth probing (Phase F), the sources.json index + Source/MediaItem model (F), per-item folder output paths + sanitizer (G), the Library view + useSources store (H), incremental diff-sync (I), and RSS + scheduled sync (J).

Suggested order

F → G are the foundation and unlock a usable "download whole channel into folders" the moment they land (drive it from a temporary button before the full Library UI exists). H makes it a real product surface. I and J turn it into a true media manager. K is polish. Phases F–H are the high-value core; I–K are incremental and each shippable on its own.