AeroFetch/ROADMAP-PINCHFLAT.md

# AeroFetch Roadmap — media-manager mode (Pinchflat parity)

The [Seal-parity roadmap](ROADMAP.md) took AeroFetch from "paste a URL → download" to a
full-featured yt-dlp frontend (Phases A–E, complete). This document is a **second, parallel
track**: turning AeroFetch from a one-shot *downloader* into a *media manager* in the spirit
of [Pinchflat](https://github.com/kieraneglin/pinchflat) — download **entire channels**,
keep them **organized into `Channel / Playlist / Title` folders**, and (eventually) **keep
them in sync** as the channel posts new videos.

Nothing here removes or replaces existing behaviour. The single-video / hand-picked-playlist
flow (probe → format pick → queue → history) stays exactly as-is and remains the default. The
media-manager features are **additive**: a new "Library" section feeds the *same* download
queue, concurrency cap, history, error log, cookies, and post-processing options already built.

Sources: [Pinchflat GitHub](https://github.com/kieraneglin/pinchflat) ·
[Pinchflat architecture (DeepWiki)](https://deepwiki.com/kieraneglin/pinchflat) ·
[Pinchflat FAQ — indexing](https://github.com/kieraneglin/pinchflat/wiki/Frequently-Asked-Questions).

> **See also:** [ROADMAP.md](ROADMAP.md) → **Post-parity** — YTDLnis-derived editing features,
> Windows-native polish, and reliability work, Phases L onward.

---

## Implementation status — Phases F–K shipped (2026-06-23)

All six phases are implemented and **rebuild-gated** (`npm run typecheck` + `npm run test`
+ `npm run build` clean after each). The unit suite grew from 76 → **106 tests** (new pure
logic in `indexerCore.ts` + the folder/sidecar args). UI phases were additionally verified
in the Vite browser preview.

| Phase | What shipped | Verification |
| --- | --- | --- |
| **F** | `classifySource`/`buildMediaItems`/`stableSourceId` (`indexerCore.ts`); channel-walk + persist (`indexer.ts`, `sources.ts`); IPC + preload | typecheck · 88 tests · build |
| **G** | `sanitizeDirSegment` + `collectionOutputTemplate` (`buildArgs.ts`); `CollectionContext` threaded through `buildCommand` | typecheck · 97 tests · build |
| **H** | Library sidebar tab + `LibraryView.tsx` + `store/sources.ts`; grouped tree, "Download N pending" → existing queue, live status pills | **preview-verified** |
| **I** | `mergeItemsPreservingState` (re-index keeps downloaded state, drops vanished, counts new); persist-on-complete via `markDownloaded` | **preview-verified** (state survives a queue clear) |
| **J** | Watched sources + RSS fast-check (`sync.ts`), per-source Watch toggle, "Check for new", `autoDownloadNew`; Task Scheduler (`schedule.ts`) + `--sync` launch | UI preview-verified; **OS-level scheduling/RSS fenced — needs a real-install smoke test** (same caveat as the `aerofetch://` wiring in ROADMAP.md) |
| **K** | `.info.json` / thumbnail / `.description` sidecars via `DownloadOptions` + `DownloadOptionsForm` "Sidecar files" group | typecheck · 106 tests · build + preview-verified |

**Deferred sub-items (noted, not yet built):** the power-user raw output-path template
setting (G bullet 3 — current layout is the fixed `Channel/Playlist/NNN - Title`); an
automatic incremental feeder so the live queue self-refills (H — today "Download pending"
enqueues a capped batch of `MAX_ENQUEUE_BATCH`); the retention / quality-upgrade re-download
(I stretch); a true headless quit-when-done for the scheduled `--sync` run (J — today it
launches the normal window unobtrusively and syncs on startup); and Kodi/Jellyfin `.nfo`
generation (K — the three native yt-dlp sidecars ship; `.nfo` needs a custom post-step).

---

## The core idea borrowed from Pinchflat

Pinchflat's insight is to **separate indexing from downloading**, with a persisted index in
between:

1. **Index** a *Source* (a channel or playlist) — enumerate *all* its videos with
   `yt-dlp --flat-playlist` and persist one lightweight record per video, **regardless of any
   filters**. Changing what you want to download never forces a re-index.
2. **Filter** that persisted list to decide what to actually fetch.
3. **Download** each chosen video as its own per-item yt-dlp run, placed into folders by an
   output-path template, with `--download-archive` + a per-item `downloaded` flag preventing
   re-downloads.

This is why "download an entire channel" doesn't have to flood the queue with thousands of
live cards (the failure mode of naïve per-video enumeration) and doesn't need a custom
playlist-progress parser (the failure mode of native `yt-dlp` playlist expansion in one
process): **the channel's full video list lives in a store; the live queue only holds what's
actively downloading.** AeroFetch's per-video progress, concurrency, cancel, retry, and
history all keep working unchanged.

### Data model — Pinchflat → AeroFetch

| Pinchflat (Elixir + SQLite + Oban) | AeroFetch (Electron + JSON store + existing queue) |
| --- | --- |
| **Source** (watched channel/playlist) | `Source` record in a new `sources.json` (mirrors `history.ts`) |
| **Media Item** (one discovered video) | `MediaItem` record (id, title, playlist, index, source id, `downloaded`) |
| **Media Profile** (download preset) | Maps onto existing `DownloadOptions` + `CommandTemplate` + a new output-path template |
| Oban `media_collection_indexing` queue | An async **index job** in main (the probe, persisted) |
| Oban `media_fetching` queue | The existing renderer download queue + concurrency cap |
| Oban `fast_indexing` (RSS) | Phase J — YouTube RSS re-sync |
| `--download-archive` + item state | The existing `--download-archive` setting + `MediaItem.downloaded` |

**Storage decision:** stay with plain-JSON stores (the pattern Phase D deliberately kept over
better-sqlite3 to avoid a native-module build step). A single channel is up to a few thousand
`MediaItem`s — still fine to hold and filter client-side. Revisit better-sqlite3 only if a
user indexes many large channels and the JSON files get unwieldy (noted as a Phase H risk).

---

## Phase F — Channel & playlist indexing (foundation) ✅ COMPLETE

Make AeroFetch able to *enumerate and remember* a whole channel without downloading anything
yet. This is the prerequisite for every later phase.

- [x] **Channel-depth probe.** Extend `src/main/probe.ts` so a channel URL (`/@handle`,
      `/channel/<id>`, `/c/<name>`, `/user/<name>`) resolves to a new `kind: 'channel'`
      `ProbeResult`. Today `buildPlaylist` reads `data.entries` one level deep and ignores the
      nesting a channel returns (the channel's tabs/playlists). Walk channel → playlists →
      videos: probe `…/playlists` (flat) for the playlist list, plus a synthetic **"Uploads"**
      playlist from `…/videos` for videos that belong to no playlist. Lazy-probe each
      playlist's video list on demand rather than all up front.
- [x] **Persisted index.** New `src/main/sources.ts` (plain JSON, mirrors `src/main/history.ts`)
      storing `Source` + `MediaItem` records. New IPC types in `src/shared/ipc.ts`:
      ```ts
      interface Source {
        id: string
        url: string
        kind: 'channel' | 'playlist'
        title: string
        channel?: string
        addedAt: number
        lastIndexedAt?: number
        itemCount: number      // cached MediaItem count for list display
        watched?: boolean      // included in scheduled / startup sync (Phase J)
        feedUrl?: string       // YouTube RSS feed for cheap "anything new?" checks
      }
      interface MediaItem {
        id: string             // globally unique, `${sourceId}:${videoId}`
        sourceId: string
        videoId: string        // yt-dlp video id — the dedup key within a source
        title: string
        url: string
        playlistTitle: string  // for the folder path; 'Uploads' fallback
        playlistIndex: number  // 1-based, for NNN numbering
        durationLabel?: string
        downloaded: boolean
        downloadedAt?: number
        filePath?: string
      }
      ```
      *(This block reflects the shipped `src/shared/ipc.ts` shape; the original sketch
      predated the `videoId`/`itemCount`/`watched`/`feedUrl` fields.)*
- [x] **Dedup on index.** A video appearing in multiple playlists collapses to one `MediaItem`
      (keyed by video id); first playlist seen wins the folder assignment (surface this choice
      in the UI so the user can reassign). The "Uploads" synthetic playlist catches anything in
      no real playlist.
- [x] **Index job.** Async indexing in main (not blocking the UI), pushing progress over a new
      IPC channel (`index:progress`) the way `download.ts` pushes download events — "indexed
      N of M playlists." Reuse `cleanError` and the same `assertHttpUrl` guard.

## Phase G — Folder organization & output-path templates ✅ COMPLETE

Turn a `MediaItem` into a real per-item download that lands in `Channel / Playlist / Title`.

- [x] **Per-item output subpath.** `StartDownloadOptions.outputDir` is *already* plumbed through
      `buildCommand` (`src/main/download.ts`). Add an `outputSubdir` (or reuse `outputDir` with a
      joined subpath) so each `MediaItem` downloads into
      `<root>/<Channel>/<Playlist>/<NNN> - <Title>.ext`. The `NNN` index comes from
      `MediaItem.playlistIndex` (computed at index time) — **not** `%(playlist_index)s`, which is
      empty under the per-item `--no-playlist` path that AeroFetch keeps using.
- [x] **Directory sanitizer.** A small helper in `buildArgs.ts` that strips Windows-illegal
      directory chars (`< > : " / \ | ? *`, trailing dots/spaces, reserved names like `CON`).
      Critical because `--restrict-filenames` only sanitizes the *filename* yt-dlp generates,
      not the directory segments AeroFetch constructs from channel/playlist names.
- [ ] **Output-path setting.** A new Settings → Downloads field for the folder layout, defaulting
      to `Channel / Playlist / Title`, with the raw yt-dlp `-o` template exposed for power users
      (parallels how Phase C exposed raw extra args). Keeps the existing flat
      `filenameTemplate` as the default for non-collection downloads.
- [x] **"Media Profile" = named download preset.** Pinchflat bundles quality + subs + output
      template into a reusable Profile. Implemented: a `MediaProfile` = named bundle of
      `{ kind, quality, options, outputTemplate, extraArgs }` a Source points at via `Source.profileId`.
      Backend mirrors the `CommandTemplate` precedent — `src/main/profiles.ts` on `createJsonStore`
      (`profiles.json`, `PROFILES_MAX` cap, `isProfileLike` guard + `sanitize()`), CRUD IPC
      (`profiles:list/save/remove`) + preload + a `useProfiles` renderer store. Resolution is the pure
      `resolveProfile` in `@shared/profileResolve.ts` (profile → global default, field by field; unit-tested
      in `test/profileResolve.test.ts`), applied in the sources store's `enqueueItems` so a source's
      downloads follow its profile. UI: a `ProfileManager` (mirrors `TemplateManager`) in a new Settings →
      Advanced → **Media profiles** card, plus a per-source **Profile** picker on the Library source detail.
      Live-verified both surfaces render in the preview.

## Phase H — Library view (the media-manager UI) ✅ COMPLETE

The new surface where channels live. Everything below feeds the **existing** download queue.

- [x] **"Library" sidebar section** (`src/renderer/src/components/Sidebar.tsx`) alongside
      Downloads / History / Settings. Lists added Sources.
- [x] **Source detail — playlist/video tree.** A collapsible `Channel → Playlist → Video` tree
      showing each `MediaItem`'s state (indexed · pending · downloading · downloaded · error),
      reusing the checkbox-selection pattern already in `DownloadBar.tsx`'s playlist panel.
      Per-playlist select-all, live counts.
- [x] **"Download all pending" → existing queue, batched.** Enqueue selected `MediaItem`s via
      the existing `addFromUrl` path, **a batch at a time** (e.g. 50) so the queue/store never
      holds the entire channel at once — the live queue stays small while the Library view is the
      source of truth for the full list. Completion flips `MediaItem.downloaded = true` and
      records to history exactly as today.
- [x] **New `useSources` store** (`src/renderer/src/store/sources.ts`, mirrors
      `store/downloads.ts` / `store/history.ts`) plus its IPC bridge. Browser-preview seed data
      so the view is demoable without Electron (same convention as `downloads.ts`'s `PREVIEW`).
- [x] **Risk to watch:** rendering a 2,000-row tree. Virtualize the list (or cap + paginate) and,
      if JSON-store load times bite, revisit the Phase-D better-sqlite3 decision.

## Phase I — Incremental sync & dedup ✅ COMPLETE

Re-running a channel should grab only what's new — the everyday media-manager loop.

- [x] **Re-index = diff.** Re-indexing a Source compares freshly enumerated video ids against the
      persisted `MediaItem`s and marks only the new ones; existing records (and their
      `downloaded` state) are preserved. "X new since last sync" badge.
- [x] **"Download new only."** One action that enqueues just the un-downloaded `MediaItem`s,
      backed by the existing **`--download-archive`** setting (`Settings.downloadArchive`,
      `getDownloadArchivePath()`) as a second, yt-dlp-level dedup guard so even a stale index
      can't re-download.
- [x] **Retention / quality-upgrade (stretch, from Pinchflat).** Optional re-download to upgrade
      quality/metadata, and optional pruning of old items. Implemented as an on-demand per-source
      maintenance panel (`SourceRetention` on the Library source detail), not a persisted schedule
      (matching the "niche for a desktop app" scope). **Prune:** keep the newest N downloaded files,
      delete the older ones' files (the download archive stops them silently re-downloading); shows a
      dry-run count and a two-step confirm before any deletion. **Upgrade:** re-enqueue items whose
      recorded `downloadedQuality` ranks below the source's current target (its profile's quality or
      the global default). The selection logic is pure + injected-now (`@shared/retention.ts`:
      `selectPruneCandidates`/`selectUpgradeCandidates`, `qualityRank` in `@shared/ipc`), unit-tested
      in `test/retention.test.ts`; the impure prune (`pruneMediaItems` in `src/main/sources.ts`)
      deletes files best-effort and clears `downloaded`. `MediaItem.downloadedQuality` is now recorded
      through the completion path. Live-verified the panel renders on an expanded source.

## Phase J — Subscriptions & scheduled auto-download ✅ COMPLETE (OS wiring needs smoke test)

Pinchflat's headline feature: a Source you *watch*, that downloads new uploads on its own.
AeroFetch already has the Windows pieces for this elsewhere in this workspace (Task Scheduler +
single-instance Mutex + a headless run mode — see the DashMail/Weather Radar patterns).

- [x] **Watched sources.** A `Source.watched` flag + an "auto-download new" toggle per source.
- [x] **Fast indexing via RSS.** YouTube exposes a per-channel Atom feed
      (`https://www.youtube.com/feeds/videos.xml?channel_id=<id>`) — a cheap check for new
      uploads without a full yt-dlp scan. **Caveat:** the feed only carries the latest ~15
      videos, so RSS is for *staying current*, not initial indexing (which stays a full
      Phase-F scan). Pinchflat draws the same line ("fast indexing is not recommended for
      playlists / initial scans").
- [x] **Scheduled headless run.** A `--sync` CLI entry that indexes all watched sources and
      enqueues new items, wired to **Windows Task Scheduler** (reuse the
      single-instance-lock already added in Phase E so a scheduled run hands off to a running
      window instead of double-launching). Notify on new downloads via the existing
      `Notification` path.

## Phase K — Media-server output (optional stretch) ✅ COMPLETE

Make AeroFetch's output drop-in for Jellyfin/Plex/Kodi the way Pinchflat does — relevant
because organized channel folders are exactly what those servers ingest.

- [x] **Sidecar metadata.** `--write-info-json`, `--write-thumbnail`, `--write-description`,
      and optional Kodi/Jellyfin `.nfo` generation. Most of this is already expressible through
      Phase C custom-command args; a checkbox group in the output settings makes it first-class.

---

## What we reuse vs. build

**Reuse unchanged:** the download queue + concurrency cap + cancel/retry (`store/downloads.ts`),
history + error log, cookies + proxy + rate-limit + aria2c (`AccessOptions`), all
post-processing `DownloadOptions`, `--download-archive`, native notifications, the
single-instance lock, and the JSON-store pattern.

**Build new:** channel-depth probing (Phase F), the `sources.json` index + `Source`/`MediaItem`
model (F), per-item folder output paths + sanitizer (G), the Library view + `useSources` store
(H), incremental diff-sync (I), and RSS + scheduled sync (J).

## Suggested order

F → G are the foundation and unlock a usable "download whole channel into folders" the moment
they land (drive it from a temporary button before the full Library UI exists). H makes it a
real product surface. I and J turn it into a true media manager. K is polish. Phases F–H are
the high-value core; I–K are incremental and each shippable on its own.