267 lines
18 KiB
Markdown
267 lines
18 KiB
Markdown
# AeroFetch Roadmap — media-manager mode (Pinchflat parity)
|
||
|
||
The [Seal-parity roadmap](ROADMAP.md) took AeroFetch from "paste a URL → download" to a
|
||
full-featured yt-dlp frontend (Phases A–E, complete). This document is a **second, parallel
|
||
track**: turning AeroFetch from a one-shot *downloader* into a *media manager* in the spirit
|
||
of [Pinchflat](https://github.com/kieraneglin/pinchflat) — download **entire channels**,
|
||
keep them **organized into `Channel / Playlist / Title` folders**, and (eventually) **keep
|
||
them in sync** as the channel posts new videos.
|
||
|
||
Nothing here removes or replaces existing behaviour. The single-video / hand-picked-playlist
|
||
flow (probe → format pick → queue → history) stays exactly as-is and remains the default. The
|
||
media-manager features are **additive**: a new "Library" section feeds the *same* download
|
||
queue, concurrency cap, history, error log, cookies, and post-processing options already built.
|
||
|
||
Sources: [Pinchflat GitHub](https://github.com/kieraneglin/pinchflat) ·
|
||
[Pinchflat architecture (DeepWiki)](https://deepwiki.com/kieraneglin/pinchflat) ·
|
||
[Pinchflat FAQ — indexing](https://github.com/kieraneglin/pinchflat/wiki/Frequently-Asked-Questions).
|
||
|
||
> **See also:** [ROADMAP.md](ROADMAP.md) → **Post-parity** — YTDLnis-derived editing features,
|
||
> Windows-native polish, and reliability work, Phases L onward.
|
||
|
||
---
|
||
|
||
## Implementation status — Phases F–K shipped (2026-06-23)
|
||
|
||
All six phases are implemented and **rebuild-gated** (`npm run typecheck` + `npm run test`
|
||
+ `npm run build` clean after each). The unit suite grew from 76 → **106 tests** (new pure
|
||
logic in `indexerCore.ts` + the folder/sidecar args). UI phases were additionally verified
|
||
in the Vite browser preview.
|
||
|
||
| Phase | What shipped | Verification |
|
||
| --- | --- | --- |
|
||
| **F** | `classifySource`/`buildMediaItems`/`stableSourceId` (`indexerCore.ts`); channel-walk + persist (`indexer.ts`, `sources.ts`); IPC + preload | typecheck · 88 tests · build |
|
||
| **G** | `sanitizeDirSegment` + `collectionOutputTemplate` (`buildArgs.ts`); `CollectionContext` threaded through `buildCommand` | typecheck · 97 tests · build |
|
||
| **H** | Library sidebar tab + `LibraryView.tsx` + `store/sources.ts`; grouped tree, "Download N pending" → existing queue, live status pills | **preview-verified** |
|
||
| **I** | `mergeItemsPreservingState` (re-index keeps downloaded state, drops vanished, counts new); persist-on-complete via `markDownloaded` | **preview-verified** (state survives a queue clear) |
|
||
| **J** | Watched sources + RSS fast-check (`sync.ts`), per-source Watch toggle, "Check for new", `autoDownloadNew`; Task Scheduler (`schedule.ts`) + `--sync` launch | UI preview-verified; **OS-level scheduling/RSS fenced — needs a real-install smoke test** (same caveat as the `aerofetch://` wiring in ROADMAP.md) |
|
||
| **K** | `.info.json` / thumbnail / `.description` sidecars via `DownloadOptions` + `DownloadOptionsForm` "Sidecar files" group | typecheck · 106 tests · build + preview-verified |
|
||
|
||
**Deferred sub-items (noted, not yet built):** the power-user raw output-path template
|
||
setting (G bullet 3 — current layout is the fixed `Channel/Playlist/NNN - Title`); an
|
||
automatic incremental feeder so the live queue self-refills (H — today "Download pending"
|
||
enqueues a capped batch of `MAX_ENQUEUE_BATCH`); the retention / quality-upgrade re-download
|
||
(I stretch); a true headless quit-when-done for the scheduled `--sync` run (J — today it
|
||
launches the normal window unobtrusively and syncs on startup); and Kodi/Jellyfin `.nfo`
|
||
generation (K — the three native yt-dlp sidecars ship; `.nfo` needs a custom post-step).
|
||
|
||
---
|
||
|
||
## The core idea borrowed from Pinchflat
|
||
|
||
Pinchflat's insight is to **separate indexing from downloading**, with a persisted index in
|
||
between:
|
||
|
||
1. **Index** a *Source* (a channel or playlist) — enumerate *all* its videos with
|
||
`yt-dlp --flat-playlist` and persist one lightweight record per video, **regardless of any
|
||
filters**. Changing what you want to download never forces a re-index.
|
||
2. **Filter** that persisted list to decide what to actually fetch.
|
||
3. **Download** each chosen video as its own per-item yt-dlp run, placed into folders by an
|
||
output-path template, with `--download-archive` + a per-item `downloaded` flag preventing
|
||
re-downloads.
|
||
|
||
This is why "download an entire channel" doesn't have to flood the queue with thousands of
|
||
live cards (the failure mode of naïve per-video enumeration) and doesn't need a custom
|
||
playlist-progress parser (the failure mode of native `yt-dlp` playlist expansion in one
|
||
process): **the channel's full video list lives in a store; the live queue only holds what's
|
||
actively downloading.** AeroFetch's per-video progress, concurrency, cancel, retry, and
|
||
history all keep working unchanged.
|
||
|
||
### Data model — Pinchflat → AeroFetch
|
||
|
||
| Pinchflat (Elixir + SQLite + Oban) | AeroFetch (Electron + JSON store + existing queue) |
|
||
| --- | --- |
|
||
| **Source** (watched channel/playlist) | `Source` record in a new `sources.json` (mirrors `history.ts`) |
|
||
| **Media Item** (one discovered video) | `MediaItem` record (id, title, playlist, index, source id, `downloaded`) |
|
||
| **Media Profile** (download preset) | Maps onto existing `DownloadOptions` + `CommandTemplate` + a new output-path template |
|
||
| Oban `media_collection_indexing` queue | An async **index job** in main (the probe, persisted) |
|
||
| Oban `media_fetching` queue | The existing renderer download queue + concurrency cap |
|
||
| Oban `fast_indexing` (RSS) | Phase J — YouTube RSS re-sync |
|
||
| `--download-archive` + item state | The existing `--download-archive` setting + `MediaItem.downloaded` |
|
||
|
||
**Storage decision:** stay with plain-JSON stores (the pattern Phase D deliberately kept over
|
||
better-sqlite3 to avoid a native-module build step). A single channel is up to a few thousand
|
||
`MediaItem`s — still fine to hold and filter client-side. Revisit better-sqlite3 only if a
|
||
user indexes many large channels and the JSON files get unwieldy (noted as a Phase H risk).
|
||
|
||
---
|
||
|
||
## Phase F — Channel & playlist indexing (foundation) ✅ COMPLETE
|
||
|
||
Make AeroFetch able to *enumerate and remember* a whole channel without downloading anything
|
||
yet. This is the prerequisite for every later phase.
|
||
|
||
- [x] **Channel-depth probe.** Extend `src/main/probe.ts` so a channel URL (`/@handle`,
|
||
`/channel/<id>`, `/c/<name>`, `/user/<name>`) resolves to a new `kind: 'channel'`
|
||
`ProbeResult`. Today `buildPlaylist` reads `data.entries` one level deep and ignores the
|
||
nesting a channel returns (the channel's tabs/playlists). Walk channel → playlists →
|
||
videos: probe `…/playlists` (flat) for the playlist list, plus a synthetic **"Uploads"**
|
||
playlist from `…/videos` for videos that belong to no playlist. Lazy-probe each
|
||
playlist's video list on demand rather than all up front.
|
||
- [x] **Persisted index.** New `src/main/sources.ts` (plain JSON, mirrors `src/main/history.ts`)
|
||
storing `Source` + `MediaItem` records. New IPC types in `src/shared/ipc.ts`:
|
||
```ts
|
||
interface Source {
|
||
id: string
|
||
url: string
|
||
kind: 'channel' | 'playlist'
|
||
title: string
|
||
channel?: string
|
||
addedAt: number
|
||
lastIndexedAt?: number
|
||
itemCount: number // cached MediaItem count for list display
|
||
watched?: boolean // included in scheduled / startup sync (Phase J)
|
||
feedUrl?: string // YouTube RSS feed for cheap "anything new?" checks
|
||
}
|
||
interface MediaItem {
|
||
id: string // globally unique, `${sourceId}:${videoId}`
|
||
sourceId: string
|
||
videoId: string // yt-dlp video id — the dedup key within a source
|
||
title: string
|
||
url: string
|
||
playlistTitle: string // for the folder path; 'Uploads' fallback
|
||
playlistIndex: number // 1-based, for NNN numbering
|
||
durationLabel?: string
|
||
downloaded: boolean
|
||
downloadedAt?: number
|
||
filePath?: string
|
||
}
|
||
```
|
||
*(This block reflects the shipped `src/shared/ipc.ts` shape; the original sketch
|
||
predated the `videoId`/`itemCount`/`watched`/`feedUrl` fields.)*
|
||
- [x] **Dedup on index.** A video appearing in multiple playlists collapses to one `MediaItem`
|
||
(keyed by video id); first playlist seen wins the folder assignment (surface this choice
|
||
in the UI so the user can reassign). The "Uploads" synthetic playlist catches anything in
|
||
no real playlist.
|
||
- [x] **Index job.** Async indexing in main (not blocking the UI), pushing progress over a new
|
||
IPC channel (`index:progress`) the way `download.ts` pushes download events — "indexed
|
||
N of M playlists." Reuse `cleanError` and the same `assertHttpUrl` guard.
|
||
|
||
## Phase G — Folder organization & output-path templates ✅ COMPLETE
|
||
|
||
Turn a `MediaItem` into a real per-item download that lands in `Channel / Playlist / Title`.
|
||
|
||
- [x] **Per-item output subpath.** `StartDownloadOptions.outputDir` is *already* plumbed through
|
||
`buildCommand` (`src/main/download.ts`). Add an `outputSubdir` (or reuse `outputDir` with a
|
||
joined subpath) so each `MediaItem` downloads into
|
||
`<root>/<Channel>/<Playlist>/<NNN> - <Title>.ext`. The `NNN` index comes from
|
||
`MediaItem.playlistIndex` (computed at index time) — **not** `%(playlist_index)s`, which is
|
||
empty under the per-item `--no-playlist` path that AeroFetch keeps using.
|
||
- [x] **Directory sanitizer.** A small helper in `buildArgs.ts` that strips Windows-illegal
|
||
directory chars (`< > : " / \ | ? *`, trailing dots/spaces, reserved names like `CON`).
|
||
Critical because `--restrict-filenames` only sanitizes the *filename* yt-dlp generates,
|
||
not the directory segments AeroFetch constructs from channel/playlist names.
|
||
- [x] **Output-path setting.** A new Settings → Downloads field for the folder layout, defaulting
|
||
to `Channel / Playlist / Title`, with the raw yt-dlp `-o` template exposed for power users
|
||
(parallels how Phase C exposed raw extra args). Keeps the existing flat
|
||
`filenameTemplate` as the default for non-collection downloads. *Implemented (Batch 26):
|
||
`Settings.collectionOutputTemplate` — blank = the built-in `Channel/Playlist/NNN - Title`
|
||
layout; a raw yt-dlp `-o` template overrides it for collection downloads.
|
||
`collectionOutputTemplate()` (buildArgs) takes the optional custom template; the zod schema
|
||
validates it as a safe relative template; surfaced as the "Collection folder layout" field in
|
||
Settings → Downloads → Filenames. A Media Profile can override it per-source (Batch 27).*
|
||
- [x] **"Media Profile" = named download preset.** Pinchflat bundles quality + subs + output
|
||
template into a reusable Profile. Implemented: a `MediaProfile` = named bundle of
|
||
`{ kind, quality, options, outputTemplate, extraArgs }` a Source points at via `Source.profileId`.
|
||
Backend mirrors the `CommandTemplate` precedent — `src/main/profiles.ts` on `createJsonStore`
|
||
(`profiles.json`, `PROFILES_MAX` cap, `isProfileLike` guard + `sanitize()`), CRUD IPC
|
||
(`profiles:list/save/remove`) + preload + a `useProfiles` renderer store. Resolution is the pure
|
||
`resolveProfile` in `@shared/profileResolve.ts` (profile → global default, field by field; unit-tested
|
||
in `test/profileResolve.test.ts`), applied in the sources store's `enqueueItems` so a source's
|
||
downloads follow its profile. UI: a `ProfileManager` (mirrors `TemplateManager`) in a new Settings →
|
||
Advanced → **Media profiles** card, plus a per-source **Profile** picker on the Library source detail.
|
||
Live-verified both surfaces render in the preview.
|
||
|
||
## Phase H — Library view (the media-manager UI) ✅ COMPLETE
|
||
|
||
The new surface where channels live. Everything below feeds the **existing** download queue.
|
||
|
||
- [x] **"Library" sidebar section** (`src/renderer/src/components/Sidebar.tsx`) alongside
|
||
Downloads / History / Settings. Lists added Sources.
|
||
- [x] **Source detail — playlist/video tree.** A collapsible `Channel → Playlist → Video` tree
|
||
showing each `MediaItem`'s state (indexed · pending · downloading · downloaded · error),
|
||
reusing the checkbox-selection pattern already in `DownloadBar.tsx`'s playlist panel.
|
||
Per-playlist select-all, live counts.
|
||
- [x] **"Download all pending" → existing queue, batched.** Enqueue selected `MediaItem`s via
|
||
the existing `addFromUrl` path, **a batch at a time** (e.g. 50) so the queue/store never
|
||
holds the entire channel at once — the live queue stays small while the Library view is the
|
||
source of truth for the full list. Completion flips `MediaItem.downloaded = true` and
|
||
records to history exactly as today.
|
||
- [x] **New `useSources` store** (`src/renderer/src/store/sources.ts`, mirrors
|
||
`store/downloads.ts` / `store/history.ts`) plus its IPC bridge. Browser-preview seed data
|
||
so the view is demoable without Electron (same convention as `downloads.ts`'s `PREVIEW`).
|
||
- [x] **Risk to watch:** rendering a 2,000-row tree. Virtualize the list (or cap + paginate) and,
|
||
if JSON-store load times bite, revisit the Phase-D better-sqlite3 decision.
|
||
|
||
## Phase I — Incremental sync & dedup ✅ COMPLETE
|
||
|
||
Re-running a channel should grab only what's new — the everyday media-manager loop.
|
||
|
||
- [x] **Re-index = diff.** Re-indexing a Source compares freshly enumerated video ids against the
|
||
persisted `MediaItem`s and marks only the new ones; existing records (and their
|
||
`downloaded` state) are preserved. "X new since last sync" badge.
|
||
- [x] **"Download new only."** One action that enqueues just the un-downloaded `MediaItem`s,
|
||
backed by the existing **`--download-archive`** setting (`Settings.downloadArchive`,
|
||
`getDownloadArchivePath()`) as a second, yt-dlp-level dedup guard so even a stale index
|
||
can't re-download.
|
||
- [x] **Retention / quality-upgrade (stretch, from Pinchflat).** Optional re-download to upgrade
|
||
quality/metadata, and optional pruning of old items. Implemented as an on-demand per-source
|
||
maintenance panel (`SourceRetention` on the Library source detail), not a persisted schedule
|
||
(matching the "niche for a desktop app" scope). **Prune:** keep the newest N downloaded files,
|
||
delete the older ones' files (the download archive stops them silently re-downloading); shows a
|
||
dry-run count and a two-step confirm before any deletion. **Upgrade:** re-enqueue items whose
|
||
recorded `downloadedQuality` ranks below the source's current target (its profile's quality or
|
||
the global default). The selection logic is pure + injected-now (`@shared/retention.ts`:
|
||
`selectPruneCandidates`/`selectUpgradeCandidates`, `qualityRank` in `@shared/ipc`), unit-tested
|
||
in `test/retention.test.ts`; the impure prune (`pruneMediaItems` in `src/main/sources.ts`)
|
||
deletes files best-effort and clears `downloaded`. `MediaItem.downloadedQuality` is now recorded
|
||
through the completion path. Live-verified the panel renders on an expanded source.
|
||
|
||
## Phase J — Subscriptions & scheduled auto-download ✅ COMPLETE (OS wiring needs smoke test)
|
||
|
||
Pinchflat's headline feature: a Source you *watch*, that downloads new uploads on its own.
|
||
AeroFetch already has the Windows pieces for this elsewhere in this workspace (Task Scheduler +
|
||
single-instance Mutex + a headless run mode — see the DashMail/Weather Radar patterns).
|
||
|
||
- [x] **Watched sources.** A `Source.watched` flag + an "auto-download new" toggle per source.
|
||
- [x] **Fast indexing via RSS.** YouTube exposes a per-channel Atom feed
|
||
(`https://www.youtube.com/feeds/videos.xml?channel_id=<id>`) — a cheap check for new
|
||
uploads without a full yt-dlp scan. **Caveat:** the feed only carries the latest ~15
|
||
videos, so RSS is for *staying current*, not initial indexing (which stays a full
|
||
Phase-F scan). Pinchflat draws the same line ("fast indexing is not recommended for
|
||
playlists / initial scans").
|
||
- [x] **Scheduled headless run.** A `--sync` CLI entry that indexes all watched sources and
|
||
enqueues new items, wired to **Windows Task Scheduler** (reuse the
|
||
single-instance-lock already added in Phase E so a scheduled run hands off to a running
|
||
window instead of double-launching). Notify on new downloads via the existing
|
||
`Notification` path.
|
||
|
||
## Phase K — Media-server output (optional stretch) ✅ COMPLETE
|
||
|
||
Make AeroFetch's output drop-in for Jellyfin/Plex/Kodi the way Pinchflat does — relevant
|
||
because organized channel folders are exactly what those servers ingest.
|
||
|
||
- [x] **Sidecar metadata.** `--write-info-json`, `--write-thumbnail`, `--write-description`,
|
||
and optional Kodi/Jellyfin `.nfo` generation. Most of this is already expressible through
|
||
Phase C custom-command args; a checkbox group in the output settings makes it first-class.
|
||
|
||
---
|
||
|
||
## What we reuse vs. build
|
||
|
||
**Reuse unchanged:** the download queue + concurrency cap + cancel/retry (`store/downloads.ts`),
|
||
history + error log, cookies + proxy + rate-limit + aria2c (`AccessOptions`), all
|
||
post-processing `DownloadOptions`, `--download-archive`, native notifications, the
|
||
single-instance lock, and the JSON-store pattern.
|
||
|
||
**Build new:** channel-depth probing (Phase F), the `sources.json` index + `Source`/`MediaItem`
|
||
model (F), per-item folder output paths + sanitizer (G), the Library view + `useSources` store
|
||
(H), incremental diff-sync (I), and RSS + scheduled sync (J).
|
||
|
||
## Suggested order
|
||
|
||
F → G are the foundation and unlock a usable "download whole channel into folders" the moment
|
||
they land (drive it from a temporary button before the full Library UI exists). H makes it a
|
||
real product surface. I and J turn it into a true media manager. K is polish. Phases F–H are
|
||
the high-value core; I–K are incremental and each shippable on its own.
|