debont80/AeroFetch
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 9 Wiki Activity

Add Pinchflat-style media manager: index channels into playlist folders

Browse Source 
Index an entire YouTube channel or playlist once, then download it into
organized <Channel>/<Playlist>/<NNN> - <Title> folders, with incremental
re-sync. Built in six rebuild-gated phases (F-K); see ROADMAP-PINCHFLAT.md.

- F: channel-walk indexer (/playlists + /videos) -> persisted Source +
  MediaItem JSON stores; pure classify/dedup logic in indexerCore.ts
- G: Windows-safe folder paths + dir sanitizer (collectionOutputTemplate)
- H: Library tab + source tree + "Download pending" into the existing queue
- I: state-preserving re-index merge + persist-downloaded-on-complete
- J: watched sources, RSS fast-check, Task Scheduler + --sync launch
  (the OS-level scheduling/RSS need a real-install smoke test)
- K: .info.json / thumbnail / .description sidecars for media servers

Also gitignore .gitea-token. 106 unit tests pass; typecheck + build clean.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Devon Bontrager
2026-06-23 20:50:13 -04:00
parent 81b97ea429
commit 47da3e6746
26 changed files with 2529 additions and 46 deletions
Download Patch File Download Diff File Expand all files Collapse all files
ROADMAP-PINCHFLAT.md
+235
View File
@@ -0,0 +1,235 @@
# AeroFetch Roadmap — media-manager mode (Pinchflat parity)
The [Seal-parity roadmap](ROADMAP.md) took AeroFetch from "paste a URL → download" to a
full-featured yt-dlp frontend (Phases A–E, complete). This document is a **second, parallel
track**: turning AeroFetch from a one-shot *downloader* into a *media manager* in the spirit
of [Pinchflat](https://github.com/kieraneglin/pinchflat) — download **entire channels**,
keep them **organized into `Channel / Playlist / Title` folders**, and (eventually) **keep
them in sync** as the channel posts new videos.
Nothing here removes or replaces existing behaviour. The single-video / hand-picked-playlist
flow (probe → format pick → queue → history) stays exactly as-is and remains the default. The
media-manager features are **additive**: a new "Library" section feeds the *same* download
queue, concurrency cap, history, error log, cookies, and post-processing options already built.
Sources: [Pinchflat GitHub](https://github.com/kieraneglin/pinchflat) ·
[Pinchflat architecture (DeepWiki)](https://deepwiki.com/kieraneglin/pinchflat) ·
[Pinchflat FAQ — indexing](https://github.com/kieraneglin/pinchflat/wiki/Frequently-Asked-Questions).
---
## Implementation status — Phases F–K shipped (2026-06-23)
All six phases are implemented and **rebuild-gated** (`npm run typecheck` + `npm run test`
+ `npm run build` clean after each). The unit suite grew from 76 → **106 tests** (new pure
logic in `indexerCore.ts` + the folder/sidecar args). UI phases were additionally verified
in the Vite browser preview.
| Phase | What shipped | Verification |
| --- | --- | --- |
| **F** | `classifySource`/`buildMediaItems`/`stableSourceId` (`indexerCore.ts`); channel-walk + persist (`indexer.ts`, `sources.ts`); IPC + preload | typecheck · 88 tests · build |
| **G** | `sanitizeDirSegment` + `collectionOutputTemplate` (`buildArgs.ts`); `CollectionContext` threaded through `buildCommand` | typecheck · 97 tests · build |
| **H** | Library sidebar tab + `LibraryView.tsx` + `store/sources.ts`; grouped tree, "Download N pending" → existing queue, live status pills | **preview-verified** |
| **I** | `mergeItemsPreservingState` (re-index keeps downloaded state, drops vanished, counts new); persist-on-complete via `markDownloaded` | **preview-verified** (state survives a queue clear) |
| **J** | Watched sources + RSS fast-check (`sync.ts`), per-source Watch toggle, "Check for new", `autoDownloadNew`; Task Scheduler (`schedule.ts`) + `--sync` launch | UI preview-verified; **OS-level scheduling/RSS fenced — needs a real-install smoke test** (same caveat as the `aerofetch://` wiring in ROADMAP.md) |
| **K** | `.info.json` / thumbnail / `.description` sidecars via `DownloadOptions` + `DownloadOptionsForm` "Sidecar files" group | typecheck · 106 tests · build + preview-verified |
**Deferred sub-items (noted, not yet built):** the power-user raw output-path template
setting (G bullet 3 — current layout is the fixed `Channel/Playlist/NNN - Title`); an
automatic incremental feeder so the live queue self-refills (H — today "Download pending"
enqueues a capped batch of `MAX_ENQUEUE_BATCH`); the retention / quality-upgrade re-download
(I stretch); a true headless quit-when-done for the scheduled `--sync` run (J — today it
launches the normal window unobtrusively and syncs on startup); and Kodi/Jellyfin `.nfo`
generation (K — the three native yt-dlp sidecars ship; `.nfo` needs a custom post-step).
---
## The core idea borrowed from Pinchflat
Pinchflat's insight is to **separate indexing from downloading**, with a persisted index in
between:
1. **Index** a *Source* (a channel or playlist) — enumerate *all* its videos with
`yt-dlp --flat-playlist` and persist one lightweight record per video, **regardless of any
filters**. Changing what you want to download never forces a re-index.
2. **Filter** that persisted list to decide what to actually fetch.
3. **Download** each chosen video as its own per-item yt-dlp run, placed into folders by an
output-path template, with `--download-archive` + a per-item `downloaded` flag preventing
re-downloads.
This is why "download an entire channel" doesn't have to flood the queue with thousands of
live cards (the failure mode of naïve per-video enumeration) and doesn't need a custom
playlist-progress parser (the failure mode of native `yt-dlp` playlist expansion in one
process): **the channel's full video list lives in a store; the live queue only holds what's
actively downloading.** AeroFetch's per-video progress, concurrency, cancel, retry, and
history all keep working unchanged.
### Data model — Pinchflat → AeroFetch
| Pinchflat (Elixir + SQLite + Oban) | AeroFetch (Electron + JSON store + existing queue) |
| --- | --- |
| **Source** (watched channel/playlist) | `Source` record in a new `sources.json` (mirrors `history.ts`) |
| **Media Item** (one discovered video) | `MediaItem` record (id, title, playlist, index, source id, `downloaded`) |
| **Media Profile** (download preset) | Maps onto existing `DownloadOptions` + `CommandTemplate` + a new output-path template |
| Oban `media_collection_indexing` queue | An async **index job** in main (the probe, persisted) |
| Oban `media_fetching` queue | The existing renderer download queue + concurrency cap |
| Oban `fast_indexing` (RSS) | Phase J — YouTube RSS re-sync |
| `--download-archive` + item state | The existing `--download-archive` setting + `MediaItem.downloaded` |
**Storage decision:** stay with plain-JSON stores (the pattern Phase D deliberately kept over
better-sqlite3 to avoid a native-module build step). A single channel is up to a few thousand
`MediaItem`s — still fine to hold and filter client-side. Revisit better-sqlite3 only if a
user indexes many large channels and the JSON files get unwieldy (noted as a Phase H risk).
---
## Phase F — Channel & playlist indexing (foundation) ✅ COMPLETE
Make AeroFetch able to *enumerate and remember* a whole channel without downloading anything
yet. This is the prerequisite for every later phase.
- [ ] **Channel-depth probe.** Extend `src/main/probe.ts` so a channel URL (`/@handle`,
`/channel/<id>`, `/c/<name>`, `/user/<name>`) resolves to a new `kind: 'channel'`
`ProbeResult`. Today `buildPlaylist` reads `data.entries` one level deep and ignores the
nesting a channel returns (the channel's tabs/playlists). Walk channel → playlists →
videos: probe `…/playlists` (flat) for the playlist list, plus a synthetic **"Uploads"**
playlist from `…/videos` for videos that belong to no playlist. Lazy-probe each
playlist's video list on demand rather than all up front.
- [ ] **Persisted index.** New `src/main/sources.ts` (plain JSON, mirrors `src/main/history.ts`)
storing `Source` + `MediaItem` records. New IPC types in `src/shared/ipc.ts`:
```ts
interface Source {
id: string
url: string
kind: 'channel' | 'playlist'
title: string
channel?: string
addedAt: number
lastIndexedAt?: number
}
interface MediaItem {
id: string // yt-dlp video id — the dedup key
sourceId: string
title: string
playlistTitle?: string // for the folder path; 'Uploads' fallback
playlistIndex?: number // 1-based, for NNN numbering
durationLabel?: string
downloaded: boolean
downloadedAt?: number
filePath?: string
}
```
- [ ] **Dedup on index.** A video appearing in multiple playlists collapses to one `MediaItem`
(keyed by video id); first playlist seen wins the folder assignment (surface this choice
in the UI so the user can reassign). The "Uploads" synthetic playlist catches anything in
no real playlist.
- [ ] **Index job.** Async indexing in main (not blocking the UI), pushing progress over a new
IPC channel (`index:progress`) the way `download.ts` pushes download events — "indexed
N of M playlists." Reuse `cleanError` and the same `assertHttpUrl` guard.
## Phase G — Folder organization & output-path templates ✅ COMPLETE
Turn a `MediaItem` into a real per-item download that lands in `Channel / Playlist / Title`.
- [ ] **Per-item output subpath.** `StartDownloadOptions.outputDir` is *already* plumbed through
`buildCommand` (`src/main/download.ts`). Add an `outputSubdir` (or reuse `outputDir` with a
joined subpath) so each `MediaItem` downloads into
`<root>/<Channel>/<Playlist>/<NNN> - <Title>.ext`. The `NNN` index comes from
`MediaItem.playlistIndex` (computed at index time) — **not** `%(playlist_index)s`, which is
empty under the per-item `--no-playlist` path that AeroFetch keeps using.
- [ ] **Directory sanitizer.** A small helper in `buildArgs.ts` that strips Windows-illegal
directory chars (`< > : " / \ | ? *`, trailing dots/spaces, reserved names like `CON`).
Critical because `--restrict-filenames` only sanitizes the *filename* yt-dlp generates,
not the directory segments AeroFetch constructs from channel/playlist names.
- [ ] **Output-path setting.** A new Settings → Downloads field for the folder layout, defaulting
to `Channel / Playlist / Title`, with the raw yt-dlp `-o` template exposed for power users
(parallels how Phase C exposed raw extra args). Keeps the existing flat
`filenameTemplate` as the default for non-collection downloads.
- [ ] **"Media Profile" = named download preset (optional sugar).** Pinchflat bundles
quality + subs + output template into a reusable Profile. AeroFetch already persists
`DownloadOptions` and named `CommandTemplate`s; a Profile is just a named bundle of
`{ DownloadOptions, outputTemplate, extraArgs }` a Source can point at. Worth it once
multiple sources want different rules; skip for v1 (use the global defaults).
## Phase H — Library view (the media-manager UI) ✅ COMPLETE
The new surface where channels live. Everything below feeds the **existing** download queue.
- [ ] **"Library" sidebar section** (`src/renderer/src/components/Sidebar.tsx`) alongside
Downloads / History / Settings. Lists added Sources.
- [ ] **Source detail — playlist/video tree.** A collapsible `Channel → Playlist → Video` tree
showing each `MediaItem`'s state (indexed · pending · downloading · downloaded · error),
reusing the checkbox-selection pattern already in `DownloadBar.tsx`'s playlist panel.
Per-playlist select-all, live counts.
- [ ] **"Download all pending" → existing queue, batched.** Enqueue selected `MediaItem`s via
the existing `addFromUrl` path, **a batch at a time** (e.g. 50) so the queue/store never
holds the entire channel at once — the live queue stays small while the Library view is the
source of truth for the full list. Completion flips `MediaItem.downloaded = true` and
records to history exactly as today.
- [ ] **New `useSources` store** (`src/renderer/src/store/sources.ts`, mirrors
`store/downloads.ts` / `store/history.ts`) plus its IPC bridge. Browser-preview seed data
so the view is demoable without Electron (same convention as `downloads.ts`'s `PREVIEW`).
- [ ] **Risk to watch:** rendering a 2,000-row tree. Virtualize the list (or cap + paginate) and,
if JSON-store load times bite, revisit the Phase-D better-sqlite3 decision.
## Phase I — Incremental sync & dedup ✅ COMPLETE
Re-running a channel should grab only what's new — the everyday media-manager loop.
- [ ] **Re-index = diff.** Re-indexing a Source compares freshly enumerated video ids against the
persisted `MediaItem`s and marks only the new ones; existing records (and their
`downloaded` state) are preserved. "X new since last sync" badge.
- [ ] **"Download new only."** One action that enqueues just the un-downloaded `MediaItem`s,
backed by the existing **`--download-archive`** setting (`Settings.downloadArchive`,
`getDownloadArchivePath()`) as a second, yt-dlp-level dedup guard so even a stale index
can't re-download.
- [ ] **Retention / quality-upgrade (stretch, from Pinchflat).** Optional re-download to upgrade
quality/metadata, and optional pruning of old items. Defer — niche for a desktop app.
## Phase J — Subscriptions & scheduled auto-download ✅ COMPLETE (OS wiring needs smoke test)
Pinchflat's headline feature: a Source you *watch*, that downloads new uploads on its own.
AeroFetch already has the Windows pieces for this elsewhere in this workspace (Task Scheduler +
single-instance Mutex + a headless run mode — see the DashMail/Weather Radar patterns).
- [ ] **Watched sources.** A `Source.watched` flag + an "auto-download new" toggle per source.
- [ ] **Fast indexing via RSS.** YouTube exposes a per-channel Atom feed
(`https://www.youtube.com/feeds/videos.xml?channel_id=<id>`) — a cheap check for new
uploads without a full yt-dlp scan. **Caveat:** the feed only carries the latest ~15
videos, so RSS is for *staying current*, not initial indexing (which stays a full
Phase-F scan). Pinchflat draws the same line ("fast indexing is not recommended for
playlists / initial scans").
- [ ] **Scheduled headless run.** A `--sync` CLI entry that indexes all watched sources and
enqueues new items, wired to **Windows Task Scheduler** (reuse the
single-instance-lock already added in Phase E so a scheduled run hands off to a running
window instead of double-launching). Notify on new downloads via the existing
`Notification` path.
## Phase K — Media-server output (optional stretch) ✅ COMPLETE
Make AeroFetch's output drop-in for Jellyfin/Plex/Kodi the way Pinchflat does — relevant
because organized channel folders are exactly what those servers ingest.
- [ ] **Sidecar metadata.** `--write-info-json`, `--write-thumbnail`, `--write-description`,
and optional Kodi/Jellyfin `.nfo` generation. Most of this is already expressible through
Phase C custom-command args; a checkbox group in the output settings makes it first-class.
---
## What we reuse vs. build
**Reuse unchanged:** the download queue + concurrency cap + cancel/retry (`store/downloads.ts`),
history + error log, cookies + proxy + rate-limit + aria2c (`AccessOptions`), all
post-processing `DownloadOptions`, `--download-archive`, native notifications, the
single-instance lock, and the JSON-store pattern.
**Build new:** channel-depth probing (Phase F), the `sources.json` index + `Source`/`MediaItem`
model (F), per-item folder output paths + sanitizer (G), the Library view + `useSources` store
(H), incremental diff-sync (I), and RSS + scheduled sync (J).
## Suggested order
F → G are the foundation and unlock a usable "download whole channel into folders" the moment
they land (drive it from a temporary button before the full Library UI exists). H makes it a
real product surface. I and J turn it into a true media manager. K is polish. Phases F–H are
the high-value core; I–K are incremental and each shippable on its own.