diff --git a/.gitignore b/.gitignore index 016ef81..7d98ca0 100644 --- a/.gitignore +++ b/.gitignore @@ -7,3 +7,6 @@ dist # Local screen-capture frames (not source) .frames/ + +# Secrets — never commit +.gitea-token diff --git a/ROADMAP-PINCHFLAT.md b/ROADMAP-PINCHFLAT.md new file mode 100644 index 0000000..999dc26 --- /dev/null +++ b/ROADMAP-PINCHFLAT.md @@ -0,0 +1,235 @@ +# AeroFetch Roadmap — media-manager mode (Pinchflat parity) + +The [Seal-parity roadmap](ROADMAP.md) took AeroFetch from "paste a URL → download" to a +full-featured yt-dlp frontend (Phases A–E, complete). This document is a **second, parallel +track**: turning AeroFetch from a one-shot *downloader* into a *media manager* in the spirit +of [Pinchflat](https://github.com/kieraneglin/pinchflat) — download **entire channels**, +keep them **organized into `Channel / Playlist / Title` folders**, and (eventually) **keep +them in sync** as the channel posts new videos. + +Nothing here removes or replaces existing behaviour. The single-video / hand-picked-playlist +flow (probe → format pick → queue → history) stays exactly as-is and remains the default. The +media-manager features are **additive**: a new "Library" section feeds the *same* download +queue, concurrency cap, history, error log, cookies, and post-processing options already built. + +Sources: [Pinchflat GitHub](https://github.com/kieraneglin/pinchflat) · +[Pinchflat architecture (DeepWiki)](https://deepwiki.com/kieraneglin/pinchflat) · +[Pinchflat FAQ — indexing](https://github.com/kieraneglin/pinchflat/wiki/Frequently-Asked-Questions). + +--- + +## Implementation status — Phases F–K shipped (2026-06-23) + +All six phases are implemented and **rebuild-gated** (`npm run typecheck` + `npm run test` ++ `npm run build` clean after each). The unit suite grew from 76 → **106 tests** (new pure +logic in `indexerCore.ts` + the folder/sidecar args). UI phases were additionally verified +in the Vite browser preview. + +| Phase | What shipped | Verification | +| --- | --- | --- | +| **F** | `classifySource`/`buildMediaItems`/`stableSourceId` (`indexerCore.ts`); channel-walk + persist (`indexer.ts`, `sources.ts`); IPC + preload | typecheck · 88 tests · build | +| **G** | `sanitizeDirSegment` + `collectionOutputTemplate` (`buildArgs.ts`); `CollectionContext` threaded through `buildCommand` | typecheck · 97 tests · build | +| **H** | Library sidebar tab + `LibraryView.tsx` + `store/sources.ts`; grouped tree, "Download N pending" → existing queue, live status pills | **preview-verified** | +| **I** | `mergeItemsPreservingState` (re-index keeps downloaded state, drops vanished, counts new); persist-on-complete via `markDownloaded` | **preview-verified** (state survives a queue clear) | +| **J** | Watched sources + RSS fast-check (`sync.ts`), per-source Watch toggle, "Check for new", `autoDownloadNew`; Task Scheduler (`schedule.ts`) + `--sync` launch | UI preview-verified; **OS-level scheduling/RSS fenced — needs a real-install smoke test** (same caveat as the `aerofetch://` wiring in ROADMAP.md) | +| **K** | `.info.json` / thumbnail / `.description` sidecars via `DownloadOptions` + `DownloadOptionsForm` "Sidecar files" group | typecheck · 106 tests · build + preview-verified | + +**Deferred sub-items (noted, not yet built):** the power-user raw output-path template +setting (G bullet 3 — current layout is the fixed `Channel/Playlist/NNN - Title`); an +automatic incremental feeder so the live queue self-refills (H — today "Download pending" +enqueues a capped batch of `MAX_ENQUEUE_BATCH`); the retention / quality-upgrade re-download +(I stretch); a true headless quit-when-done for the scheduled `--sync` run (J — today it +launches the normal window unobtrusively and syncs on startup); and Kodi/Jellyfin `.nfo` +generation (K — the three native yt-dlp sidecars ship; `.nfo` needs a custom post-step). + +--- + +## The core idea borrowed from Pinchflat + +Pinchflat's insight is to **separate indexing from downloading**, with a persisted index in +between: + +1. **Index** a *Source* (a channel or playlist) — enumerate *all* its videos with + `yt-dlp --flat-playlist` and persist one lightweight record per video, **regardless of any + filters**. Changing what you want to download never forces a re-index. +2. **Filter** that persisted list to decide what to actually fetch. +3. **Download** each chosen video as its own per-item yt-dlp run, placed into folders by an + output-path template, with `--download-archive` + a per-item `downloaded` flag preventing + re-downloads. + +This is why "download an entire channel" doesn't have to flood the queue with thousands of +live cards (the failure mode of naïve per-video enumeration) and doesn't need a custom +playlist-progress parser (the failure mode of native `yt-dlp` playlist expansion in one +process): **the channel's full video list lives in a store; the live queue only holds what's +actively downloading.** AeroFetch's per-video progress, concurrency, cancel, retry, and +history all keep working unchanged. + +### Data model — Pinchflat → AeroFetch + +| Pinchflat (Elixir + SQLite + Oban) | AeroFetch (Electron + JSON store + existing queue) | +| --- | --- | +| **Source** (watched channel/playlist) | `Source` record in a new `sources.json` (mirrors `history.ts`) | +| **Media Item** (one discovered video) | `MediaItem` record (id, title, playlist, index, source id, `downloaded`) | +| **Media Profile** (download preset) | Maps onto existing `DownloadOptions` + `CommandTemplate` + a new output-path template | +| Oban `media_collection_indexing` queue | An async **index job** in main (the probe, persisted) | +| Oban `media_fetching` queue | The existing renderer download queue + concurrency cap | +| Oban `fast_indexing` (RSS) | Phase J — YouTube RSS re-sync | +| `--download-archive` + item state | The existing `--download-archive` setting + `MediaItem.downloaded` | + +**Storage decision:** stay with plain-JSON stores (the pattern Phase D deliberately kept over +better-sqlite3 to avoid a native-module build step). A single channel is up to a few thousand +`MediaItem`s — still fine to hold and filter client-side. Revisit better-sqlite3 only if a +user indexes many large channels and the JSON files get unwieldy (noted as a Phase H risk). + +--- + +## Phase F — Channel & playlist indexing (foundation) ✅ COMPLETE + +Make AeroFetch able to *enumerate and remember* a whole channel without downloading anything +yet. This is the prerequisite for every later phase. + +- [ ] **Channel-depth probe.** Extend `src/main/probe.ts` so a channel URL (`/@handle`, + `/channel/`, `/c/`, `/user/`) resolves to a new `kind: 'channel'` + `ProbeResult`. Today `buildPlaylist` reads `data.entries` one level deep and ignores the + nesting a channel returns (the channel's tabs/playlists). Walk channel → playlists → + videos: probe `…/playlists` (flat) for the playlist list, plus a synthetic **"Uploads"** + playlist from `…/videos` for videos that belong to no playlist. Lazy-probe each + playlist's video list on demand rather than all up front. +- [ ] **Persisted index.** New `src/main/sources.ts` (plain JSON, mirrors `src/main/history.ts`) + storing `Source` + `MediaItem` records. New IPC types in `src/shared/ipc.ts`: + ```ts + interface Source { + id: string + url: string + kind: 'channel' | 'playlist' + title: string + channel?: string + addedAt: number + lastIndexedAt?: number + } + interface MediaItem { + id: string // yt-dlp video id — the dedup key + sourceId: string + title: string + playlistTitle?: string // for the folder path; 'Uploads' fallback + playlistIndex?: number // 1-based, for NNN numbering + durationLabel?: string + downloaded: boolean + downloadedAt?: number + filePath?: string + } + ``` +- [ ] **Dedup on index.** A video appearing in multiple playlists collapses to one `MediaItem` + (keyed by video id); first playlist seen wins the folder assignment (surface this choice + in the UI so the user can reassign). The "Uploads" synthetic playlist catches anything in + no real playlist. +- [ ] **Index job.** Async indexing in main (not blocking the UI), pushing progress over a new + IPC channel (`index:progress`) the way `download.ts` pushes download events — "indexed + N of M playlists." Reuse `cleanError` and the same `assertHttpUrl` guard. + +## Phase G — Folder organization & output-path templates ✅ COMPLETE + +Turn a `MediaItem` into a real per-item download that lands in `Channel / Playlist / Title`. + +- [ ] **Per-item output subpath.** `StartDownloadOptions.outputDir` is *already* plumbed through + `buildCommand` (`src/main/download.ts`). Add an `outputSubdir` (or reuse `outputDir` with a + joined subpath) so each `MediaItem` downloads into + `/// - .ext`. The `NNN` index comes from + `MediaItem.playlistIndex` (computed at index time) — **not** `%(playlist_index)s`, which is + empty under the per-item `--no-playlist` path that AeroFetch keeps using. +- [ ] **Directory sanitizer.** A small helper in `buildArgs.ts` that strips Windows-illegal + directory chars (`< > : " / \ | ? *`, trailing dots/spaces, reserved names like `CON`). + Critical because `--restrict-filenames` only sanitizes the *filename* yt-dlp generates, + not the directory segments AeroFetch constructs from channel/playlist names. +- [ ] **Output-path setting.** A new Settings → Downloads field for the folder layout, defaulting + to `Channel / Playlist / Title`, with the raw yt-dlp `-o` template exposed for power users + (parallels how Phase C exposed raw extra args). Keeps the existing flat + `filenameTemplate` as the default for non-collection downloads. +- [ ] **"Media Profile" = named download preset (optional sugar).** Pinchflat bundles + quality + subs + output template into a reusable Profile. AeroFetch already persists + `DownloadOptions` and named `CommandTemplate`s; a Profile is just a named bundle of + `{ DownloadOptions, outputTemplate, extraArgs }` a Source can point at. Worth it once + multiple sources want different rules; skip for v1 (use the global defaults). + +## Phase H — Library view (the media-manager UI) ✅ COMPLETE + +The new surface where channels live. Everything below feeds the **existing** download queue. + +- [ ] **"Library" sidebar section** (`src/renderer/src/components/Sidebar.tsx`) alongside + Downloads / History / Settings. Lists added Sources. +- [ ] **Source detail — playlist/video tree.** A collapsible `Channel → Playlist → Video` tree + showing each `MediaItem`'s state (indexed · pending · downloading · downloaded · error), + reusing the checkbox-selection pattern already in `DownloadBar.tsx`'s playlist panel. + Per-playlist select-all, live counts. +- [ ] **"Download all pending" → existing queue, batched.** Enqueue selected `MediaItem`s via + the existing `addFromUrl` path, **a batch at a time** (e.g. 50) so the queue/store never + holds the entire channel at once — the live queue stays small while the Library view is the + source of truth for the full list. Completion flips `MediaItem.downloaded = true` and + records to history exactly as today. +- [ ] **New `useSources` store** (`src/renderer/src/store/sources.ts`, mirrors + `store/downloads.ts` / `store/history.ts`) plus its IPC bridge. Browser-preview seed data + so the view is demoable without Electron (same convention as `downloads.ts`'s `PREVIEW`). +- [ ] **Risk to watch:** rendering a 2,000-row tree. Virtualize the list (or cap + paginate) and, + if JSON-store load times bite, revisit the Phase-D better-sqlite3 decision. + +## Phase I — Incremental sync & dedup ✅ COMPLETE + +Re-running a channel should grab only what's new — the everyday media-manager loop. + +- [ ] **Re-index = diff.** Re-indexing a Source compares freshly enumerated video ids against the + persisted `MediaItem`s and marks only the new ones; existing records (and their + `downloaded` state) are preserved. "X new since last sync" badge. +- [ ] **"Download new only."** One action that enqueues just the un-downloaded `MediaItem`s, + backed by the existing **`--download-archive`** setting (`Settings.downloadArchive`, + `getDownloadArchivePath()`) as a second, yt-dlp-level dedup guard so even a stale index + can't re-download. +- [ ] **Retention / quality-upgrade (stretch, from Pinchflat).** Optional re-download to upgrade + quality/metadata, and optional pruning of old items. Defer — niche for a desktop app. + +## Phase J — Subscriptions & scheduled auto-download ✅ COMPLETE (OS wiring needs smoke test) + +Pinchflat's headline feature: a Source you *watch*, that downloads new uploads on its own. +AeroFetch already has the Windows pieces for this elsewhere in this workspace (Task Scheduler + +single-instance Mutex + a headless run mode — see the DashMail/Weather Radar patterns). + +- [ ] **Watched sources.** A `Source.watched` flag + an "auto-download new" toggle per source. +- [ ] **Fast indexing via RSS.** YouTube exposes a per-channel Atom feed + (`https://www.youtube.com/feeds/videos.xml?channel_id=<id>`) — a cheap check for new + uploads without a full yt-dlp scan. **Caveat:** the feed only carries the latest ~15 + videos, so RSS is for *staying current*, not initial indexing (which stays a full + Phase-F scan). Pinchflat draws the same line ("fast indexing is not recommended for + playlists / initial scans"). +- [ ] **Scheduled headless run.** A `--sync` CLI entry that indexes all watched sources and + enqueues new items, wired to **Windows Task Scheduler** (reuse the + single-instance-lock already added in Phase E so a scheduled run hands off to a running + window instead of double-launching). Notify on new downloads via the existing + `Notification` path. + +## Phase K — Media-server output (optional stretch) ✅ COMPLETE + +Make AeroFetch's output drop-in for Jellyfin/Plex/Kodi the way Pinchflat does — relevant +because organized channel folders are exactly what those servers ingest. + +- [ ] **Sidecar metadata.** `--write-info-json`, `--write-thumbnail`, `--write-description`, + and optional Kodi/Jellyfin `.nfo` generation. Most of this is already expressible through + Phase C custom-command args; a checkbox group in the output settings makes it first-class. + +--- + +## What we reuse vs. build + +**Reuse unchanged:** the download queue + concurrency cap + cancel/retry (`store/downloads.ts`), +history + error log, cookies + proxy + rate-limit + aria2c (`AccessOptions`), all +post-processing `DownloadOptions`, `--download-archive`, native notifications, the +single-instance lock, and the JSON-store pattern. + +**Build new:** channel-depth probing (Phase F), the `sources.json` index + `Source`/`MediaItem` +model (F), per-item folder output paths + sanitizer (G), the Library view + `useSources` store +(H), incremental diff-sync (I), and RSS + scheduled sync (J). + +## Suggested order + +F → G are the foundation and unlock a usable "download whole channel into folders" the moment +they land (drive it from a temporary button before the full Library UI exists). H makes it a +real product surface. I and J turn it into a true media manager. K is polish. Phases F–H are +the high-value core; I–K are incremental and each shippable on its own. diff --git a/ROADMAP.md b/ROADMAP.md index 33c4f82..e2be86c 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -9,6 +9,11 @@ Sources: [Seal GitHub](https://github.com/JunkFood02/Seal) · [Seal on F-Droid](https://f-droid.org/packages/com.junkfood.seal/) · [It's FOSS overview](https://itsfoss.com/news/seal/). +> **See also:** [ROADMAP-PINCHFLAT.md](ROADMAP-PINCHFLAT.md) — a parallel track that extends +> AeroFetch from a one-shot downloader into a [Pinchflat](https://github.com/kieraneglin/pinchflat)-style +> media manager (index entire channels → organize into `Channel / Playlist / Title` folders → +> keep in sync), reusing the existing queue/history/options rather than replacing them. + --- ## Already at parity ✅ diff --git a/src/main/buildArgs.ts b/src/main/buildArgs.ts index 8ecc536..9e6e592 100644 --- a/src/main/buildArgs.ts +++ b/src/main/buildArgs.ts @@ -8,10 +8,12 @@ * resolving it itself; the caller in download.ts passes `getBinDir()`. */ +import { join } from 'path' import { BEST_FORMAT_ID, type StartDownloadOptions, type DownloadOptions, + type CollectionContext, type CookieBrowser } from '@shared/ipc' @@ -138,6 +140,52 @@ export function formatCommandLine(exe: string, args: string[]): string { return [exe, ...args].map(quoteForDisplay).join(' ') } +// --- Collection (media-manager) folder paths -------------------------------- + +/** + * Sanitize one path segment (a channel or playlist name) into a Windows-safe + * directory name. This matters because, unlike the filename yt-dlp itself writes + * (which `--restrict-filenames` can clean), these directory segments are built by + * AeroFetch from untrusted channel/playlist titles and joined onto the output + * dir — so they must be neutered for both illegal characters AND path traversal. + * + * - illegal chars (`< > : " / \ | ? *`) and control chars → space + * - leading/trailing dots and spaces stripped (illegal / invisible on Windows), + * which also turns a bare `..` traversal segment into nothing + * - reserved device names (CON, PRN, NUL, COM1…) get an underscore prefix + * - length-capped so the full path stays well under MAX_PATH + * - empty result falls back to 'Untitled' + */ +export function sanitizeDirSegment(name: string): string { + // C0 control chars (charCode < 0x20) and Windows-illegal chars both become a + // space. The control-char filter is done by char code so no literal control + // byte ever appears in this source file. + let s = Array.from(name ?? '', (ch) => (ch.charCodeAt(0) < 0x20 ? ' ' : ch)).join('') + s = s.replace(/[<>:"/\\|?*]/g, ' ') + s = s.replace(/\s+/g, ' ').trim().replace(/[. ]+$/, '').replace(/^[. ]+/, '') + if (/^(con|prn|aux|nul|com[1-9]|lpt[1-9])$/i.test(s)) s = `_${s}` + s = s.slice(0, 80).trim() + return s || 'Untitled' +} + +/** + * Build the `-o` output template for a collection download: + * <outDir>/<Channel>/<Playlist>/<NNN> - <baseFilename> + * where NNN is the 1-based playlist index zero-padded to three digits and + * baseFilename keeps its yt-dlp field tokens (e.g. '%(title)s.%(ext)s') so they + * still expand. Channel/playlist are sanitized via sanitizeDirSegment. + */ +export function collectionOutputTemplate( + outDir: string, + c: CollectionContext, + baseFilename: string +): string { + const n = Number.isFinite(c.index) && c.index > 0 ? Math.floor(c.index) : 1 + const nnn = String(n).padStart(3, '0') + const segments = [sanitizeDirSegment(c.channel), sanitizeDirSegment(c.playlist)] + return join(outDir, ...segments, `${nnn} - ${baseFilename}`) +} + function accessArgs(a: AccessOptions): string[] { const args: string[] = [] if (a.proxy.trim()) args.push('--proxy', a.proxy.trim()) @@ -179,6 +227,12 @@ function postProcessArgs(opts: StartDownloadOptions, o: DownloadOptions): string if (o.embedChapters) args.push('--embed-chapters') if (o.embedMetadata) args.push('--embed-metadata') + // Media-server sidecar files (Phase K) — written next to the output file so + // Jellyfin/Plex/Kodi can ingest metadata, poster art, and the description. + if (o.writeInfoJson) args.push('--write-info-json') + if (o.writeThumbnailFile) args.push('--write-thumbnail') + if (o.writeDescription) args.push('--write-description') + return args } diff --git a/src/main/download.ts b/src/main/download.ts index 037f6c1..0e24b01 100644 --- a/src/main/download.ts +++ b/src/main/download.ts @@ -7,7 +7,12 @@ import { getSettings, getDownloadArchivePath } from './settings' import { getCookiesFilePath } from './cookies' import { listTemplates } from './templates' import { assertHttpUrl } from './url' -import { buildArgs, parseExtraArgs, formatCommandLine } from './buildArgs' +import { + buildArgs, + parseExtraArgs, + formatCommandLine, + collectionOutputTemplate +} from './buildArgs' import { cleanError } from './log' import { addErrorLog } from './errorlog' import { @@ -161,7 +166,12 @@ function resolveExtraArgs(opts: StartDownloadOptions, settings: Settings): strin export function buildCommand(opts: StartDownloadOptions): string[] { const settings = getSettings() const outDir = opts.outputDir?.trim() || settings.outputDir || app.getPath('downloads') - const template = settings.filenameTemplate?.trim() || '%(title)s.%(ext)s' + // A collection (media-manager) download is filed into <channel>/<playlist>/ + // <NNN> - <title> folders; an ordinary download uses the flat filenameTemplate. + const filenameTemplate = settings.filenameTemplate?.trim() || '%(title)s.%(ext)s' + const outputTemplate = opts.collection + ? collectionOutputTemplate(outDir, opts.collection, '%(title)s.%(ext)s') + : join(outDir, filenameTemplate) // Per-download override wins; otherwise use the persisted defaults. const options = opts.options ?? settings.downloadOptions // Silently fall back to yt-dlp's own downloader if aria2c.exe wasn't dropped @@ -183,7 +193,7 @@ export function buildCommand(opts: StartDownloadOptions): string[] { downloadArchivePath: settings.downloadArchive ? getDownloadArchivePath() : undefined } const extraArgs = resolveExtraArgs(opts, settings) - return buildArgs(opts, join(outDir, template), options, getBinDir(), access, extraArgs) + return buildArgs(opts, outputTemplate, options, getBinDir(), access, extraArgs) } /** Build the exact command line for the current form state, without running it. */ diff --git a/src/main/index.ts b/src/main/index.ts index 68bddb2..42ca203 100644 --- a/src/main/index.ts +++ b/src/main/index.ts @@ -22,6 +22,17 @@ import { openCookieLoginWindow, getCookiesStatus, clearCookies } from './cookies import { listErrorLog, addErrorLog, clearErrorLog } from './errorlog' import { exportBackup, importBackup } from './backup' import { extractIncomingUrl, registerSendToShortcut, focusWindow } from './deeplink' +import { + listSources, + getSource, + removeSource, + listMediaItems, + setMediaItemDownloaded, + setSourceWatched +} from './sources' +import { indexSource } from './indexer' +import { syncWatchedSources } from './sync' +import { getScheduledSync, setScheduledSync, isSyncLaunch } from './schedule' // Only one instance ever runs. A second launch — e.g. the OS invoking us again // for an aerofetch:// link or a "Send to AeroFetch" file — hands its argv to @@ -96,7 +107,10 @@ function createWindow(): void { mainWindow = win win.on('ready-to-show', () => { - win.show() + // A scheduled `--sync` launch starts unobtrusively (shown but not focused) so + // the daily background sync doesn't steal focus; a normal launch shows + focuses. + if (isSyncLaunch(process.argv)) win.showInactive() + else win.show() }) win.on('closed', () => { @@ -229,6 +243,38 @@ function registerIpcHandlers(): void { } return result }) + + // --- Media-manager sources (Pinchflat-style index; see ROADMAP-PINCHFLAT.md) --- + ipcMain.handle(IpcChannels.sourcesList, () => listSources()) + ipcMain.handle(IpcChannels.sourceItems, (_e, sourceId: string) => listMediaItems(sourceId)) + ipcMain.handle(IpcChannels.sourceRemove, (_e, id: string) => removeSource(id)) + ipcMain.handle(IpcChannels.sourceItemDownloaded, (_e, id: string, filePath?: string) => + setMediaItemDownloaded(id, filePath) + ) + // Indexing pushes live progress to the requesting renderer over `indexProgress` + // and resolves with the final result. + ipcMain.handle(IpcChannels.sourceIndex, (e, url: string) => + indexSource(url, (p) => { + if (!e.sender.isDestroyed()) e.sender.send(IpcChannels.indexProgress, p) + }) + ) + ipcMain.handle(IpcChannels.sourceReindex, (e, id: string) => { + const src = getSource(id) + if (!src) return { ok: false, error: 'Source not found.' } + return indexSource(src.url, (p) => { + if (!e.sender.isDestroyed()) e.sender.send(IpcChannels.indexProgress, p) + }) + }) + ipcMain.handle(IpcChannels.sourceSetWatched, (_e, id: string, watched: boolean) => + setSourceWatched(id, watched) + ) + ipcMain.handle(IpcChannels.sourcesSync, (e) => + syncWatchedSources((p) => { + if (!e.sender.isDestroyed()) e.sender.send(IpcChannels.indexProgress, p) + }) + ) + ipcMain.handle(IpcChannels.scheduledSyncGet, () => getScheduledSync()) + ipcMain.handle(IpcChannels.scheduledSyncSet, (_e, enabled: boolean) => setScheduledSync(enabled)) } // Push OS theme/contrast changes to every window, and keep the native diff --git a/src/main/indexer.ts b/src/main/indexer.ts new file mode 100644 index 0000000..8ab7235 --- /dev/null +++ b/src/main/indexer.ts @@ -0,0 +1,204 @@ +/** + * Source indexing orchestration (Pinchflat-style media manager; see + * ROADMAP-PINCHFLAT.md). Walks a channel (its /playlists + /videos tabs) or a + * single playlist with `yt-dlp --flat-playlist`, merges the result into a deduped + * MediaItem list, and persists it as a Source. The download step (Phase G+) then + * pulls from that persisted list rather than the live queue holding it all. + * + * The pure URL-classification + merge logic lives in indexerCore.ts (unit-tested); + * this module is the impure shell that spawns yt-dlp and writes to disk. + */ + +import { execFile } from 'child_process' +import { existsSync } from 'fs' +import { getYtdlpPath } from './binaries' +import { cleanError } from './log' +import { assertHttpUrl } from './url' +import { + classifySource, + buildMediaItems, + buildFeedUrl, + entryUrl, + stripTabSuffix, + stableSourceId, + type RawEntry, + type NamedPlaylist +} from './indexerCore' +import { getSource, upsertSource, mergeMediaItems } from './sources' +import type { IndexProgress, IndexSourceResult, Source, SourceKind } from '@shared/ipc' + +/** The slice of `yt-dlp -J --flat-playlist` output the indexer reads. */ +interface FlatInfo { + _type?: string + id?: string + title?: string + uploader?: string + channel?: string + /** the UC… channel id, used to build the RSS feed URL */ + channel_id?: string + entries?: RawEntry[] +} + +/** + * Run `yt-dlp -J --flat-playlist` on a URL and parse the JSON. Rejects on a + * non-zero exit or unparseable output. maxBuffer is large because a big channel's + * flat upload list can be a few MB of JSON; the timeout is generous for the same + * reason. `--` terminates option parsing so the URL can't be read as a flag. + */ +function probeFlat(url: string): Promise<FlatInfo> { + return new Promise((resolve, reject) => { + execFile( + getYtdlpPath(), + ['-J', '--flat-playlist', '--no-warnings', '--', url], + { windowsHide: true, maxBuffer: 256 * 1024 * 1024, timeout: 180_000 }, + (err, stdout, stderr) => { + if (err) { + const msg = (err as { killed?: boolean }).killed + ? 'Timed out indexing source. Check the link or your connection.' + : cleanError(stderr) || err.message + reject(new Error(msg)) + return + } + try { + resolve(JSON.parse(stdout) as FlatInfo) + } catch { + reject(new Error('Could not parse source info from yt-dlp.')) + } + } + ) + }) +} + +/** Probe a channel tab (e.g. /videos, /playlists); never throws — returns null. */ +function probeTab(url: string): Promise<FlatInfo | null> { + return probeFlat(url).catch(() => null) +} + +/** + * Index (or re-index) a Source. Resolves to the persisted Source + item count, or + * an error. `onProgress` is invoked throughout so the renderer can show a live + * status line — it's best-effort and never affects the result. + */ +export async function indexSource( + url: string, + onProgress: (p: IndexProgress) => void +): Promise<IndexSourceResult> { + try { + assertHttpUrl(url) + } catch (e) { + return { ok: false, error: (e as Error).message } + } + if (!existsSync(getYtdlpPath())) { + return { ok: false, error: 'yt-dlp.exe not found. Drop it into resources/bin/.' } + } + + const cls = classifySource(url) + onProgress({ url, phase: 'start', message: 'Reading source…' }) + + try { + let kind: SourceKind + let title: string + let channel: string | undefined + let feedId: string | undefined + const playlists: NamedPlaylist[] = [] + let uploads: RawEntry[] = [] + + if (cls?.kind === 'channel') { + kind = 'channel' + + // 1. Enumerate the channel's playlists, then each playlist's videos. + onProgress({ url, phase: 'playlists', message: 'Finding playlists…' }) + const playlistTab = await probeTab(`${cls.base}/playlists`) + const playlistEntries = playlistTab?.entries ?? [] + const total = playlistEntries.length + let done = 0 + for (const pe of playlistEntries) { + done++ + const purl = entryUrl(pe) + if (!purl) continue + onProgress({ + url, + phase: 'playlist', + message: `Indexing playlist ${done}/${total}: ${pe.title ?? ''}`.trim(), + current: done, + total + }) + const pdata = await probeTab(purl) + if (pdata?.entries?.length) { + playlists.push({ title: pe.title || 'Playlist', entries: pdata.entries }) + } + } + + // 2. The full uploads feed — the catch-all for videos in no playlist. + onProgress({ url, phase: 'uploads', message: 'Indexing channel uploads…' }) + const videoTab = await probeTab(`${cls.base}/videos`) + uploads = videoTab?.entries ?? [] + + title = + stripTabSuffix(videoTab?.channel) || + stripTabSuffix(videoTab?.title) || + stripTabSuffix(playlistTab?.title) || + cls.base + channel = stripTabSuffix(videoTab?.channel || videoTab?.uploader) || title + feedId = videoTab?.channel_id || playlistTab?.channel_id + } else { + // A single playlist (classified) — or an unclassified URL that might still + // resolve to a playlist when probed. A lone video has no entries → error. + kind = cls?.kind ?? 'playlist' + onProgress({ url, phase: 'uploads', message: 'Indexing playlist…' }) + const data = await probeFlat(cls?.base ?? url) + const entries = data.entries ?? [] + if (entries.length === 0) { + return { + ok: false, + error: 'That link is a single video, not a channel or playlist. Use the download bar for one video.' + } + } + title = data.title || 'Playlist' + channel = data.uploader || data.channel + // A playlist feed keys off the playlist id (PL…), carried as `id` here. + feedId = data.id + // File everything under the playlist's own name (not the 'Uploads' fallback). + playlists.push({ title, entries }) + } + + const sourceId = stableSourceId(cls?.base ?? url.trim()) + const fresh = buildMediaItems(sourceId, playlists, uploads) + if (fresh.length === 0) { + return { ok: false, error: 'No downloadable videos found for this source.' } + } + + // Incremental merge: preserve the downloaded state of anything already on + // disk, and report how many videos are new since the last index. + const { items, newCount } = mergeMediaItems(sourceId, fresh) + + const prev = getSource(sourceId) + const source: Source = { + id: sourceId, + url: url.trim(), + kind, + title, + channel, + addedAt: prev?.addedAt ?? Date.now(), + lastIndexedAt: Date.now(), + itemCount: items.length, + // Preserve the watched flag across a re-index; refresh the RSS feed URL. + watched: prev?.watched, + feedUrl: buildFeedUrl(kind, feedId) ?? prev?.feedUrl + } + upsertSource(source) + + const newNote = newCount === items.length ? '' : ` (${newCount} new)` + onProgress({ + url, + phase: 'done', + message: `Indexed ${items.length} videos${newNote}.`, + total: items.length + }) + return { ok: true, source, itemCount: items.length, newCount } + } catch (e) { + const msg = e instanceof Error ? e.message : String(e) + onProgress({ url, phase: 'error', message: msg }) + return { ok: false, error: msg } + } +} diff --git a/src/main/indexerCore.ts b/src/main/indexerCore.ts new file mode 100644 index 0000000..6a20a41 --- /dev/null +++ b/src/main/indexerCore.ts @@ -0,0 +1,226 @@ +/** + * Pure, dependency-free core for Source indexing (Pinchflat-style media manager; + * see ROADMAP-PINCHFLAT.md). Like buildArgs.ts / validation.ts this module imports + * nothing from electron or the node runtime, so its URL-classification and + * item-merge logic can be unit-tested without spinning up Electron. + * + * The impure orchestration (spawning yt-dlp, persisting to disk) lives in + * indexer.ts, which composes these helpers. + */ + +import type { MediaItem, SourceKind } from '@shared/ipc' + +// --- yt-dlp --flat-playlist entry shape (shared with probe.ts) -------------- + +/** The slice of a flat-playlist entry we read (a video, or a nested playlist). */ +export interface RawEntry { + id?: string + title?: string + url?: string + webpage_url?: string + duration?: number + uploader?: string + channel?: string +} + +/** + * Resolve the canonical URL for a flat-playlist entry. Prefer an explicit http(s) + * URL; otherwise build a YouTube watch URL from the id (the common case where flat + * entries carry only an id). Returns null when nothing usable is present. + * + * Note: the returned value is always either an http(s) URL or null, so it can + * never begin with '-' and be mis-read as a yt-dlp option (callers also pass `--`). + */ +export function entryUrl(e: RawEntry): string | null { + const cand = e.url || e.webpage_url + if (cand && /^https?:\/\//i.test(cand)) return cand + if (e.id) return `https://www.youtube.com/watch?v=${e.id}` + return null +} + +/** Seconds → 'M:SS' or 'H:MM:SS'. Flat entries give duration as a number. */ +export function fmtDuration(sec?: number): string | undefined { + if (sec == null || !Number.isFinite(sec)) return undefined + const s = Math.max(0, Math.round(sec)) + const h = Math.floor(s / 3600) + const m = Math.floor((s % 3600) / 60) + const r = s % 60 + const mm = h ? String(m).padStart(2, '0') : String(m) + return `${h ? `${h}:` : ''}${mm}:${String(r).padStart(2, '0')}` +} + +// --- Source-URL classification ---------------------------------------------- + +/** A classified Source URL: the kind, plus a normalised base for tab probing. */ +export interface SourceClass { + kind: SourceKind + /** + * For 'channel': the channel root (e.g. https://www.youtube.com/@handle) onto + * which '/videos' and '/playlists' tabs are appended. For 'playlist': the + * canonical playlist URL. + */ + base: string +} + +/** + * Classify a pasted URL as a YouTube channel, a playlist, or neither. + * + * Channel forms: /@handle, /channel/<id>, /c/<name>, /user/<name> — any trailing + * tab (/videos, /playlists, /streams, /shorts, /featured) is stripped to the root. + * Playlist form: any URL carrying a `list=` query param. Returns null for a lone + * video or a non-YouTube URL (Phase F scopes the channel walk to YouTube, whose + * /videos + /playlists tab structure this relies on). + */ +export function classifySource(raw: string): SourceClass | null { + let u: URL + try { + u = new URL((raw ?? '').trim()) + } catch { + return null + } + if (u.protocol !== 'http:' && u.protocol !== 'https:') return null + + const host = u.hostname.replace(/^www\./i, '').toLowerCase() + const isYouTube = host === 'youtube.com' || host.endsWith('.youtube.com') || host === 'youtu.be' + + // A playlist is identified purely by its list= param (works on any youtube host). + const list = u.searchParams.get('list') + if (isYouTube && list) { + return { kind: 'playlist', base: `https://www.youtube.com/playlist?list=${encodeURIComponent(list)}` } + } + + if (!isYouTube) return null + + // Channel roots. The handle/id segment is captured; later path segments + // (the tab) are discarded so we always probe from the channel root. + const m = u.pathname.match(/^\/(@[^/]+|channel\/[^/]+|c\/[^/]+|user\/[^/]+)/i) + if (m) return { kind: 'channel', base: `https://www.youtube.com/${m[1]}` } + + return null +} + +/** + * Channel tab titles from yt-dlp often look like "<Name> - Videos" or + * "<Name> - Playlists". Strip a trailing known-tab suffix to recover the bare + * channel name. Leaves anything else untouched. + */ +export function stripTabSuffix(title: string | undefined): string | undefined { + if (!title) return title + return title.replace(/\s[-–—]\s(Videos|Playlists|Shorts|Live|Streams|Home|Featured)$/i, '').trim() +} + +/** + * Deterministic 32-bit FNV-1a hash → 8 hex chars. Used to derive a stable Source + * id from its normalised base URL so re-indexing the same channel updates the + * existing record instead of creating a duplicate. Pure (no crypto import). + */ +export function stableSourceId(base: string): string { + let h = 0x811c9dc5 + for (let i = 0; i < base.length; i++) { + h ^= base.charCodeAt(i) + h = Math.imul(h, 0x01000193) + } + return (h >>> 0).toString(16).padStart(8, '0') +} + +// --- RSS fast-check (Phase J: watched sources) ------------------------------ + +/** + * Build a YouTube RSS feed URL for cheap "anything new?" polling of a watched + * source. Channels feed by `channel_id` (UC…), playlists by `playlist_id` (PL…). + * Returns undefined when the id is missing (the sync then falls back to a full + * re-index). The feed only carries the latest ~15 uploads, so it's a freshness + * check, not a substitute for the initial full index. + */ +export function buildFeedUrl(kind: SourceKind, ytId: string | undefined): string | undefined { + if (!ytId) return undefined + const param = kind === 'channel' ? 'channel_id' : 'playlist_id' + return `https://www.youtube.com/feeds/videos.xml?${param}=${encodeURIComponent(ytId)}` +} + +/** Extract the video ids from a YouTube RSS/Atom feed body (the <yt:videoId> tags). */ +export function parseRssVideoIds(xml: string): string[] { + const ids: string[] = [] + const re = /<yt:videoId>\s*([\w-]+)\s*<\/yt:videoId>/g + let m: RegExpExecArray | null + while ((m = re.exec(xml)) !== null) ids.push(m[1]) + return ids +} + +// --- Entry merge / dedup ---------------------------------------------------- + +/** A named playlist and its (flat) video entries, ready to merge. */ +export interface NamedPlaylist { + title: string + entries: RawEntry[] +} + +/** + * Merge a Source's playlists (and its catch-all uploads) into a deduped list of + * MediaItem records. Dedup is by video id: the FIRST playlist a video appears in + * wins its folder assignment, so videos grouped into a real playlist land there, + * and only videos in no playlist fall through to the synthetic 'Uploads' folder. + * + * `playlistIndex` is the 1-based position WITHIN the winning playlist (not the + * global upload order), so the on-disk "NNN - Title" numbering matches the + * playlist the file is filed under. + */ +export function buildMediaItems( + sourceId: string, + playlists: NamedPlaylist[], + uploads: RawEntry[] +): MediaItem[] { + const byVideo = new Map<string, MediaItem>() + + const add = (e: RawEntry, playlistTitle: string, index: number): void => { + const videoId = e.id + if (!videoId) return + if (byVideo.has(videoId)) return // first playlist wins + const url = entryUrl(e) + if (!url) return // not turnable into a downloadable URL + byVideo.set(videoId, { + id: `${sourceId}:${videoId}`, + sourceId, + videoId, + title: e.title || `Video ${videoId}`, + url, + playlistTitle, + playlistIndex: index, + durationLabel: fmtDuration(e.duration), + downloaded: false + }) + } + + for (const p of playlists) p.entries.forEach((e, i) => add(e, p.title, i + 1)) + uploads.forEach((e, i) => add(e, 'Uploads', i + 1)) + + return [...byVideo.values()] +} + +/** + * Merge a freshly-enumerated item list with the previously-persisted one for the + * same source (incremental re-index; see ROADMAP-PINCHFLAT.md Phase I). The fresh + * list defines the current membership/ordering, but any video already marked + * downloaded keeps its `downloaded`/`downloadedAt`/`filePath` so a re-sync never + * loses that state or re-downloads it. Videos that vanished from the source (now + * private/deleted) are dropped. `newCount` is how many fresh videos weren't in + * the previous set — the "X new since last sync" figure. + */ +export function mergeItemsPreservingState( + existing: MediaItem[], + fresh: MediaItem[] +): { items: MediaItem[]; newCount: number } { + const prevByVideo = new Map(existing.map((m) => [m.videoId, m])) + let newCount = 0 + const items = fresh.map((f) => { + const prev = prevByVideo.get(f.videoId) + if (!prev) { + newCount++ + return f + } + return prev.downloaded + ? { ...f, downloaded: true, downloadedAt: prev.downloadedAt, filePath: prev.filePath } + : f + }) + return { items, newCount } +} diff --git a/src/main/probe.ts b/src/main/probe.ts index 0e675ef..4559fef 100644 --- a/src/main/probe.ts +++ b/src/main/probe.ts @@ -4,6 +4,7 @@ import { getYtdlpPath } from './binaries' import { fmtBytes } from './download' import { cleanError } from './log' import { assertHttpUrl } from './url' +import { entryUrl, fmtDuration, type RawEntry } from './indexerCore' import { BEST_FORMAT_ID, type ProbeResult, @@ -26,16 +27,6 @@ interface RawFormat { filesize_approx?: number } -interface RawEntry { - id?: string - title?: string - url?: string - webpage_url?: string - duration?: number - uploader?: string - channel?: string -} - interface RawInfo { _type?: string title?: string @@ -92,29 +83,6 @@ function buildInfo(data: RawInfo): MediaInfo { } } -/** Seconds → 'M:SS' or 'H:MM:SS'. Flat playlist entries give duration as a number. */ -function fmtDuration(sec?: number): string | undefined { - if (sec == null || !Number.isFinite(sec)) return undefined - const s = Math.max(0, Math.round(sec)) - const h = Math.floor(s / 3600) - const m = Math.floor((s % 3600) / 60) - const r = s % 60 - const mm = h ? String(m).padStart(2, '0') : String(m) - return `${h ? `${h}:` : ''}${mm}:${String(r).padStart(2, '0')}` -} - -/** - * Resolve the URL we'll enqueue for a playlist entry. Prefer an explicit http(s) - * URL; fall back to a YouTube watch URL built from the id (the common case where - * flat entries carry only an id). Returns null when nothing usable is present. - */ -function entryUrl(e: RawEntry): string | null { - const cand = e.url || e.webpage_url - if (cand && /^https?:\/\//i.test(cand)) return cand - if (e.id) return `https://www.youtube.com/watch?v=${e.id}` - return null -} - function buildPlaylist(data: RawInfo): PlaylistInfo { const entries: PlaylistEntry[] = [] ;(data.entries ?? []).forEach((e, i) => { diff --git a/src/main/schedule.ts b/src/main/schedule.ts new file mode 100644 index 0000000..eb7f346 --- /dev/null +++ b/src/main/schedule.ts @@ -0,0 +1,67 @@ +/** + * Windows Task Scheduler integration for the daily watched-source sync + * (ROADMAP-PINCHFLAT.md Phase J). Registers a task that launches AeroFetch with + * `--sync` once a day; the app then runs its startup sync of watched sources. + * + * NOTE: this is OS-level wiring and cannot be exercised in the Vite UI preview or + * the unit tests — like the `aerofetch://` protocol registration, it needs a real + * install + manual smoke test. `schtasks` is invoked via execFile (no shell), and + * the only interpolated value is the trusted `process.execPath`. + */ + +import { execFile } from 'child_process' +import type { ScheduledSyncStatus } from '@shared/ipc' + +const TASK_NAME = 'AeroFetchDailySync' +/** The argv flag the scheduled task passes so startup knows it's a sync launch. */ +export const SYNC_FLAG = '--sync' + +function schtasks(args: string[]): Promise<{ code: number; stdout: string; stderr: string }> { + return new Promise((resolve) => { + execFile('schtasks', args, { windowsHide: true }, (err, stdout, stderr) => { + const code = err ? ((err as { code?: number }).code ?? 1) : 0 + resolve({ code, stdout: String(stdout), stderr: String(stderr) }) + }) + }) +} + +/** True when this launch came from the scheduled task (argv carries --sync). */ +export function isSyncLaunch(argv: string[]): boolean { + return argv.includes(SYNC_FLAG) +} + +/** Whether the daily-sync scheduled task is currently registered. */ +export async function getScheduledSync(): Promise<ScheduledSyncStatus> { + const r = await schtasks(['/Query', '/TN', TASK_NAME]) + return { enabled: r.code === 0 } +} + +/** + * Register or remove the daily-sync scheduled task. Creating runs AeroFetch with + * `--sync` every day at 09:00 (overwriting any prior task of the same name). + */ +export async function setScheduledSync(enabled: boolean): Promise<ScheduledSyncStatus> { + if (enabled) { + const tr = `"${process.execPath}" ${SYNC_FLAG}` + const r = await schtasks([ + '/Create', + '/F', + '/SC', + 'DAILY', + '/ST', + '09:00', + '/TN', + TASK_NAME, + '/TR', + tr + ]) + return { + enabled: r.code === 0, + error: r.code === 0 ? undefined : r.stderr.trim() || 'Could not create the scheduled task.' + } + } + const r = await schtasks(['/Delete', '/F', '/TN', TASK_NAME]) + // A missing task ("cannot find") is success for our purposes — it's already gone. + const gone = r.code === 0 || /cannot find|does not exist/i.test(r.stderr) + return { enabled: gone ? false : true, error: gone ? undefined : r.stderr.trim() } +} diff --git a/src/main/settings.ts b/src/main/settings.ts index 9f33c88..8b4add0 100644 --- a/src/main/settings.ts +++ b/src/main/settings.ts @@ -36,6 +36,7 @@ const DEFAULTS: Settings = { customCommandEnabled: false, defaultTemplateId: null, notifyOnComplete: true, + autoDownloadNew: true, hasCompletedOnboarding: false } @@ -77,7 +78,10 @@ function sanitizeOptions(input: unknown): DownloadOptions { embedChapters: bool(o.embedChapters, d.embedChapters), embedMetadata: bool(o.embedMetadata, d.embedMetadata), embedThumbnail: bool(o.embedThumbnail, d.embedThumbnail), - cropThumbnail: bool(o.cropThumbnail, d.cropThumbnail) + cropThumbnail: bool(o.cropThumbnail, d.cropThumbnail), + writeInfoJson: bool(o.writeInfoJson, d.writeInfoJson), + writeThumbnailFile: bool(o.writeThumbnailFile, d.writeThumbnailFile), + writeDescription: bool(o.writeDescription, d.writeDescription) } } @@ -163,6 +167,7 @@ export function setSettings(partial: Partial<Settings>): Settings { case 'downloadArchive': case 'customCommandEnabled': case 'notifyOnComplete': + case 'autoDownloadNew': case 'hasCompletedOnboarding': if (typeof value === 'boolean') s.set(key, value) break diff --git a/src/main/sources.ts b/src/main/sources.ts new file mode 100644 index 0000000..bf0ef1a --- /dev/null +++ b/src/main/sources.ts @@ -0,0 +1,135 @@ +/** + * Persistence for the media-manager index (Pinchflat-style; see + * ROADMAP-PINCHFLAT.md). Two plain-JSON stores in userData, mirroring the + * history.ts pattern (and the same deliberate choice to stay on JSON rather than + * better-sqlite3 — revisit if a user indexes many large channels, see the + * Phase H risk note in the roadmap): + * + * sources.json — one Source record per added channel/playlist + * media-items.json — every MediaItem across all sources (queried by sourceId) + * + * Per-row validation on read (validation.ts) so a hand-edited or corrupted file + * can't feed the UI malformed records (same approach as history/errorlog). + */ + +import { app } from 'electron' +import { join } from 'path' +import { readFileSync, writeFileSync, existsSync } from 'fs' +import type { Source, MediaItem } from '@shared/ipc' +import { isValidSource, isValidMediaItem } from './validation' +import { mergeItemsPreservingState } from './indexerCore' + +// A generous global cap so a runaway index can't grow the file unbounded; large +// enough for several big channels. When exceeded, the most-recently-written +// source's items are kept (they're placed first by replaceMediaItems). +const MAX_ITEMS = 20000 + +function sourcesFile(): string { + return join(app.getPath('userData'), 'sources.json') +} + +function itemsFile(): string { + return join(app.getPath('userData'), 'media-items.json') +} + +function readJsonArray<T>(path: string, isValid: (o: unknown) => o is T): T[] { + try { + if (!existsSync(path)) return [] + const data = JSON.parse(readFileSync(path, 'utf8')) + return Array.isArray(data) ? data.filter(isValid) : [] + } catch { + return [] + } +} + +function writeJson(path: string, value: unknown): void { + try { + writeFileSync(path, JSON.stringify(value, null, 2)) + } catch { + /* best-effort; a read-only data dir just means no persisted index */ + } +} + +// --- Sources ---------------------------------------------------------------- + +export function listSources(): Source[] { + return readJsonArray(sourcesFile(), isValidSource) +} + +export function getSource(id: string): Source | undefined { + return listSources().find((s) => s.id === id) +} + +/** Insert or replace a source by id (a re-index updates the existing record). */ +export function upsertSource(source: Source): Source[] { + const sources = [source, ...listSources().filter((s) => s.id !== source.id)] + writeJson(sourcesFile(), sources) + return sources +} + +/** Toggle whether a source is watched for new uploads (Phase J). Returns all sources. */ +export function setSourceWatched(id: string, watched: boolean): Source[] { + const sources = listSources().map((s) => (s.id === id ? { ...s, watched } : s)) + writeJson(sourcesFile(), sources) + return sources +} + +/** Remove a source and all of its media items. Returns the remaining sources. */ +export function removeSource(id: string): Source[] { + const sources = listSources().filter((s) => s.id !== id) + writeJson(sourcesFile(), sources) + const items = listAllItems().filter((m) => m.sourceId !== id) + writeJson(itemsFile(), items) + return sources +} + +// --- Media items ------------------------------------------------------------ + +function listAllItems(): MediaItem[] { + return readJsonArray(itemsFile(), isValidMediaItem) +} + +export function listMediaItems(sourceId: string): MediaItem[] { + return listAllItems().filter((m) => m.sourceId === sourceId) +} + +/** + * Replace all media items for one source with a fresh set (the result of a + * (re)index). Other sources' items are preserved; the new items are placed first + * so they survive the MAX_ITEMS cap. + */ +export function replaceMediaItems(sourceId: string, items: MediaItem[]): void { + const others = listAllItems().filter((m) => m.sourceId !== sourceId) + writeJson(itemsFile(), [...items, ...others].slice(0, MAX_ITEMS)) +} + +/** + * Incrementally merge a freshly-indexed item list into the persisted store, + * preserving the downloaded state of videos already on disk (see Phase I). The + * fresh list defines current membership/order; returns the merged items and how + * many were new since the last index. + */ +export function mergeMediaItems( + sourceId: string, + fresh: MediaItem[] +): { items: MediaItem[]; newCount: number } { + const result = mergeItemsPreservingState(listMediaItems(sourceId), fresh) + replaceMediaItems(sourceId, result.items) + return result +} + +/** + * Mark one media item downloaded, recording its file path + time. Returns the + * updated list for that item's source (or [] if the id is unknown). Used by the + * library view once a queued item completes (Phase H/I). + */ +export function setMediaItemDownloaded(id: string, filePath?: string): MediaItem[] { + const all = listAllItems() + const target = all.find((m) => m.id === id) + if (!target) return [] + const updated = all.map((m) => + m.id === id ? { ...m, downloaded: true, downloadedAt: Date.now(), filePath } : m + ) + writeJson(itemsFile(), updated) + return updated.filter((m) => m.sourceId === target.sourceId) +} diff --git a/src/main/sync.ts b/src/main/sync.ts new file mode 100644 index 0000000..94fdea3 --- /dev/null +++ b/src/main/sync.ts @@ -0,0 +1,50 @@ +/** + * Watched-source sync (Pinchflat-style; ROADMAP-PINCHFLAT.md Phase J). Re-indexes + * every watched Source and reports the videos that are new since its last index. + * A YouTube RSS feed is used as a cheap pre-check so a source with no new uploads + * is skipped before the (more expensive) full `yt-dlp` re-index. + */ + +import { listSources, listMediaItems } from './sources' +import { indexSource } from './indexer' +import { parseRssVideoIds } from './indexerCore' +import type { IndexProgress, MediaItem, SyncResult } from '@shared/ipc' + +/** Fetch a YouTube RSS feed and return its recent video ids. Throws on failure. */ +async function fetchFeedIds(feedUrl: string): Promise<string[]> { + const res = await fetch(feedUrl, { signal: AbortSignal.timeout(15_000) }) + if (!res.ok) throw new Error(`feed responded ${res.status}`) + return parseRssVideoIds(await res.text()) +} + +/** + * Re-index every watched source and collect the videos new since the last index. + * RSS pre-check: if the feed shows only ids the source already knows, the full + * re-index is skipped. `onProgress` is forwarded from the underlying indexSource. + */ +export async function syncWatchedSources( + onProgress: (p: IndexProgress) => void +): Promise<SyncResult> { + try { + const watched = listSources().filter((s) => s.watched) + const newItems: MediaItem[] = [] + for (const src of watched) { + // Cheap freshness check — skip the full re-index when nothing is new. + if (src.feedUrl) { + const known = new Set(listMediaItems(src.id).map((m) => m.videoId)) + const recent = await fetchFeedIds(src.feedUrl).catch(() => null) + if (recent && recent.length > 0 && recent.every((id) => known.has(id))) continue + } + const before = new Set(listMediaItems(src.id).map((m) => m.videoId)) + const res = await indexSource(src.url, onProgress) + if (res.ok) { + for (const it of listMediaItems(src.id)) { + if (!before.has(it.videoId)) newItems.push(it) + } + } + } + return { ok: true, newItems } + } catch (e) { + return { ok: false, newItems: [], error: e instanceof Error ? e.message : String(e) } + } +} diff --git a/src/main/validation.ts b/src/main/validation.ts index e9dac21..1303e62 100644 --- a/src/main/validation.ts +++ b/src/main/validation.ts @@ -7,7 +7,7 @@ */ import { isAbsolute } from 'path' -import type { HistoryEntry, ErrorLogEntry, CommandTemplate } from '@shared/ipc' +import type { HistoryEntry, ErrorLogEntry, CommandTemplate, Source, MediaItem } from '@shared/ipc' // --- Path-traversal sanitization (audit S4) --------------------------------- @@ -75,3 +75,38 @@ export function isTemplateLike(o: unknown): o is CommandTemplate { const t = o as Record<string, unknown> return typeof t.id === 'string' || typeof t.id === 'number' } + +/** A persisted sources.json row must have the right shape or it's dropped on read. */ +export function isValidSource(o: unknown): o is Source { + if (!o || typeof o !== 'object') return false + const s = o as Record<string, unknown> + return ( + typeof s.id === 'string' && + typeof s.url === 'string' && + (s.kind === 'channel' || s.kind === 'playlist') && + typeof s.title === 'string' && + typeof s.addedAt === 'number' && + typeof s.itemCount === 'number' && + isOptionalString(s.channel) && + (s.lastIndexedAt === undefined || typeof s.lastIndexedAt === 'number') + ) +} + +/** A persisted media-items.json row must have the right shape or it's dropped on read. */ +export function isValidMediaItem(o: unknown): o is MediaItem { + if (!o || typeof o !== 'object') return false + const m = o as Record<string, unknown> + return ( + typeof m.id === 'string' && + typeof m.sourceId === 'string' && + typeof m.videoId === 'string' && + typeof m.title === 'string' && + typeof m.url === 'string' && + typeof m.playlistTitle === 'string' && + typeof m.playlistIndex === 'number' && + typeof m.downloaded === 'boolean' && + isOptionalString(m.durationLabel) && + isOptionalString(m.filePath) && + (m.downloadedAt === undefined || typeof m.downloadedAt === 'number') + ) +} diff --git a/src/preload/index.ts b/src/preload/index.ts index 158f33a..9628b65 100644 --- a/src/preload/index.ts +++ b/src/preload/index.ts @@ -17,7 +17,13 @@ import { type ErrorLogEntry, type BackupExportResult, type BackupImportResult, - type SystemThemeInfo + type SystemThemeInfo, + type Source, + type MediaItem, + type IndexProgress, + type IndexSourceResult, + type SyncResult, + type ScheduledSyncStatus } from '@shared/ipc' // The surface exposed to the renderer. Keep this thin: it only forwards to IPC. @@ -121,6 +127,48 @@ const api = { const listener = (_e: IpcRendererEvent, url: string): void => cb(url) ipcRenderer.on(IpcChannels.externalUrl, listener) return () => ipcRenderer.removeListener(IpcChannels.externalUrl, listener) + }, + + // --- Media-manager sources (Pinchflat-style; see ROADMAP-PINCHFLAT.md) --- + + listSources: (): Promise<Source[]> => ipcRenderer.invoke(IpcChannels.sourcesList), + + /** Index (or re-index) a channel/playlist URL into the persisted source list. */ + indexSource: (url: string): Promise<IndexSourceResult> => + ipcRenderer.invoke(IpcChannels.sourceIndex, url), + + /** Re-index an existing source by id (refreshes its media-item list). */ + reindexSource: (id: string): Promise<IndexSourceResult> => + ipcRenderer.invoke(IpcChannels.sourceReindex, id), + + removeSource: (id: string): Promise<Source[]> => + ipcRenderer.invoke(IpcChannels.sourceRemove, id), + + listSourceItems: (sourceId: string): Promise<MediaItem[]> => + ipcRenderer.invoke(IpcChannels.sourceItems, sourceId), + + /** Persist that a media item has finished downloading (drives incremental sync). */ + markSourceItemDownloaded: (id: string, filePath?: string): Promise<MediaItem[]> => + ipcRenderer.invoke(IpcChannels.sourceItemDownloaded, id, filePath), + + /** Toggle whether a source is watched for new uploads. */ + setSourceWatched: (id: string, watched: boolean): Promise<Source[]> => + ipcRenderer.invoke(IpcChannels.sourceSetWatched, id, watched), + + /** Re-index all watched sources; resolves with the videos found new. */ + syncSources: (): Promise<SyncResult> => ipcRenderer.invoke(IpcChannels.sourcesSync), + + /** Read / write the Windows daily-sync scheduled task. */ + getScheduledSync: (): Promise<ScheduledSyncStatus> => + ipcRenderer.invoke(IpcChannels.scheduledSyncGet), + setScheduledSync: (enabled: boolean): Promise<ScheduledSyncStatus> => + ipcRenderer.invoke(IpcChannels.scheduledSyncSet, enabled), + + /** Subscribe to live indexing progress. Returns an unsubscribe function. */ + onIndexProgress: (cb: (p: IndexProgress) => void): (() => void) => { + const listener = (_e: IpcRendererEvent, p: IndexProgress): void => cb(p) + ipcRenderer.on(IpcChannels.indexProgress, listener) + return () => ipcRenderer.removeListener(IpcChannels.indexProgress, listener) } } diff --git a/src/renderer/src/App.tsx b/src/renderer/src/App.tsx index 1f56ce3..1645e4b 100644 --- a/src/renderer/src/App.tsx +++ b/src/renderer/src/App.tsx @@ -2,6 +2,7 @@ import { useState } from 'react' import { FluentProvider, makeStyles, tokens } from '@fluentui/react-components' import { Sidebar, type TabValue } from './components/Sidebar' import { DownloadsView } from './components/DownloadsView' +import { LibraryView } from './components/LibraryView' import { HistoryView } from './components/HistoryView' import { SettingsView } from './components/SettingsView' import { Onboarding } from './components/Onboarding' @@ -66,6 +67,7 @@ function App(): React.JSX.Element { <main className={styles.content}> {tab === 'downloads' && <DownloadsView />} + {tab === 'library' && <LibraryView />} {tab === 'history' && <HistoryView />} {tab === 'settings' && <SettingsView />} </main> diff --git a/src/renderer/src/components/DownloadOptionsForm.tsx b/src/renderer/src/components/DownloadOptionsForm.tsx index f599925..d5c4aae 100644 --- a/src/renderer/src/components/DownloadOptionsForm.tsx +++ b/src/renderer/src/components/DownloadOptionsForm.tsx @@ -230,6 +230,29 @@ export function DownloadOptionsForm({ value, onChange }: Props): React.JSX.Eleme </Caption1> </div> )} + + <Field + label="Sidecar files" + hint="Write separate metadata/poster/description files next to each download — handy for Jellyfin, Plex or Kodi libraries." + > + <div className={styles.subGroup}> + <Checkbox + checked={value.writeInfoJson} + onChange={(_, d) => setOpt('writeInfoJson', !!d.checked)} + label="Metadata (.info.json)" + /> + <Checkbox + checked={value.writeThumbnailFile} + onChange={(_, d) => setOpt('writeThumbnailFile', !!d.checked)} + label="Thumbnail image file" + /> + <Checkbox + checked={value.writeDescription} + onChange={(_, d) => setOpt('writeDescription', !!d.checked)} + label="Description (.description)" + /> + </div> + </Field> </div> ) } diff --git a/src/renderer/src/components/LibraryView.tsx b/src/renderer/src/components/LibraryView.tsx new file mode 100644 index 0000000..0e7a2ce --- /dev/null +++ b/src/renderer/src/components/LibraryView.tsx @@ -0,0 +1,568 @@ +import { useEffect, useMemo, useState } from 'react' +import { + Subtitle2, + Body1, + Caption1, + Text, + Input, + Button, + Checkbox, + Switch, + Spinner, + makeStyles, + mergeClasses, + tokens, + shorthands +} from '@fluentui/react-components' +import { + SearchRegular, + ArrowSyncRegular, + ArrowClockwiseRegular, + DeleteRegular, + ArrowDownloadRegular, + ChevronDownRegular, + ChevronRightRegular, + AppsListRegular, + VideoClipMultipleRegular, + AlertRegular, + LibraryRegular +} from '@fluentui/react-icons' +import type { MediaItem, Source } from '@shared/ipc' +import { useSources, MAX_ENQUEUE_BATCH } from '../store/sources' +import { useSettings } from '../store/settings' +import { useDownloads, type DownloadStatus } from '../store/downloads' + +// True in the standalone browser preview (no Electron preload). +const PREVIEW = typeof window === 'undefined' || !window.electron + +/** Per-item status shown in the library: a live queue status, or pending/downloaded. */ +type ItemStatus = DownloadStatus | 'pending' + +const STATUS_LABEL: Record<ItemStatus, string> = { + pending: 'Pending', + queued: 'Queued', + downloading: 'Downloading', + completed: 'Downloaded', + error: 'Failed', + canceled: 'Canceled' +} + +const useStyles = makeStyles({ + root: { display: 'flex', flexDirection: 'column', gap: '18px' }, + header: { display: 'flex', flexDirection: 'column', gap: '2px' }, + sub: { color: tokens.colorNeutralForeground3 }, + addRow: { display: 'flex', gap: '8px' }, + addInput: { flexGrow: 1 }, + toolbar: { + display: 'flex', + alignItems: 'center', + gap: '14px', + flexWrap: 'wrap', + paddingTop: '2px' + }, + toolbarSpacer: { flexGrow: 1 }, + switchRow: { + display: 'flex', + alignItems: 'center', + gap: '2px', + color: tokens.colorNeutralForeground2 + }, + progress: { + display: 'flex', + alignItems: 'center', + gap: '8px', + color: tokens.colorNeutralForeground2, + fontSize: tokens.fontSizeBase200 + }, + error: { color: tokens.colorPaletteRedForeground1 }, + empty: { + display: 'flex', + flexDirection: 'column', + alignItems: 'center', + gap: '8px', + padding: '56px 16px', + color: tokens.colorNeutralForeground3, + textAlign: 'center' + }, + list: { display: 'flex', flexDirection: 'column', gap: '10px' }, + card: { + border: `1px solid ${tokens.colorNeutralStroke2}`, + ...shorthands.borderRadius(tokens.borderRadiusLarge), + backgroundColor: tokens.colorNeutralBackground1, + overflow: 'hidden' + }, + cardHead: { + display: 'flex', + alignItems: 'center', + gap: '12px', + padding: '12px 14px', + cursor: 'pointer', + ':hover': { backgroundColor: tokens.colorNeutralBackground1Hover } + }, + srcIcon: { + width: '34px', + height: '34px', + flexShrink: 0, + borderRadius: tokens.borderRadiusMedium, + backgroundColor: tokens.colorBrandBackground2, + color: tokens.colorBrandForeground2, + display: 'flex', + alignItems: 'center', + justifyContent: 'center', + fontSize: '18px' + }, + srcMeta: { display: 'flex', flexDirection: 'column', minWidth: 0, flexGrow: 1 }, + srcTitleRow: { display: 'flex', alignItems: 'center', gap: '8px', minWidth: 0 }, + srcTitle: { fontWeight: tokens.fontWeightSemibold, color: tokens.colorNeutralForeground1 }, + watchBadge: { + flexShrink: 0, + fontSize: tokens.fontSizeBase100, + padding: '0 7px', + ...shorthands.borderRadius(tokens.borderRadiusCircular), + backgroundColor: tokens.colorBrandBackground2, + color: tokens.colorBrandForeground2 + }, + srcSub: { color: tokens.colorNeutralForeground3 }, + detail: { + borderTop: `1px solid ${tokens.colorNeutralStroke2}`, + padding: '12px 14px', + display: 'flex', + flexDirection: 'column', + gap: '12px' + }, + actionRow: { display: 'flex', alignItems: 'center', gap: '8px', flexWrap: 'wrap' }, + actionSpacer: { flexGrow: 1 }, + group: { display: 'flex', flexDirection: 'column' }, + groupHead: { + display: 'flex', + alignItems: 'center', + gap: '8px', + padding: '6px 4px', + color: tokens.colorNeutralForeground2 + }, + groupTitle: { fontWeight: tokens.fontWeightSemibold, flexGrow: 1, minWidth: 0 }, + rows: { display: 'flex', flexDirection: 'column' }, + row: { + display: 'flex', + alignItems: 'center', + gap: '10px', + padding: '5px 4px 5px 18px' + }, + rowMain: { display: 'flex', flexDirection: 'column', minWidth: 0, flexGrow: 1 }, + rowTitle: { + whiteSpace: 'nowrap', + overflow: 'hidden', + textOverflow: 'ellipsis', + color: tokens.colorNeutralForeground1 + }, + rowMeta: { color: tokens.colorNeutralForeground3 }, + pill: { + flexShrink: 0, + fontSize: tokens.fontSizeBase200, + padding: '1px 8px', + ...shorthands.borderRadius(tokens.borderRadiusCircular), + backgroundColor: tokens.colorNeutralBackground3, + color: tokens.colorNeutralForeground3 + }, + pillDownloading: { + backgroundColor: tokens.colorBrandBackground2, + color: tokens.colorBrandForeground2 + }, + pillCompleted: { + backgroundColor: tokens.colorPaletteGreenBackground2, + color: tokens.colorPaletteGreenForeground2 + }, + pillError: { + backgroundColor: tokens.colorPaletteRedBackground2, + color: tokens.colorPaletteRedForeground2 + } +}) + +/** Group items by playlist, sorted by index within a group; 'Uploads' sinks last. */ +function groupByPlaylist(items: MediaItem[]): { title: string; items: MediaItem[] }[] { + const map = new Map<string, MediaItem[]>() + for (const it of items) { + const arr = map.get(it.playlistTitle) ?? [] + arr.push(it) + map.set(it.playlistTitle, arr) + } + const groups = [...map.entries()].map(([title, its]) => ({ + title, + items: [...its].sort((a, b) => a.playlistIndex - b.playlistIndex) + })) + groups.sort( + (a, b) => + (a.title === 'Uploads' ? 1 : 0) - (b.title === 'Uploads' ? 1 : 0) || + a.title.localeCompare(b.title) + ) + return groups +} + +function relTime(ms?: number): string { + if (!ms) return 'never' + const mins = Math.round((Date.now() - ms) / 60000) + if (mins < 1) return 'just now' + if (mins < 60) return `${mins} min ago` + const hrs = Math.round(mins / 60) + if (hrs < 24) return `${hrs} h ago` + return `${Math.round(hrs / 24)} d ago` +} + +export function LibraryView(): React.JSX.Element { + const styles = useStyles() + const sources = useSources((s) => s.sources) + const itemsBySource = useSources((s) => s.itemsBySource) + const selectedSourceId = useSources((s) => s.selectedSourceId) + const indexing = useSources((s) => s.indexing) + const selectSource = useSources((s) => s.selectSource) + const indexSource = useSources((s) => s.indexSource) + const reindexSource = useSources((s) => s.reindexSource) + const removeSource = useSources((s) => s.removeSource) + const enqueueItems = useSources((s) => s.enqueueItems) + const setWatched = useSources((s) => s.setWatched) + const syncWatched = useSources((s) => s.syncWatched) + const syncing = useSources((s) => s.syncing) + const downloadItems = useDownloads((s) => s.items) + const autoDownloadNew = useSettings((s) => s.autoDownloadNew) + const updateSettings = useSettings((s) => s.update) + + const [url, setUrl] = useState('') + const [error, setError] = useState<string | null>(null) + const [selected, setSelected] = useState<Set<string>>(new Set()) + const [syncNote, setSyncNote] = useState<string | null>(null) + const [scheduled, setScheduled] = useState(false) + + const watchedCount = sources.filter((s) => s.watched).length + + // Load the current scheduled-sync (Task Scheduler) state once. + useEffect(() => { + if (PREVIEW) return + window.api.getScheduledSync().then((s) => setScheduled(s.enabled)).catch(() => {}) + }, []) + + async function onCheckNew(): Promise<void> { + setSyncNote(null) + const n = await syncWatched() + setSyncNote(n > 0 ? `Found ${n} new video${n === 1 ? '' : 's'}.` : 'No new videos.') + } + + async function toggleScheduled(next: boolean): Promise<void> { + setScheduled(next) // optimistic + if (PREVIEW) return + const res = await window.api.setScheduledSync(next) + setScheduled(res.enabled) + if (res.error) setError(res.error) + } + + // Reset the selection whenever the expanded source changes. + useEffect(() => setSelected(new Set()), [selectedSourceId]) + + // Live per-URL queue status so a video row reflects its real download state. + const statusByUrl = useMemo(() => { + const m = new Map<string, DownloadStatus>() + for (const d of downloadItems) m.set(d.url, d.status) + return m + }, [downloadItems]) + + const items = selectedSourceId ? (itemsBySource[selectedSourceId] ?? []) : [] + const groups = useMemo(() => groupByPlaylist(items), [items]) + + const effStatus = (it: MediaItem): ItemStatus => + statusByUrl.get(it.url) ?? (it.downloaded ? 'completed' : 'pending') + + const pendingItems = items.filter((it) => effStatus(it) === 'pending') + + async function onIndex(): Promise<void> { + setError(null) + const u = url.trim() + if (!u || indexing.active) return + const res = await indexSource(u) + if (res.ok) setUrl('') + else setError(res.error ?? 'Could not index that link.') + } + + function toggle(id: string, on: boolean): void { + setSelected((prev) => { + const next = new Set(prev) + if (on) next.add(id) + else next.delete(id) + return next + }) + } + + function toggleGroup(groupItems: MediaItem[], on: boolean): void { + setSelected((prev) => { + const next = new Set(prev) + for (const it of groupItems) { + if (on) next.add(it.id) + else next.delete(it.id) + } + return next + }) + } + + function downloadSelected(): void { + if (!selectedSourceId) return + const chosen = items.filter((it) => selected.has(it.id)) + enqueueItems(selectedSourceId, chosen) + setSelected(new Set()) + } + + function downloadPending(): void { + if (!selectedSourceId) return + enqueueItems(selectedSourceId, pendingItems) + } + + function pillClass(status: ItemStatus): string { + if (status === 'downloading' || status === 'queued') return mergeClasses(styles.pill, styles.pillDownloading) + if (status === 'completed') return mergeClasses(styles.pill, styles.pillCompleted) + if (status === 'error') return mergeClasses(styles.pill, styles.pillError) + return styles.pill + } + + return ( + <div className={styles.root}> + <div className={styles.header}> + <Subtitle2>Library</Subtitle2> + <Caption1 className={styles.sub}> + Index a channel or playlist once, then download it into organized folders. + </Caption1> + </div> + + <div className={styles.addRow}> + <Input + className={styles.addInput} + value={url} + onChange={(_, d) => setUrl(d.value)} + onKeyDown={(e) => e.key === 'Enter' && onIndex()} + placeholder="Paste a channel or playlist URL…" + size="large" + contentBefore={<LibraryRegular />} + disabled={indexing.active} + /> + <Button + size="large" + appearance="primary" + icon={indexing.active ? <Spinner size="tiny" /> : <SearchRegular />} + onClick={onIndex} + disabled={!url.trim() || indexing.active} + > + Index + </Button> + </div> + + <div className={styles.toolbar}> + <Button + size="small" + appearance="secondary" + icon={syncing ? <Spinner size="tiny" /> : <ArrowClockwiseRegular />} + onClick={onCheckNew} + disabled={syncing || watchedCount === 0} + > + Check {watchedCount > 0 ? `${watchedCount} watched` : 'watched'} for new + </Button> + {syncNote && <Caption1 className={styles.sub}>{syncNote}</Caption1>} + <div className={styles.toolbarSpacer} /> + <span className={styles.switchRow}> + <Caption1>Auto-download new</Caption1> + <Switch + checked={autoDownloadNew} + onChange={(_, d) => updateSettings({ autoDownloadNew: d.checked })} + aria-label="Auto-download new uploads" + /> + </span> + <span className={styles.switchRow}> + <Caption1>Daily sync</Caption1> + <Switch + checked={scheduled} + onChange={(_, d) => toggleScheduled(d.checked)} + aria-label="Daily background sync" + /> + </span> + </div> + + {indexing.active && ( + <div className={styles.progress}> + <Spinner size="tiny" /> + <Text> + {indexing.message ?? 'Indexing…'} + {indexing.current && indexing.total ? ` (${indexing.current}/${indexing.total})` : ''} + </Text> + </div> + )} + {error && <Caption1 className={styles.error}>{error}</Caption1>} + + {sources.length === 0 && !indexing.active ? ( + <div className={styles.empty}> + <LibraryRegular fontSize={40} /> + <Body1>No channels or playlists yet. Paste one above to index it.</Body1> + </div> + ) : ( + <div className={styles.list}> + {sources.map((src) => ( + <SourceCard + key={src.id} + styles={styles} + source={src} + expanded={selectedSourceId === src.id} + onToggleExpand={() => + selectSource(selectedSourceId === src.id ? null : src.id) + } + > + <div className={styles.detail}> + <div className={styles.actionRow}> + <Caption1 className={styles.srcSub}> + {items.length} videos · {pendingItems.length} pending · indexed{' '} + {relTime(src.lastIndexedAt)} + </Caption1> + <div className={styles.actionSpacer} /> + {selected.size > 0 ? ( + <> + <Button size="small" appearance="subtle" onClick={() => setSelected(new Set())}> + Clear + </Button> + <Button + size="small" + appearance="primary" + icon={<ArrowDownloadRegular />} + onClick={downloadSelected} + > + Download {Math.min(selected.size, MAX_ENQUEUE_BATCH)} selected + </Button> + </> + ) : ( + <Button + size="small" + appearance="primary" + icon={<ArrowDownloadRegular />} + onClick={downloadPending} + disabled={pendingItems.length === 0} + > + Download {Math.min(pendingItems.length, MAX_ENQUEUE_BATCH)} pending + </Button> + )} + <span className={styles.switchRow}> + <Caption1>Watch</Caption1> + <Switch + checked={!!src.watched} + onChange={(_, d) => setWatched(src.id, !!d.checked)} + aria-label={`Watch ${src.title} for new uploads`} + /> + </span> + <Button + size="small" + appearance="subtle" + icon={<ArrowSyncRegular />} + onClick={() => reindexSource(src.id)} + disabled={indexing.active} + > + Re-index + </Button> + <Button + size="small" + appearance="subtle" + icon={<DeleteRegular />} + onClick={() => removeSource(src.id)} + > + Remove + </Button> + </div> + + {groups.map((g) => { + const allOn = g.items.every((it) => selected.has(it.id)) + return ( + <div key={g.title} className={styles.group}> + <div className={styles.groupHead}> + <AppsListRegular /> + <span className={styles.groupTitle}>{g.title}</span> + <Caption1 className={styles.srcSub}>{g.items.length}</Caption1> + <Button + size="small" + appearance="subtle" + onClick={() => toggleGroup(g.items, !allOn)} + > + {allOn ? 'None' : 'All'} + </Button> + </div> + <div className={styles.rows}> + {g.items.map((it) => { + const status = effStatus(it) + return ( + <div key={it.id} className={styles.row}> + <Checkbox + checked={selected.has(it.id)} + onChange={(_, d) => toggle(it.id, !!d.checked)} + aria-label={`Select ${it.title}`} + /> + <div className={styles.rowMain}> + <span className={styles.rowTitle}> + {it.playlistIndex}. {it.title} + </span> + {it.durationLabel && ( + <Caption1 className={styles.rowMeta}>{it.durationLabel}</Caption1> + )} + </div> + <span className={pillClass(status)}>{STATUS_LABEL[status]}</span> + </div> + ) + })} + </div> + </div> + ) + })} + </div> + </SourceCard> + ))} + </div> + )} + </div> + ) +} + +/** One collapsible source card (header always shown; children render when expanded). */ +function SourceCard({ + styles, + source, + expanded, + onToggleExpand, + children +}: { + styles: ReturnType<typeof useStyles> + source: Source + expanded: boolean + onToggleExpand: () => void + children: React.ReactNode +}): React.JSX.Element { + return ( + <div className={styles.card}> + <div + className={styles.cardHead} + onClick={onToggleExpand} + role="button" + tabIndex={0} + onKeyDown={(e) => (e.key === 'Enter' || e.key === ' ') && onToggleExpand()} + aria-expanded={expanded} + > + {expanded ? <ChevronDownRegular /> : <ChevronRightRegular />} + <div className={styles.srcIcon}> + <VideoClipMultipleRegular /> + </div> + <div className={styles.srcMeta}> + <span className={styles.srcTitleRow}> + <span className={styles.srcTitle}>{source.title}</span> + {source.watched && ( + <span className={styles.watchBadge}> + <AlertRegular fontSize={11} /> Watching + </span> + )} + </span> + <Caption1 className={styles.srcSub}> + {source.kind === 'channel' ? 'Channel' : 'Playlist'} · {source.itemCount} videos + {source.channel && source.channel !== source.title ? ` · ${source.channel}` : ''} + </Caption1> + </div> + </div> + {expanded && children} + </div> + ) +} diff --git a/src/renderer/src/components/Sidebar.tsx b/src/renderer/src/components/Sidebar.tsx index a531351..9473a62 100644 --- a/src/renderer/src/components/Sidebar.tsx +++ b/src/renderer/src/components/Sidebar.tsx @@ -10,12 +10,13 @@ import { ArrowDownloadFilled, ArrowDownloadRegular, HistoryRegular, + LibraryRegular, SettingsRegular, WeatherMoonRegular, WeatherSunnyRegular } from '@fluentui/react-icons' -export type TabValue = 'downloads' | 'history' | 'settings' +export type TabValue = 'downloads' | 'library' | 'history' | 'settings' const useStyles = makeStyles({ root: { @@ -114,6 +115,7 @@ const useStyles = makeStyles({ const NAV: { value: TabValue; label: string; icon: React.JSX.Element }[] = [ { value: 'downloads', label: 'Downloads', icon: <ArrowDownloadRegular /> }, + { value: 'library', label: 'Library', icon: <LibraryRegular /> }, { value: 'history', label: 'History', icon: <HistoryRegular /> }, { value: 'settings', label: 'Settings', icon: <SettingsRegular /> } ] diff --git a/src/renderer/src/main.tsx b/src/renderer/src/main.tsx index 45e7133..524de57 100644 --- a/src/renderer/src/main.tsx +++ b/src/renderer/src/main.tsx @@ -31,6 +31,7 @@ if (import.meta.env.DEV && !window.api) { customCommandEnabled: false, defaultTemplateId: null, notifyOnComplete: true, + autoDownloadNew: true, hasCompletedOnboarding: true } // Stands in for the cookies.txt file's mtime — lets the Cookies card's @@ -139,7 +140,58 @@ if (import.meta.env.DEV && !window.api) { getSystemTheme: async () => ({ shouldUseDarkColors: false, shouldUseHighContrastColors: false }), onSystemThemeUpdate: () => () => {}, openHighContrastSettings: async () => {}, - onExternalUrl: () => () => {} + onExternalUrl: () => () => {}, + // Media-manager sources — a seeded channel so the (Phase H) Library view has + // demo data in this browser-only preview. + listSources: async () => [ + { + id: 'src-demo', + url: 'https://www.youtube.com/@DevChannel', + kind: 'channel', + title: 'DevChannel', + channel: 'DevChannel', + addedAt: Date.now() - 86_400_000, + lastIndexedAt: Date.now() - 3_600_000, + itemCount: 7 + } + ], + indexSource: async (url) => { + await new Promise((r) => setTimeout(r, 800)) // simulate the index walk + return { + ok: true, + source: { + id: 'src-demo', + url, + kind: 'channel', + title: 'DevChannel', + channel: 'DevChannel', + addedAt: Date.now(), + lastIndexedAt: Date.now(), + itemCount: 7 + }, + itemCount: 7 + } + }, + reindexSource: async () => ({ ok: true, itemCount: 7, newCount: 0 }), + removeSource: async () => [], + markSourceItemDownloaded: async () => [], + setSourceWatched: async () => [], + syncSources: async () => ({ ok: true, newItems: [] }), + getScheduledSync: async () => ({ enabled: false }), + setScheduledSync: async (enabled: boolean) => ({ enabled }), + listSourceItems: async (sourceId) => + Array.from({ length: 7 }, (_, i) => ({ + id: `${sourceId}:vid${i + 1}`, + sourceId, + videoId: `vid${i + 1}`, + title: `Episode ${i + 1}`, + url: `https://youtube.com/watch?v=vid${i + 1}`, + playlistTitle: i < 5 ? 'Full Electron Course' : 'Uploads', + playlistIndex: i < 5 ? i + 1 : i - 4, + durationLabel: `${10 + i}:0${i}`, + downloaded: i < 2 + })), + onIndexProgress: () => () => {} } } diff --git a/src/renderer/src/store/downloads.ts b/src/renderer/src/store/downloads.ts index 5b2f727..d9e800c 100644 --- a/src/renderer/src/store/downloads.ts +++ b/src/renderer/src/store/downloads.ts @@ -1,7 +1,8 @@ import { create } from 'zustand' -import type { DownloadEvent, DownloadMeta, DownloadOptions } from '@shared/ipc' +import type { DownloadEvent, DownloadMeta, DownloadOptions, CollectionContext } from '@shared/ipc' import { useSettings } from './settings' import { useHistory } from './history' +import { useSources } from './sources' export type MediaKind = 'video' | 'audio' @@ -42,6 +43,10 @@ export interface DownloadItem { incognito?: boolean /** metadata already fetched at probe time; forwarded to main so it skips a redundant probe */ probedMeta?: DownloadMeta + /** media-manager folder context — files this into <channel>/<playlist>/<NNN> - <title> */ + collection?: CollectionContext + /** when set, the source MediaItem id this download came from — marked downloaded on completion */ + mediaItemId?: string } export const QUALITY_OPTIONS: Record<MediaKind, string[]> = { @@ -68,6 +73,8 @@ interface DownloadState { options?: DownloadOptions extraArgs?: string incognito?: boolean + collection?: CollectionContext + mediaItemId?: string } ) => void retry: (id: string) => void @@ -210,6 +217,11 @@ function startFakeTicker( completedAt: Date.now() }) } + // Mirror the real applyEvent path so collection completions mark their + // source item downloaded in the preview too. + if (finished?.mediaItemId) { + useSources.getState().markDownloaded(finished.mediaItemId, finished.filePath) + } get().pump() // a slot just freed — promote the next queued item } }, 650) @@ -240,7 +252,8 @@ export const useDownloads = create<DownloadState>((set, get) => { formatHasAudio: item.formatHasAudio, options: item.options, extraArgs: item.extraArgs, - meta: item.probedMeta + meta: item.probedMeta, + collection: item.collection }) .then((res) => { if (!res.ok) markError(item.id, res.error) @@ -301,6 +314,8 @@ export const useDownloads = create<DownloadState>((set, get) => { options: opts?.options, extraArgs: opts?.extraArgs, incognito: opts?.incognito, + collection: opts?.collection, + mediaItemId: opts?.mediaItemId, probedMeta } set((s) => ({ items: [item, ...s.items] })) @@ -411,6 +426,10 @@ export const useDownloads = create<DownloadState>((set, get) => { completedAt: Date.now() }) } + // Persist media-manager completion so an incremental re-sync skips it. + if (item?.mediaItemId) { + useSources.getState().markDownloaded(item.mediaItemId, item.filePath) + } } // A finished item frees a slot — promote whatever is queued next. if (ev.type === 'done' || ev.type === 'error') pump() diff --git a/src/renderer/src/store/settings.ts b/src/renderer/src/store/settings.ts index bcd9cd5..22a3ad5 100644 --- a/src/renderer/src/store/settings.ts +++ b/src/renderer/src/store/settings.ts @@ -25,6 +25,7 @@ const FALLBACK: Settings = { customCommandEnabled: false, defaultTemplateId: null, notifyOnComplete: true, + autoDownloadNew: true, // True in preview so design work isn't blocked behind the welcome screen; // a real first launch gets `false` from main/settings.ts's DEFAULTS instead. hasCompletedOnboarding: PREVIEW diff --git a/src/renderer/src/store/sources.ts b/src/renderer/src/store/sources.ts new file mode 100644 index 0000000..1b89a4c --- /dev/null +++ b/src/renderer/src/store/sources.ts @@ -0,0 +1,288 @@ +import { create } from 'zustand' +import type { Source, MediaItem, IndexProgress } from '@shared/ipc' +import { useDownloads } from './downloads' +import { useSettings } from './settings' + +// True in the standalone browser preview (no Electron preload). +const PREVIEW = typeof window === 'undefined' || !window.electron + +// Cap on how many items one "Download" click enqueues, so an entire channel +// can't flood the live queue at once — the rest stay pending for a follow-up +// click. (The automatic incremental feeder is Phase I; see ROADMAP-PINCHFLAT.md.) +export const MAX_ENQUEUE_BATCH = 100 + +// --- Preview seed data ------------------------------------------------------ + +function seedItems(sourceId: string, playlist: string, n: number, downloadedUpTo = 0): MediaItem[] { + return Array.from({ length: n }, (_, i) => ({ + id: `${sourceId}:${playlist}-${i + 1}`, + sourceId, + videoId: `${playlist}-${i + 1}`, + title: `${playlist} — part ${i + 1}`, + url: `https://youtube.com/watch?v=${sourceId}${i + 1}`, + playlistTitle: playlist, + playlistIndex: i + 1, + durationLabel: `${10 + i}:${String(i * 7).padStart(2, '0')}`, + downloaded: i < downloadedUpTo + })) +} + +const seedSources: Source[] = PREVIEW + ? [ + { + id: 'src-dev', + url: 'https://www.youtube.com/@DevChannel', + kind: 'channel', + title: 'DevChannel', + channel: 'DevChannel', + addedAt: Date.now() - 86_400_000, + lastIndexedAt: Date.now() - 3_600_000, + itemCount: 9 + }, + { + id: 'src-mix', + url: 'https://www.youtube.com/playlist?list=PLmix', + kind: 'playlist', + title: 'Lo-fi Coding Mixes', + channel: 'ChillStudio', + addedAt: Date.now() - 2 * 86_400_000, + lastIndexedAt: Date.now() - 7_200_000, + itemCount: 5 + } + ] + : [] + +const seedItemsBySource: Record<string, MediaItem[]> = PREVIEW + ? { + 'src-dev': [ + ...seedItems('src-dev', 'Full Electron Course', 6, 2), + ...seedItems('src-dev', 'Uploads', 3) + ], + 'src-mix': seedItems('src-mix', 'Lo-fi Coding Mixes', 5, 1) + } + : {} + +// --- Store ------------------------------------------------------------------ + +/** Live indexing status for the add-source bar. */ +interface IndexingState { + active: boolean + url?: string + message?: string + current?: number + total?: number +} + +interface SourcesState { + sources: Source[] + /** media items per source, lazily loaded when a source is expanded */ + itemsBySource: Record<string, MediaItem[]> + /** which source is expanded in the library view (null = none) */ + selectedSourceId: string | null + indexing: IndexingState + loadSources: () => void + selectSource: (id: string | null) => void + /** index a pasted channel/playlist URL; resolves with ok + an error message */ + indexSource: (url: string) => Promise<{ ok: boolean; error?: string }> + reindexSource: (id: string) => Promise<void> + removeSource: (id: string) => void + /** + * Enqueue the given media items into the download queue with folder context, + * capped at MAX_ENQUEUE_BATCH. Returns how many were actually enqueued. + */ + enqueueItems: (sourceId: string, items: MediaItem[]) => number + /** mark an item downloaded once its queue download completes (called by the downloads store) */ + markDownloaded: (itemId: string, filePath?: string) => void + /** toggle a source's watched-for-new-uploads flag (Phase J) */ + setWatched: (id: string, watched: boolean) => void + /** whether a sync is currently running (the "Check for new" button's busy state) */ + syncing: boolean + /** + * Re-index all watched sources and (when autoDownloadNew is on) enqueue the new + * videos. Resolves with how many new videos were found. + */ + syncWatched: () => Promise<number> +} + +export const useSources = create<SourcesState>((set, get) => ({ + sources: seedSources, + itemsBySource: seedItemsBySource, + selectedSourceId: null, + indexing: { active: false }, + syncing: false, + + loadSources: () => { + if (PREVIEW) return + window.api + .listSources() + .then((sources) => set({ sources })) + .catch(() => {}) + }, + + selectSource: (id) => { + set({ selectedSourceId: id }) + // Lazily fetch a source's items the first time it's expanded. + if (id && !get().itemsBySource[id] && !PREVIEW) { + window.api + .listSourceItems(id) + .then((items) => set((s) => ({ itemsBySource: { ...s.itemsBySource, [id]: items } }))) + .catch(() => {}) + } + }, + + indexSource: async (url) => { + set({ indexing: { active: true, url, message: 'Reading source…' } }) + if (PREVIEW) { + await new Promise((r) => setTimeout(r, 900)) // simulate the index walk + set({ indexing: { active: false } }) + return { ok: true } + } + try { + const res = await window.api.indexSource(url) + set({ indexing: { active: false } }) + if (res.ok) { + get().loadSources() + if (res.source) get().selectSource(res.source.id) + } + return { ok: res.ok, error: res.error } + } catch (e) { + set({ indexing: { active: false } }) + return { ok: false, error: e instanceof Error ? e.message : String(e) } + } + }, + + reindexSource: async (id) => { + const src = get().sources.find((s) => s.id === id) + set({ indexing: { active: true, url: src?.url, message: 'Re-indexing…' } }) + if (PREVIEW) { + await new Promise((r) => setTimeout(r, 700)) + set({ indexing: { active: false } }) + return + } + try { + const res = await window.api.reindexSource(id) + set({ indexing: { active: false } }) + if (res.ok) { + get().loadSources() + const items = await window.api.listSourceItems(id) + set((s) => ({ itemsBySource: { ...s.itemsBySource, [id]: items } })) + } + } catch { + set({ indexing: { active: false } }) + } + }, + + removeSource: (id) => { + set((s) => ({ + sources: s.sources.filter((x) => x.id !== id), + selectedSourceId: s.selectedSourceId === id ? null : s.selectedSourceId + })) + if (!PREVIEW) window.api.removeSource(id).catch(() => {}) + }, + + enqueueItems: (sourceId, items) => { + const src = get().sources.find((s) => s.id === sourceId) + const settings = useSettings.getState() + const kind = settings.defaultKind + const quality = kind === 'video' ? settings.defaultVideoQuality : settings.defaultAudioQuality + const batch = items.slice(0, MAX_ENQUEUE_BATCH) + const add = useDownloads.getState().addFromUrl + for (const it of batch) { + add(it.url, kind, quality, { + title: it.title, + channel: src?.channel, + durationLabel: it.durationLabel, + collection: { + channel: src?.channel || src?.title || 'Channel', + playlist: it.playlistTitle, + index: it.playlistIndex + }, + mediaItemId: it.id + }) + } + return batch.length + }, + + markDownloaded: (itemId, filePath) => { + set((s) => { + const next: Record<string, MediaItem[]> = {} + let changed = false + for (const [sid, list] of Object.entries(s.itemsBySource)) { + if (list.some((m) => m.id === itemId)) { + changed = true + next[sid] = list.map((m) => + m.id === itemId ? { ...m, downloaded: true, downloadedAt: Date.now(), filePath } : m + ) + } else { + next[sid] = list + } + } + return changed ? { itemsBySource: next } : {} + }) + if (!PREVIEW) window.api.markSourceItemDownloaded(itemId, filePath).catch(() => {}) + }, + + setWatched: (id, watched) => { + set((s) => ({ sources: s.sources.map((x) => (x.id === id ? { ...x, watched } : x)) })) + if (!PREVIEW) window.api.setSourceWatched(id, watched).catch(() => {}) + }, + + syncWatched: async () => { + if (get().syncing) return 0 + set({ syncing: true }) + try { + if (PREVIEW) { + await new Promise((r) => setTimeout(r, 700)) + return 0 // preview has no real feed to poll + } + const res = await window.api.syncSources() + if (!res.ok) return 0 + get().loadSources() + // Refresh any expanded source's items so new videos appear in the tree. + const sel = get().selectedSourceId + if (sel) { + const items = await window.api.listSourceItems(sel) + set((s) => ({ itemsBySource: { ...s.itemsBySource, [sel]: items } })) + } + // Auto-enqueue the new videos, grouped by source, when the setting is on. + if (useSettings.getState().autoDownloadNew && res.newItems.length > 0) { + const bySource = new Map<string, MediaItem[]>() + for (const it of res.newItems) { + const arr = bySource.get(it.sourceId) ?? [] + arr.push(it) + bySource.set(it.sourceId, arr) + } + for (const [sid, items] of bySource) get().enqueueItems(sid, items) + } + return res.newItems.length + } finally { + set({ syncing: false }) + } + } +})) + +// Load persisted sources on startup, and subscribe to live indexing progress. +if (!PREVIEW) { + window.api + .listSources() + .then((sources) => { + useSources.setState({ sources }) + // If any source is watched, kick off a sync shortly after launch (covers the + // scheduled `--sync` launch and a normal launch alike). + if (sources.some((s) => s.watched)) { + setTimeout(() => useSources.getState().syncWatched(), 1500) + } + }) + .catch(() => {}) + window.api.onIndexProgress((p: IndexProgress) => { + useSources.setState({ + indexing: { + active: p.phase !== 'done' && p.phase !== 'error', + url: p.url, + message: p.message, + current: p.current, + total: p.total + } + }) + }) +} diff --git a/src/shared/ipc.ts b/src/shared/ipc.ts index 5789414..f7689db 100644 --- a/src/shared/ipc.ts +++ b/src/shared/ipc.ts @@ -40,7 +40,24 @@ export const IpcChannels = { openHighContrastSettings: 'shell:open-high-contrast-settings', /** main → renderer push channel: a URL handed to AeroFetch from outside the app * (the aerofetch:// protocol, or a .url file via Explorer's "Send to" menu) */ - externalUrl: 'external-url' + externalUrl: 'external-url', + // --- Sources & media-manager indexing (Pinchflat-style, see ROADMAP-PINCHFLAT.md) --- + sourcesList: 'sources:list', + sourceIndex: 'sources:index', + sourceReindex: 'sources:reindex', + sourceRemove: 'sources:remove', + sourceItems: 'sources:items', + /** mark a media item downloaded once its collection download completes */ + sourceItemDownloaded: 'sources:item-downloaded', + /** toggle whether a source is watched for new uploads (Phase J) */ + sourceSetWatched: 'sources:set-watched', + /** re-index all watched sources and return the videos that are new */ + sourcesSync: 'sources:sync', + /** get / set the Windows scheduled-sync task (Task Scheduler) */ + scheduledSyncGet: 'sources:scheduled-sync-get', + scheduledSyncSet: 'sources:scheduled-sync-set', + /** main → renderer push channel for live indexing progress */ + indexProgress: 'sources:index-progress' } as const /** Sentinel format id meaning "let yt-dlp pick the best video+audio". */ @@ -134,6 +151,12 @@ export interface DownloadOptions { embedThumbnail: boolean /** crop embedded audio artwork to a centred square (Seal's "crop artwork") */ cropThumbnail: boolean + /** write a .info.json metadata sidecar (Jellyfin/Plex/Kodi ingest — Phase K) */ + writeInfoJson: boolean + /** write the thumbnail as a separate image file alongside the video */ + writeThumbnailFile: boolean + /** write the video description to a .description sidecar file */ + writeDescription: boolean } export const DEFAULT_DOWNLOAD_OPTIONS: DownloadOptions = { @@ -149,7 +172,10 @@ export const DEFAULT_DOWNLOAD_OPTIONS: DownloadOptions = { embedChapters: false, embedMetadata: true, embedThumbnail: true, - cropThumbnail: false + cropThumbnail: false, + writeInfoJson: false, + writeThumbnailFile: false, + writeDescription: false } export interface YtdlpVersionResult { @@ -225,6 +251,24 @@ export interface ProbeResult { error?: string } +/** + * Media-manager folder context for a collection download (see ROADMAP-PINCHFLAT.md + * Phase G). When present on a StartDownloadOptions, the file is filed into + * `<outputDir>/<channel>/<playlist>/<NNN> - <title>.<ext>` instead of the flat + * filenameTemplate. The directory segments are sanitized for Windows at argv-build + * time, and `index` (the 1-based playlist position from the persisted MediaItem) + * drives the NNN prefix — not yt-dlp's %(playlist_index)s, which is empty under + * the per-video `--no-playlist` path these downloads still take. + */ +export interface CollectionContext { + /** channel / uploader name — the top folder level */ + channel: string + /** playlist title — the second folder level ('Uploads' for the catch-all) */ + playlist: string + /** 1-based position within the playlist, for the NNN filename prefix */ + index: number +} + /** Sent renderer → main to kick off a download. The renderer owns the id. */ export interface StartDownloadOptions { id: string @@ -255,6 +299,12 @@ export interface StartDownloadOptions { * race where the late probe overwrites a good title the renderer supplied. */ meta?: DownloadMeta + /** + * When set, this is a media-manager (collection) download — file it into + * `<channel>/<playlist>/<NNN> - <title>` folders instead of the flat + * filenameTemplate. See CollectionContext. + */ + collection?: CollectionContext } export interface StartDownloadResult { @@ -327,6 +377,8 @@ export interface Settings { defaultTemplateId: string | null /** show a native OS notification when a download finishes or fails */ notifyOnComplete: boolean + /** auto-enqueue newly-found videos from watched sources during a sync (Phase J) */ + autoDownloadNew: boolean /** false until the first-run welcome screen has been dismissed */ hasCompletedOnboarding: boolean } @@ -410,3 +462,96 @@ export interface BackupImportResult { ok: boolean error?: string } + +// --- Sources & media-manager indexing (Pinchflat-style) --------------------- +// See ROADMAP-PINCHFLAT.md. A Source (channel/playlist) is indexed once into a +// persisted list of MediaItem records; the live download queue then pulls from +// that list a batch at a time, so "download an entire channel" never has to hold +// the whole collection in the queue at once. + +/** Whether a Source is a whole channel or a single playlist. */ +export type SourceKind = 'channel' | 'playlist' + +/** + * A monitored channel or playlist that AeroFetch has indexed. Its full video + * list lives as MediaItem records (one JSON store), keyed back by `id`. + */ +export interface Source { + id: string + /** the URL the user added (used for re-indexing) */ + url: string + kind: SourceKind + title: string + /** channel / uploader name, when known */ + channel?: string + /** epoch ms the source was first added */ + addedAt: number + /** epoch ms of the last successful (re)index; undefined if never indexed */ + lastIndexedAt?: number + /** number of MediaItems at the last index (cached for list display) */ + itemCount: number + /** watched for new uploads — included in scheduled / startup sync (Phase J) */ + watched?: boolean + /** YouTube RSS feed URL for cheap "anything new?" checks; undefined if unknown */ + feedUrl?: string +} + +/** + * One video discovered while indexing a Source. `playlistTitle`/`playlistIndex` + * drive the on-disk folder layout (<Channel>/<Playlist>/<NNN> - <Title>); the + * `downloaded` flag + `filePath` let the library view show per-item state and + * let re-syncs skip what's already on disk. + */ +export interface MediaItem { + /** globally unique, formed as `${sourceId}:${videoId}` */ + id: string + sourceId: string + /** the yt-dlp video id — the dedup key within a source */ + videoId: string + title: string + url: string + /** the playlist this item is filed under; 'Uploads' for videos in no playlist */ + playlistTitle: string + /** 1-based position within its playlist, for NNN numbering */ + playlistIndex: number + durationLabel?: string + downloaded: boolean + downloadedAt?: number + filePath?: string +} + +/** Live progress pushed while a Source is being indexed (main → renderer). */ +export interface IndexProgress { + /** the Source URL being indexed (correlates events back to the request) */ + url: string + phase: 'start' | 'playlists' | 'playlist' | 'uploads' | 'done' | 'error' + /** human-readable status line */ + message: string + /** for the per-playlist phase: playlists processed so far / total */ + current?: number + total?: number +} + +/** Result of indexing (or re-indexing) a Source. */ +export interface IndexSourceResult { + ok: boolean + source?: Source + itemCount?: number + /** how many videos were new since the previous index (all of them on first index) */ + newCount?: number + error?: string +} + +/** Result of syncing all watched sources: the videos found new across them. */ +export interface SyncResult { + ok: boolean + /** newly-discovered media items across all watched sources (empty if nothing new) */ + newItems: MediaItem[] + error?: string +} + +/** Whether the Windows scheduled-sync task is currently registered. */ +export interface ScheduledSyncStatus { + enabled: boolean + error?: string +} diff --git a/test/buildArgs.test.ts b/test/buildArgs.test.ts index 9ed95aa..bd14396 100644 --- a/test/buildArgs.test.ts +++ b/test/buildArgs.test.ts @@ -3,6 +3,8 @@ import { buildArgs, parseExtraArgs, formatCommandLine, + sanitizeDirSegment, + collectionOutputTemplate, CROP_SQUARE_PPA, ARIA2C_ARGS, type AccessOptions @@ -360,3 +362,91 @@ describe('buildArgs extraArgs', () => { expect(withEmpty).toEqual(withoutParam) }) }) + +// --- Media-server sidecar files (Phase K) ----------------------------------- + +describe('sidecar files', () => { + it('emits --write-info-json / --write-thumbnail / --write-description when enabled', () => { + const argv = build( + opts(), + dlo({ writeInfoJson: true, writeThumbnailFile: true, writeDescription: true }) + ) + expect(argv).toContain('--write-info-json') + expect(argv).toContain('--write-thumbnail') + expect(argv).toContain('--write-description') + }) + + it('emits none of them by default', () => { + const argv = build(opts(), dlo()) + expect(argv).not.toContain('--write-info-json') + expect(argv).not.toContain('--write-description') + // (--write-thumbnail is sidecar-only here; embedThumbnail uses --embed-thumbnail) + expect(argv).not.toContain('--write-thumbnail') + }) +}) + +// --- Collection folder paths (Phase G, media-manager) ----------------------- + +describe('sanitizeDirSegment', () => { + it('keeps an ordinary name (incl. spaces and hyphens) intact', () => { + expect(sanitizeDirSegment('Linus Tech Tips')).toBe('Linus Tech Tips') + expect(sanitizeDirSegment('Two-Minute Papers')).toBe('Two-Minute Papers') + }) + + it('replaces Windows-illegal characters with a space and collapses runs', () => { + expect(sanitizeDirSegment('A/B\\C:D*E?F')).toBe('A B C D E F') + expect(sanitizeDirSegment('Best of 2024 <Live>')).toBe('Best of 2024 Live') + }) + + it('strips leading/trailing dots and spaces, neutralising `..` traversal', () => { + expect(sanitizeDirSegment('..')).toBe('Untitled') + expect(sanitizeDirSegment('../../etc')).toBe('etc') + expect(sanitizeDirSegment(' trailing. ')).toBe('trailing') + expect(sanitizeDirSegment('My Playlist.')).toBe('My Playlist') + }) + + it('prefixes reserved Windows device names', () => { + expect(sanitizeDirSegment('CON')).toBe('_CON') + expect(sanitizeDirSegment('com1')).toBe('_com1') + }) + + it('falls back to Untitled for empty / all-illegal input', () => { + expect(sanitizeDirSegment('')).toBe('Untitled') + expect(sanitizeDirSegment('???')).toBe('Untitled') + }) + + it('caps length to keep the overall path bounded', () => { + expect(sanitizeDirSegment('x'.repeat(200)).length).toBe(80) + }) +}) + +describe('collectionOutputTemplate', () => { + const ctx = { channel: 'DevChannel', playlist: 'Full Course', index: 3 } + + it('files into <out>/<channel>/<playlist>/<NNN> - <filename>', () => { + const t = collectionOutputTemplate('C:/out', ctx, '%(title)s.%(ext)s') + // normalise separators so the assertion is OS-independent + expect(t.replace(/\\/g, '/')).toBe('C:/out/DevChannel/Full Course/003 - %(title)s.%(ext)s') + }) + + it('zero-pads the index to three digits and clamps bad values to 001', () => { + expect(collectionOutputTemplate('C:/o', { ...ctx, index: 42 }, 'f').replace(/\\/g, '/')).toContain( + '/042 - f' + ) + expect(collectionOutputTemplate('C:/o', { ...ctx, index: 0 }, 'f').replace(/\\/g, '/')).toContain( + '/001 - f' + ) + expect( + collectionOutputTemplate('C:/o', { ...ctx, index: NaN }, 'f').replace(/\\/g, '/') + ).toContain('/001 - f') + }) + + it('sanitizes the channel and playlist folder names', () => { + const t = collectionOutputTemplate( + 'C:/out', + { channel: 'A/B', playlist: '..', index: 1 }, + '%(title)s.%(ext)s' + ) + expect(t.replace(/\\/g, '/')).toBe('C:/out/A B/Untitled/001 - %(title)s.%(ext)s') + }) +}) diff --git a/test/indexer.test.ts b/test/indexer.test.ts new file mode 100644 index 0000000..bfeac7d --- /dev/null +++ b/test/indexer.test.ts @@ -0,0 +1,202 @@ +import { describe, it, expect } from 'vitest' +import { + classifySource, + buildMediaItems, + mergeItemsPreservingState, + buildFeedUrl, + parseRssVideoIds, + entryUrl, + fmtDuration, + stripTabSuffix, + stableSourceId, + type RawEntry +} from '../src/main/indexerCore' +import type { MediaItem } from '@shared/ipc' + +describe('classifySource', () => { + it('classifies channel handle / id / c / user forms and strips the tab', () => { + expect(classifySource('https://www.youtube.com/@LinusTechTips')).toEqual({ + kind: 'channel', + base: 'https://www.youtube.com/@LinusTechTips' + }) + expect(classifySource('https://youtube.com/@Foo/videos')).toEqual({ + kind: 'channel', + base: 'https://www.youtube.com/@Foo' + }) + expect(classifySource('https://www.youtube.com/channel/UC123/playlists')?.base).toBe( + 'https://www.youtube.com/channel/UC123' + ) + expect(classifySource('https://www.youtube.com/c/SomeName')?.kind).toBe('channel') + expect(classifySource('https://www.youtube.com/user/Legacy')?.kind).toBe('channel') + }) + + it('classifies a playlist by its list= param and canonicalises it', () => { + expect(classifySource('https://www.youtube.com/playlist?list=PL_abc')).toEqual({ + kind: 'playlist', + base: 'https://www.youtube.com/playlist?list=PL_abc' + }) + // a watch URL that also carries list= is treated as the playlist + expect(classifySource('https://www.youtube.com/watch?v=xyz&list=PLfoo')?.kind).toBe('playlist') + }) + + it('returns null for a lone video, a bad URL, or a non-YouTube host', () => { + expect(classifySource('https://www.youtube.com/watch?v=dQw4w9WgXcQ')).toBeNull() + expect(classifySource('not a url')).toBeNull() + expect(classifySource('ftp://example.com/@chan')).toBeNull() + expect(classifySource('https://vimeo.com/channels/staffpicks')).toBeNull() + }) +}) + +describe('entryUrl', () => { + it('prefers an explicit http(s) url, else builds a watch url from the id', () => { + expect(entryUrl({ url: 'https://youtu.be/abc' })).toBe('https://youtu.be/abc') + expect(entryUrl({ webpage_url: 'https://x/y' })).toBe('https://x/y') + expect(entryUrl({ id: 'vid123' })).toBe('https://www.youtube.com/watch?v=vid123') + }) + it('rejects a non-http url and returns null when nothing usable is present', () => { + expect(entryUrl({ url: 'javascript:alert(1)' })).toBeNull() + expect(entryUrl({})).toBeNull() + }) +}) + +describe('fmtDuration', () => { + it('formats seconds as M:SS or H:MM:SS', () => { + expect(fmtDuration(0)).toBe('0:00') + expect(fmtDuration(75)).toBe('1:15') + expect(fmtDuration(3661)).toBe('1:01:01') + expect(fmtDuration(undefined)).toBeUndefined() + expect(fmtDuration(NaN)).toBeUndefined() + }) +}) + +describe('stripTabSuffix', () => { + it('strips a trailing " - <Tab>" suffix only', () => { + expect(stripTabSuffix('Creator - Videos')).toBe('Creator') + expect(stripTabSuffix('Creator - Playlists')).toBe('Creator') + expect(stripTabSuffix('Just A Name')).toBe('Just A Name') + expect(stripTabSuffix('Tech - Tips')).toBe('Tech - Tips') // 'Tips' isn't a tab + expect(stripTabSuffix(undefined)).toBeUndefined() + }) +}) + +describe('stableSourceId', () => { + it('is deterministic and 8 hex chars', () => { + const a = stableSourceId('https://www.youtube.com/@Foo') + expect(a).toMatch(/^[0-9a-f]{8}$/) + expect(stableSourceId('https://www.youtube.com/@Foo')).toBe(a) + expect(stableSourceId('https://www.youtube.com/@Bar')).not.toBe(a) + }) +}) + +describe('buildMediaItems', () => { + const vids = (...ids: string[]): RawEntry[] => ids.map((id) => ({ id, title: `T-${id}` })) + + it('files videos under their playlist with 1-based per-playlist index', () => { + const items = buildMediaItems('src1', [{ title: 'Series A', entries: vids('a', 'b') }], []) + expect(items).toHaveLength(2) + expect(items[0]).toMatchObject({ + id: 'src1:a', + sourceId: 'src1', + videoId: 'a', + playlistTitle: 'Series A', + playlistIndex: 1, + downloaded: false + }) + expect(items[1]).toMatchObject({ videoId: 'b', playlistIndex: 2 }) + }) + + it('dedups across playlists — the first playlist a video appears in wins', () => { + const items = buildMediaItems( + 'src1', + [ + { title: 'First', entries: vids('x', 'y') }, + { title: 'Second', entries: vids('y', 'z') } + ], + [] + ) + expect(items.map((i) => i.videoId)).toEqual(['x', 'y', 'z']) + expect(items.find((i) => i.videoId === 'y')?.playlistTitle).toBe('First') + }) + + it('falls back to the synthetic Uploads folder for videos in no playlist', () => { + const items = buildMediaItems('src1', [{ title: 'P', entries: vids('a') }], vids('a', 'b')) + // 'a' already filed under P; only 'b' becomes an Uploads item + expect(items).toHaveLength(2) + expect(items.find((i) => i.videoId === 'a')?.playlistTitle).toBe('P') + const b = items.find((i) => i.videoId === 'b') + expect(b).toMatchObject({ playlistTitle: 'Uploads', playlistIndex: 2 }) + }) + + it('skips entries with no id or no resolvable URL', () => { + const items = buildMediaItems('src1', [{ title: 'P', entries: [{ title: 'no id' }] }], []) + expect(items).toHaveLength(0) + }) +}) + +describe('mergeItemsPreservingState', () => { + const item = (videoId: string, over: Partial<MediaItem> = {}): MediaItem => ({ + id: `s:${videoId}`, + sourceId: 's', + videoId, + title: videoId, + url: `https://youtube.com/watch?v=${videoId}`, + playlistTitle: 'P', + playlistIndex: 1, + downloaded: false, + ...over + }) + + it('carries downloaded state forward for videos already on disk', () => { + const existing = [item('a', { downloaded: true, downloadedAt: 111, filePath: 'C:/a.mp4' })] + const fresh = [item('a'), item('b')] + const { items, newCount } = mergeItemsPreservingState(existing, fresh) + expect(newCount).toBe(1) // only 'b' is new + expect(items.find((i) => i.videoId === 'a')).toMatchObject({ + downloaded: true, + downloadedAt: 111, + filePath: 'C:/a.mp4' + }) + expect(items.find((i) => i.videoId === 'b')?.downloaded).toBe(false) + }) + + it('adopts the fresh membership/order and drops vanished videos', () => { + const existing = [item('a', { downloaded: true }), item('gone', { downloaded: true })] + const fresh = [item('a', { playlistTitle: 'Moved', playlistIndex: 5 })] + const { items, newCount } = mergeItemsPreservingState(existing, fresh) + expect(items).toHaveLength(1) // 'gone' dropped + expect(newCount).toBe(0) + // fresh playlist assignment wins, but downloaded state is preserved + expect(items[0]).toMatchObject({ playlistTitle: 'Moved', playlistIndex: 5, downloaded: true }) + }) + + it('counts every video as new on a first index (empty existing)', () => { + const { newCount } = mergeItemsPreservingState([], [item('a'), item('b'), item('c')]) + expect(newCount).toBe(3) + }) +}) + +describe('buildFeedUrl', () => { + it('builds a channel feed by channel_id and a playlist feed by playlist_id', () => { + expect(buildFeedUrl('channel', 'UCabc')).toBe( + 'https://www.youtube.com/feeds/videos.xml?channel_id=UCabc' + ) + expect(buildFeedUrl('playlist', 'PLxyz')).toBe( + 'https://www.youtube.com/feeds/videos.xml?playlist_id=PLxyz' + ) + }) + it('returns undefined when the id is missing', () => { + expect(buildFeedUrl('channel', undefined)).toBeUndefined() + }) +}) + +describe('parseRssVideoIds', () => { + it('extracts every <yt:videoId> from a feed body, in order', () => { + const xml = ` + <feed><entry><yt:videoId>aaa111</yt:videoId></entry> + <entry><yt:videoId> bbb222 </yt:videoId></entry></feed>` + expect(parseRssVideoIds(xml)).toEqual(['aaa111', 'bbb222']) + }) + it('returns an empty array for a feed with no entries', () => { + expect(parseRssVideoIds('<feed></feed>')).toEqual([]) + }) +})