20 Commits

Author SHA1 Message Date
Alex Shnitman
e601ce99f5 add file command to the docker image (fixes #870) 2026-01-08 22:08:44 +02:00
Alex
a74b201ed8 Merge pull request #862 from AlvinRamoutar/task/repair-persistentqueues
feature/repair-persistent-queues
2026-01-08 21:29:23 +02:00
AlvinRamoutar
191f17ee38 syntax changes + null logic update for dbm repair 2026-01-05 18:13:42 -05:00
Alex
a002af9bf2 Merge pull request #864 from alexta69/dependabot/github_actions/github-actions-151e9c363d
Bump the github-actions group with 5 updates
2026-01-02 08:21:27 +02:00
dependabot[bot]
37aaa29efb Bump the github-actions group with 5 updates
Bumps the github-actions group with 5 updates:

| Package | From | To |
| --- | --- | --- |
| [actions/checkout](https://github.com/actions/checkout) | `4` | `6` |
| [docker/build-push-action](https://github.com/docker/build-push-action) | `5` | `6` |
| [softprops/action-gh-release](https://github.com/softprops/action-gh-release) | `1` | `2` |
| [actions/setup-python](https://github.com/actions/setup-python) | `4` | `6` |
| [astral-sh/setup-uv](https://github.com/astral-sh/setup-uv) | `6` | `7` |


Updates `actions/checkout` from 4 to 6
- [Release notes](https://github.com/actions/checkout/releases)
- [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md)
- [Commits](https://github.com/actions/checkout/compare/v4...v6)

Updates `docker/build-push-action` from 5 to 6
- [Release notes](https://github.com/docker/build-push-action/releases)
- [Commits](https://github.com/docker/build-push-action/compare/v5...v6)

Updates `softprops/action-gh-release` from 1 to 2
- [Release notes](https://github.com/softprops/action-gh-release/releases)
- [Changelog](https://github.com/softprops/action-gh-release/blob/master/CHANGELOG.md)
- [Commits](https://github.com/softprops/action-gh-release/compare/v1...v2)

Updates `actions/setup-python` from 4 to 6
- [Release notes](https://github.com/actions/setup-python/releases)
- [Commits](https://github.com/actions/setup-python/compare/v4...v6)

Updates `astral-sh/setup-uv` from 6 to 7
- [Release notes](https://github.com/astral-sh/setup-uv/releases)
- [Commits](https://github.com/astral-sh/setup-uv/compare/v6...v7)

---
updated-dependencies:
- dependency-name: actions/checkout
  dependency-version: '6'
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: github-actions
- dependency-name: docker/build-push-action
  dependency-version: '6'
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: github-actions
- dependency-name: softprops/action-gh-release
  dependency-version: '2'
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: github-actions
- dependency-name: actions/setup-python
  dependency-version: '6'
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: github-actions
- dependency-name: astral-sh/setup-uv
  dependency-version: '7'
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: github-actions
...

Signed-off-by: dependabot[bot] <support@github.com>
2026-01-02 06:15:39 +00:00
Alex
d10f2a0358 Merge pull request #863 from cclauss/patch-1
Keep GitHub Actions up to date with GitHub's Dependabot
2026-01-02 08:14:46 +02:00
Alex
c7008763d7 Merge pull request #861 from ikatkov/split-chapters-in-ui
Add video/audio chapter splitting with UI controls
2026-01-01 23:22:57 +02:00
Christian Clauss
351058e9f4 Keep GitHub Actions up to date with GitHub's Dependabot
* [Keeping your software supply chain secure with Dependabot](https://docs.github.com/en/code-security/dependabot)
* [Keeping your actions up to date with Dependabot](https://docs.github.com/en/code-security/dependabot/working-with-dependabot/keeping-your-actions-up-to-date-with-dependabot)
* [Configuration options for the `dependabot.yml` file - package-ecosystem](https://docs.github.com/en/code-security/dependabot/dependabot-version-updates/configuration-options-for-the-dependabot.yml-file#package-ecosystem)

To see all GitHub Actions dependencies, run the command:
% `git grep 'uses: ' .github/workflows/`
2025-12-31 13:05:26 +01:00
AlvinRamoutar
d799a4a8eb feature/repair-persistent-queues 2025-12-31 04:25:51 -05:00
ikatkov
df87a1aa2b Merge branch 'alexta69:master' into split-chapters-in-ui 2025-12-31 00:16:52 -08:00
Igor Katkov
02480afddf feat: Use OUTPUT_TEMPLATE_CHAPTER default setting 2025-12-31 00:13:55 -08:00
Igor Katkov
d51f2ce628 feat: Undo bogus formatting changes 2025-12-30 23:33:01 -08:00
Igor Katkov
962929d42d feat: Implement chapter splitting functionality with UI controls, yt-dlp integration, and chapter file tracking. 2025-12-30 22:07:49 -08:00
Alex
179452b4f4 Merge pull request #858 from ikatkov/master
Improves logging, helpful when debugging yt-dlp options
2025-12-30 22:40:16 +02:00
ikatkov
4fce74d1ed Merge pull request #1 from ikatkov/logging-fix
Logging fix
2025-12-30 10:22:31 -08:00
Igor Katkov
09a2e95515 fix: Root logger aligns with config.LOGLEVEL 2025-12-30 10:19:30 -08:00
Igor Katkov
d947876a71 fix: pass DEBUG log level to ytdl 2025-12-30 10:01:43 -08:00
Igor Katkov
6ba681a3cd fix: Moved code to respect loggin level in main.py 2025-12-30 08:45:54 -08:00
Alex
1f8fa7744e Merge pull request #857 from mercury233/patch-1
fix completed result tooltip
2025-12-27 12:17:40 +02:00
mercury233
092765535f fix completed result tooltip 2025-12-27 10:48:57 +08:00
11 changed files with 315 additions and 53 deletions

13
.github/dependabot.yml vendored Normal file
View File

@@ -0,0 +1,13 @@
# Keep GitHub Actions up to date with GitHub's Dependabot...
# https://docs.github.com/en/code-security/dependabot/working-with-dependabot/keeping-your-actions-up-to-date-with-dependabot
# https://docs.github.com/en/code-security/dependabot/working-with-dependabot/dependabot-options-reference#package-ecosystem
version: 2
updates:
- package-ecosystem: github-actions
directory: /
groups:
github-actions:
patterns:
- "*" # Group all Actions updates into a single larger pull request
schedule:
interval: weekly

View File

@@ -15,7 +15,7 @@ jobs:
run: echo "::set-output name=date::$(date +'%Y.%m.%d')" run: echo "::set-output name=date::$(date +'%Y.%m.%d')"
- -
name: Checkout name: Checkout
uses: actions/checkout@v4 uses: actions/checkout@v6
- -
name: Set up QEMU name: Set up QEMU
uses: docker/setup-qemu-action@v3 uses: docker/setup-qemu-action@v3
@@ -37,7 +37,7 @@ jobs:
password: ${{ secrets.GITHUB_TOKEN }} password: ${{ secrets.GITHUB_TOKEN }}
- -
name: Build and push name: Build and push
uses: docker/build-push-action@v5 uses: docker/build-push-action@v6
with: with:
context: . context: .
platforms: linux/amd64,linux/arm64 platforms: linux/amd64,linux/arm64
@@ -74,7 +74,7 @@ jobs:
id: date id: date
run: echo "date=$(date +'%Y.%m.%d')" >> $GITHUB_OUTPUT run: echo "date=$(date +'%Y.%m.%d')" >> $GITHUB_OUTPUT
- name: Checkout - name: Checkout
uses: actions/checkout@v4 uses: actions/checkout@v6
with: with:
fetch-depth: 0 fetch-depth: 0
- name: Get commits since last release - name: Get commits since last release
@@ -167,7 +167,7 @@ jobs:
git push origin ":refs/tags/$TAG_NAME" || true git push origin ":refs/tags/$TAG_NAME" || true
fi fi
- name: Create GitHub Release - name: Create GitHub Release
uses: softprops/action-gh-release@v1 uses: softprops/action-gh-release@v2
with: with:
tag_name: ${{ steps.date.outputs.date }} tag_name: ${{ steps.date.outputs.date }}
name: Release ${{ steps.date.outputs.date }} name: Release ${{ steps.date.outputs.date }}

View File

@@ -10,17 +10,17 @@ jobs:
steps: steps:
- -
name: Checkout name: Checkout
uses: actions/checkout@v4 uses: actions/checkout@v6
with: with:
token: ${{ secrets.AUTOUPDATE_PAT }} token: ${{ secrets.AUTOUPDATE_PAT }}
- -
name: Set up Python name: Set up Python
uses: actions/setup-python@v4 uses: actions/setup-python@v6
with: with:
python-version: '3.13' python-version: '3.13'
- -
name: Install uv name: Install uv
uses: astral-sh/setup-uv@v6 uses: astral-sh/setup-uv@v7
- -
name: Update yt-dlp name: Update yt-dlp
run: | run: |

View File

@@ -16,7 +16,7 @@ COPY pyproject.toml uv.lock docker-entrypoint.sh ./
# Install dependencies # Install dependencies
RUN sed -i 's/\r$//g' docker-entrypoint.sh && \ RUN sed -i 's/\r$//g' docker-entrypoint.sh && \
chmod +x docker-entrypoint.sh && \ chmod +x docker-entrypoint.sh && \
apk add --update ffmpeg aria2 coreutils shadow su-exec curl tini deno && \ apk add --update ffmpeg aria2 coreutils shadow su-exec curl tini deno gdbm-tools sqlite file && \
apk add --update --virtual .build-deps gcc g++ musl-dev uv && \ apk add --update --virtual .build-deps gcc g++ musl-dev uv && \
UV_PROJECT_ENVIRONMENT=/usr/local uv sync --frozen --no-dev --compile-bytecode && \ UV_PROJECT_ENVIRONMENT=/usr/local uv sync --frozen --no-dev --compile-bytecode && \
apk del .build-deps && \ apk del .build-deps && \

View File

@@ -270,8 +270,9 @@ MeTube development relies on code contributions by the community. The program as
Make sure you have Node.js 22+ and Python 3.13 installed. Make sure you have Node.js 22+ and Python 3.13 installed.
```bash ```bash
cd metube/ui
# install Angular and build the UI # install Angular and build the UI
cd ui
curl -fsSL https://get.pnpm.io/install.sh | sh -
pnpm install pnpm install
pnpm run build pnpm run build
# install python dependencies # install python dependencies

View File

@@ -21,6 +21,26 @@ from yt_dlp.version import __version__ as yt_dlp_version
log = logging.getLogger('main') log = logging.getLogger('main')
def parseLogLevel(logLevel):
match logLevel:
case 'DEBUG':
return logging.DEBUG
case 'INFO':
return logging.INFO
case 'WARNING':
return logging.WARNING
case 'ERROR':
return logging.ERROR
case 'CRITICAL':
return logging.CRITICAL
case _:
return None
# Configure logging before Config() uses it so early messages are not dropped.
# Only configure if no handlers are set (avoid clobbering hosting app settings).
if not logging.getLogger().hasHandlers():
logging.basicConfig(level=parseLogLevel(os.environ.get('LOGLEVEL', 'INFO')) or logging.INFO)
class Config: class Config:
_DEFAULTS = { _DEFAULTS = {
'DOWNLOAD_DIR': '.', 'DOWNLOAD_DIR': '.',
@@ -36,7 +56,7 @@ class Config:
'PUBLIC_HOST_URL': 'download/', 'PUBLIC_HOST_URL': 'download/',
'PUBLIC_HOST_AUDIO_URL': 'audio_download/', 'PUBLIC_HOST_AUDIO_URL': 'audio_download/',
'OUTPUT_TEMPLATE': '%(title)s.%(ext)s', 'OUTPUT_TEMPLATE': '%(title)s.%(ext)s',
'OUTPUT_TEMPLATE_CHAPTER': '%(title)s - %(section_number)s %(section_title)s.%(ext)s', 'OUTPUT_TEMPLATE_CHAPTER': '%(title)s - %(section_number)02d - %(section_title)s.%(ext)s',
'OUTPUT_TEMPLATE_PLAYLIST': '%(playlist_title)s/%(title)s.%(ext)s', 'OUTPUT_TEMPLATE_PLAYLIST': '%(playlist_title)s/%(title)s.%(ext)s',
'DEFAULT_OPTION_PLAYLIST_STRICT_MODE' : 'false', 'DEFAULT_OPTION_PLAYLIST_STRICT_MODE' : 'false',
'DEFAULT_OPTION_PLAYLIST_ITEM_LIMIT' : '0', 'DEFAULT_OPTION_PLAYLIST_ITEM_LIMIT' : '0',
@@ -112,6 +132,10 @@ class Config:
return (True, '') return (True, '')
config = Config() config = Config()
# Align root logger level with Config (keeps a single source of truth).
# This re-applies the log level after Config loads, in case LOGLEVEL was
# overridden by config file settings or differs from the environment variable.
logging.getLogger().setLevel(parseLogLevel(str(config.LOGLEVEL)) or logging.INFO)
class ObjectSerializer(json.JSONEncoder): class ObjectSerializer(json.JSONEncoder):
def default(self, obj): def default(self, obj):
@@ -223,6 +247,8 @@ async def add(request):
playlist_strict_mode = post.get('playlist_strict_mode') playlist_strict_mode = post.get('playlist_strict_mode')
playlist_item_limit = post.get('playlist_item_limit') playlist_item_limit = post.get('playlist_item_limit')
auto_start = post.get('auto_start') auto_start = post.get('auto_start')
split_by_chapters = post.get('split_by_chapters')
chapter_template = post.get('chapter_template')
if custom_name_prefix is None: if custom_name_prefix is None:
custom_name_prefix = '' custom_name_prefix = ''
@@ -232,10 +258,14 @@ async def add(request):
playlist_strict_mode = config.DEFAULT_OPTION_PLAYLIST_STRICT_MODE playlist_strict_mode = config.DEFAULT_OPTION_PLAYLIST_STRICT_MODE
if playlist_item_limit is None: if playlist_item_limit is None:
playlist_item_limit = config.DEFAULT_OPTION_PLAYLIST_ITEM_LIMIT playlist_item_limit = config.DEFAULT_OPTION_PLAYLIST_ITEM_LIMIT
if split_by_chapters is None:
split_by_chapters = False
if chapter_template is None:
chapter_template = config.OUTPUT_TEMPLATE_CHAPTER
playlist_item_limit = int(playlist_item_limit) playlist_item_limit = int(playlist_item_limit)
status = await dqueue.add(url, quality, format, folder, custom_name_prefix, playlist_strict_mode, playlist_item_limit, auto_start) status = await dqueue.add(url, quality, format, folder, custom_name_prefix, playlist_strict_mode, playlist_item_limit, auto_start, split_by_chapters, chapter_template)
return web.Response(text=serializer.encode(status)) return web.Response(text=serializer.encode(status))
@routes.post(config.URL_PREFIX + 'delete') @routes.post(config.URL_PREFIX + 'delete')
@@ -386,21 +416,6 @@ def supports_reuse_port():
except (AttributeError, OSError): except (AttributeError, OSError):
return False return False
def parseLogLevel(logLevel):
match logLevel:
case 'DEBUG':
return logging.DEBUG
case 'INFO':
return logging.INFO
case 'WARNING':
return logging.WARNING
case 'ERROR':
return logging.ERROR
case 'CRITICAL':
return logging.CRITICAL
case _:
return None
def isAccessLogEnabled(): def isAccessLogEnabled():
if config.ENABLE_ACCESSLOG: if config.ENABLE_ACCESSLOG:
return access_logger return access_logger
@@ -408,7 +423,7 @@ def isAccessLogEnabled():
return None return None
if __name__ == '__main__': if __name__ == '__main__':
logging.basicConfig(level=parseLogLevel(config.LOGLEVEL)) logging.getLogger().setLevel(parseLogLevel(config.LOGLEVEL) or logging.INFO)
log.info(f"Listening on {config.HOST}:{config.PORT}") log.info(f"Listening on {config.HOST}:{config.PORT}")
if config.HTTPS: if config.HTTPS:

View File

@@ -1,4 +1,5 @@
import os import os
import shutil
import yt_dlp import yt_dlp
from collections import OrderedDict from collections import OrderedDict
import shelve import shelve
@@ -8,6 +9,8 @@ import multiprocessing
import logging import logging
import re import re
import types import types
import dbm
import subprocess
import yt_dlp.networking.impersonate import yt_dlp.networking.impersonate
from dl_formats import get_format, get_opts, AUDIO_FORMATS from dl_formats import get_format, get_opts, AUDIO_FORMATS
@@ -43,7 +46,7 @@ class DownloadQueueNotifier:
raise NotImplementedError raise NotImplementedError
class DownloadInfo: class DownloadInfo:
def __init__(self, id, title, url, quality, format, folder, custom_name_prefix, error, entry, playlist_item_limit): def __init__(self, id, title, url, quality, format, folder, custom_name_prefix, error, entry, playlist_item_limit, split_by_chapters, chapter_template):
self.id = id if len(custom_name_prefix) == 0 else f'{custom_name_prefix}.{id}' self.id = id if len(custom_name_prefix) == 0 else f'{custom_name_prefix}.{id}'
self.title = title if len(custom_name_prefix) == 0 else f'{custom_name_prefix}.{title}' self.title = title if len(custom_name_prefix) == 0 else f'{custom_name_prefix}.{title}'
self.url = url self.url = url
@@ -59,6 +62,8 @@ class DownloadInfo:
# Convert generators to lists to make entry pickleable # Convert generators to lists to make entry pickleable
self.entry = _convert_generators_to_lists(entry) if entry is not None else None self.entry = _convert_generators_to_lists(entry) if entry is not None else None
self.playlist_item_limit = playlist_item_limit self.playlist_item_limit = playlist_item_limit
self.split_by_chapters = split_by_chapters
self.chapter_template = chapter_template
class Download: class Download:
manager = None manager = None
@@ -83,6 +88,7 @@ class Download:
def _download(self): def _download(self):
log.info(f"Starting download for: {self.info.title} ({self.info.url})") log.info(f"Starting download for: {self.info.title} ({self.info.url})")
try: try:
debug_logging = logging.getLogger().isEnabledFor(logging.DEBUG)
def put_status(st): def put_status(st):
self.status_queue.put({k: v for k, v in st.items() if k in ( self.status_queue.put({k: v for k, v in st.items() if k in (
'tmpfilename', 'tmpfilename',
@@ -104,8 +110,20 @@ class Download:
filename = d['info_dict']['filepath'] filename = d['info_dict']['filepath']
self.status_queue.put({'status': 'finished', 'filename': filename}) self.status_queue.put({'status': 'finished', 'filename': filename})
ret = yt_dlp.YoutubeDL(params={ # Capture all chapter files when SplitChapters finishes
'quiet': True, elif d.get('postprocessor') == 'SplitChapters' and d.get('status') == 'finished':
chapters = d.get('info_dict', {}).get('chapters', [])
if chapters:
for chapter in chapters:
if isinstance(chapter, dict) and 'filepath' in chapter:
log.info(f"Captured chapter file: {chapter['filepath']}")
self.status_queue.put({'chapter_file': chapter['filepath']})
else:
log.warning("SplitChapters finished but no chapter files found in info_dict")
ytdl_params = {
'quiet': not debug_logging,
'verbose': debug_logging,
'no_color': True, 'no_color': True,
'paths': {"home": self.download_dir, "temp": self.temp_dir}, 'paths': {"home": self.download_dir, "temp": self.temp_dir},
'outtmpl': { "default": self.output_template, "chapter": self.output_template_chapter }, 'outtmpl': { "default": self.output_template, "chapter": self.output_template_chapter },
@@ -115,7 +133,19 @@ class Download:
'progress_hooks': [put_status], 'progress_hooks': [put_status],
'postprocessor_hooks': [put_status_postprocessor], 'postprocessor_hooks': [put_status_postprocessor],
**self.ytdl_opts, **self.ytdl_opts,
}).download([self.info.url]) }
# Add chapter splitting options if enabled
if self.info.split_by_chapters:
ytdl_params['outtmpl']['chapter'] = self.info.chapter_template
if 'postprocessors' not in ytdl_params:
ytdl_params['postprocessors'] = []
ytdl_params['postprocessors'].append({
'key': 'FFmpegSplitChapters',
'force_keyframes': False
})
ret = yt_dlp.YoutubeDL(params=ytdl_params).download([self.info.url])
self.status_queue.put({'status': 'finished' if ret == 0 else 'error'}) self.status_queue.put({'status': 'finished' if ret == 0 else 'error'})
log.info(f"Finished download for: {self.info.title}") log.info(f"Finished download for: {self.info.title}")
except yt_dlp.utils.YoutubeDLError as exc: except yt_dlp.utils.YoutubeDLError as exc:
@@ -179,6 +209,22 @@ class Download:
self.info.size = os.path.getsize(fileName) if os.path.exists(fileName) else None self.info.size = os.path.getsize(fileName) if os.path.exists(fileName) else None
if self.info.format == 'thumbnail': if self.info.format == 'thumbnail':
self.info.filename = re.sub(r'\.webm$', '.jpg', self.info.filename) self.info.filename = re.sub(r'\.webm$', '.jpg', self.info.filename)
# Handle chapter files
log.debug(f"Update status for {self.info.title}: {status}")
if 'chapter_file' in status:
chapter_file = status.get('chapter_file')
if not hasattr(self.info, 'chapter_files'):
self.info.chapter_files = []
rel_path = os.path.relpath(chapter_file, self.download_dir)
file_size = os.path.getsize(chapter_file) if os.path.exists(chapter_file) else None
#Postprocessor hook called multiple times with chapters. Only insert if not already present.
existing = next((cf for cf in self.info.chapter_files if cf['filename'] == rel_path), None)
if not existing:
self.info.chapter_files.append({'filename': rel_path, 'size': file_size})
# Skip the rest of status processing for chapter files
continue
self.info.status = status['status'] self.info.status = status['status']
self.info.msg = status.get('msg') self.info.msg = status.get('msg')
if 'downloaded_bytes' in status: if 'downloaded_bytes' in status:
@@ -191,13 +237,16 @@ class Download:
await self.notifier.updated(self.info) await self.notifier.updated(self.info)
class PersistentQueue: class PersistentQueue:
def __init__(self, path): def __init__(self, name, path):
self.identifier = name
pdir = os.path.dirname(path) pdir = os.path.dirname(path)
if not os.path.isdir(pdir): if not os.path.isdir(pdir):
os.mkdir(pdir) os.mkdir(pdir)
with shelve.open(path, 'c'): with shelve.open(path, 'c'):
pass pass
self.path = path self.path = path
self.repair()
self.dict = OrderedDict() self.dict = OrderedDict()
def load(self): def load(self):
@@ -236,13 +285,84 @@ class PersistentQueue:
def empty(self): def empty(self):
return not bool(self.dict) return not bool(self.dict)
def repair(self):
# check DB format
type_check = subprocess.run(
["file", self.path],
capture_output=True,
text=True
)
db_type = type_check.stdout.lower()
# create backup (<queue>.old)
try:
shutil.copy2(self.path, f"{self.path}.old")
except Exception as e:
# if we cannot backup then its not safe to attempt a repair
# since it could be due to a filesystem error
log.debug(f"PersistentQueue:{self.identifier} backup failed, skipping repair")
return
if "gnu dbm" in db_type:
# perform gdbm repair
log_prefix = f"PersistentQueue:{self.identifier} repair (dbm/file)"
log.debug(f"{log_prefix} started")
try:
result = subprocess.run(
["gdbmtool", self.path],
input="recover verbose summary\n",
text=True,
capture_output=True,
timeout=60
)
log.debug(f"{log_prefix} {result.stdout}")
if result.stderr:
log.debug(f"{log_prefix} failed: {result.stderr}")
except FileNotFoundError:
log.debug(f"{log_prefix} failed: 'gdbmtool' was not found")
# perform null key cleanup
log_prefix = f"PersistentQueue:{self.identifier} repair (null keys)"
log.debug(f"{log_prefix} started")
deleted = 0
try:
with dbm.open(self.path, "w") as db:
for key in list(db.keys()):
if len(key) > 0 and all(b == 0x00 for b in key):
log.debug(f"{log_prefix} deleting key of length {len(key)} (all NUL bytes)")
del db[key]
deleted += 1
log.debug(f"{log_prefix} done - deleted {deleted} key(s)")
except dbm.error:
log.debug(f"{log_prefix} failed: db type is dbm.gnu, but the module is not available (dbm.error; module support may be missing or the file may be corrupted)")
elif "sqlite" in db_type:
# perform sqlite3 recovery
log_prefix = f"PersistentQueue:{self.identifier} repair (sqlite3/file)"
log.debug(f"{log_prefix} started")
try:
result = subprocess.run(
f"sqlite3 {self.path} '.recover' | sqlite3 {self.path}.tmp",
capture_output=True,
text=True,
shell=True,
timeout=60
)
if result.stderr:
log.debug(f"{log_prefix} failed: {result.stderr}")
else:
shutil.move(f"{self.path}.tmp", self.path)
log.debug(f"{log_prefix}{result.stdout or " was successful, no output"}")
except FileNotFoundError:
log.debug(f"{log_prefix} failed: 'sqlite3' was not found")
class DownloadQueue: class DownloadQueue:
def __init__(self, config, notifier): def __init__(self, config, notifier):
self.config = config self.config = config
self.notifier = notifier self.notifier = notifier
self.queue = PersistentQueue(self.config.STATE_DIR + '/queue') self.queue = PersistentQueue("queue", self.config.STATE_DIR + '/queue')
self.done = PersistentQueue(self.config.STATE_DIR + '/completed') self.done = PersistentQueue("completed", self.config.STATE_DIR + '/completed')
self.pending = PersistentQueue(self.config.STATE_DIR + '/pending') self.pending = PersistentQueue("pending", self.config.STATE_DIR + '/pending')
self.active_downloads = set() self.active_downloads = set()
self.semaphore = None self.semaphore = None
# For sequential mode, use an asyncio lock to ensure one-at-a-time execution. # For sequential mode, use an asyncio lock to ensure one-at-a-time execution.
@@ -314,8 +434,10 @@ class DownloadQueue:
asyncio.create_task(self.notifier.completed(download.info)) asyncio.create_task(self.notifier.completed(download.info))
def __extract_info(self, url, playlist_strict_mode): def __extract_info(self, url, playlist_strict_mode):
debug_logging = logging.getLogger().isEnabledFor(logging.DEBUG)
return yt_dlp.YoutubeDL(params={ return yt_dlp.YoutubeDL(params={
'quiet': True, 'quiet': not debug_logging,
'verbose': debug_logging,
'no_color': True, 'no_color': True,
'extract_flat': True, 'extract_flat': True,
'ignore_no_formats_error': True, 'ignore_no_formats_error': True,
@@ -368,7 +490,7 @@ class DownloadQueue:
self.pending.put(download) self.pending.put(download)
await self.notifier.added(dl) await self.notifier.added(dl)
async def __add_entry(self, entry, quality, format, folder, custom_name_prefix, playlist_strict_mode, playlist_item_limit, auto_start, already): async def __add_entry(self, entry, quality, format, folder, custom_name_prefix, playlist_strict_mode, playlist_item_limit, auto_start, split_by_chapters, chapter_template, already):
if not entry: if not entry:
return {'status': 'error', 'msg': "Invalid/empty data was given."} return {'status': 'error', 'msg': "Invalid/empty data was given."}
@@ -384,7 +506,7 @@ class DownloadQueue:
if etype.startswith('url'): if etype.startswith('url'):
log.debug('Processing as an url') log.debug('Processing as an url')
return await self.add(entry['url'], quality, format, folder, custom_name_prefix, playlist_strict_mode, playlist_item_limit, auto_start, already) return await self.add(entry['url'], quality, format, folder, custom_name_prefix, playlist_strict_mode, playlist_item_limit, auto_start, split_by_chapters, chapter_template, already)
elif etype == 'playlist': elif etype == 'playlist':
log.debug('Processing as a playlist') log.debug('Processing as a playlist')
entries = entry['entries'] entries = entry['entries']
@@ -404,7 +526,7 @@ class DownloadQueue:
for property in ("id", "title", "uploader", "uploader_id"): for property in ("id", "title", "uploader", "uploader_id"):
if property in entry: if property in entry:
etr[f"playlist_{property}"] = entry[property] etr[f"playlist_{property}"] = entry[property]
results.append(await self.__add_entry(etr, quality, format, folder, custom_name_prefix, playlist_strict_mode, playlist_item_limit, auto_start, already)) results.append(await self.__add_entry(etr, quality, format, folder, custom_name_prefix, playlist_strict_mode, playlist_item_limit, auto_start, split_by_chapters, chapter_template, already))
if any(res['status'] == 'error' for res in results): if any(res['status'] == 'error' for res in results):
return {'status': 'error', 'msg': ', '.join(res['msg'] for res in results if res['status'] == 'error' and 'msg' in res)} return {'status': 'error', 'msg': ', '.join(res['msg'] for res in results if res['status'] == 'error' and 'msg' in res)}
return {'status': 'ok'} return {'status': 'ok'}
@@ -412,13 +534,13 @@ class DownloadQueue:
log.debug('Processing as a video') log.debug('Processing as a video')
key = entry.get('webpage_url') or entry['url'] key = entry.get('webpage_url') or entry['url']
if not self.queue.exists(key): if not self.queue.exists(key):
dl = DownloadInfo(entry['id'], entry.get('title') or entry['id'], key, quality, format, folder, custom_name_prefix, error, entry, playlist_item_limit) dl = DownloadInfo(entry['id'], entry.get('title') or entry['id'], key, quality, format, folder, custom_name_prefix, error, entry, playlist_item_limit, split_by_chapters, chapter_template)
await self.__add_download(dl, auto_start) await self.__add_download(dl, auto_start)
return {'status': 'ok'} return {'status': 'ok'}
return {'status': 'error', 'msg': f'Unsupported resource "{etype}"'} return {'status': 'error', 'msg': f'Unsupported resource "{etype}"'}
async def add(self, url, quality, format, folder, custom_name_prefix, playlist_strict_mode, playlist_item_limit, auto_start=True, already=None): async def add(self, url, quality, format, folder, custom_name_prefix, playlist_strict_mode, playlist_item_limit, auto_start=True, split_by_chapters=False, chapter_template=None, already=None):
log.info(f'adding {url}: {quality=} {format=} {already=} {folder=} {custom_name_prefix=} {playlist_strict_mode=} {playlist_item_limit=} {auto_start=}') log.info(f'adding {url}: {quality=} {format=} {already=} {folder=} {custom_name_prefix=} {playlist_strict_mode=} {playlist_item_limit=} {auto_start=} {split_by_chapters=} {chapter_template=}')
already = set() if already is None else already already = set() if already is None else already
if url in already: if url in already:
log.info('recursion detected, skipping') log.info('recursion detected, skipping')
@@ -429,7 +551,7 @@ class DownloadQueue:
entry = await asyncio.get_running_loop().run_in_executor(None, self.__extract_info, url, playlist_strict_mode) entry = await asyncio.get_running_loop().run_in_executor(None, self.__extract_info, url, playlist_strict_mode)
except yt_dlp.utils.YoutubeDLError as exc: except yt_dlp.utils.YoutubeDLError as exc:
return {'status': 'error', 'msg': str(exc)} return {'status': 'error', 'msg': str(exc)}
return await self.__add_entry(entry, quality, format, folder, custom_name_prefix, playlist_strict_mode, playlist_item_limit, auto_start, already) return await self.__add_entry(entry, quality, format, folder, custom_name_prefix, playlist_strict_mode, playlist_item_limit, auto_start, split_by_chapters, chapter_template, already)
async def start_pending(self, ids): async def start_pending(self, ids):
for id in ids: for id in ids:

View File

@@ -231,6 +231,29 @@
<label class="form-check-label" for="checkbox-strict-mode">Strict Playlist Mode</label> <label class="form-check-label" for="checkbox-strict-mode">Strict Playlist Mode</label>
</div> </div>
</div> </div>
<div class="col-12">
<div class="row g-2 align-items-center">
<div class="col-auto">
<div class="form-check form-switch">
<input class="form-check-input" type="checkbox" role="switch" id="checkbox-split-chapters"
name="splitByChapters" [(ngModel)]="splitByChapters" (change)="splitByChaptersChanged()"
[disabled]="addInProgress || downloads.loading"
ngbTooltip="Split video into separate files by chapters">
<label class="form-check-label" for="checkbox-split-chapters">Split by chapters</label>
</div>
</div>
@if (splitByChapters) {
<div class="col">
<div class="input-group">
<span class="input-group-text">Template</span>
<input type="text" class="form-control" name="chapterTemplate" [(ngModel)]="chapterTemplate"
(change)="chapterTemplateChanged()" [disabled]="addInProgress || downloads.loading"
ngbTooltip="Output template for chapter files">
</div>
</div>
}
</div>
</div>
</div> </div>
<!-- Advanced Actions --> <!-- Advanced Actions -->
@@ -395,7 +418,7 @@
<fa-icon [icon]="faTimesCircle" class="text-danger" /> <fa-icon [icon]="faTimesCircle" class="text-danger" />
} }
</div> </div>
<span ngbTooltip="{{download.value.msg}} | {{download.value.error}}">@if (!!download.value.filename) { <span ngbTooltip="{{buildResultItemTooltip(download.value)}}">@if (!!download.value.filename) {
<a href="{{buildDownloadLink(download.value)}}" target="_blank">{{ download.value.title }}</a> <a href="{{buildDownloadLink(download.value)}}" target="_blank">{{ download.value.title }}</a>
} @else { } @else {
{{download.value.title}} {{download.value.title}}
@@ -425,6 +448,31 @@
</div> </div>
</td> </td>
</tr> </tr>
@if (download.value.chapter_files && download.value.chapter_files.length > 0) {
@for (chapterFile of download.value.chapter_files; track chapterFile.filename) {
<tr [class.disabled]='download.value.deleting'>
<td></td>
<td>
<div style="padding-left: 2rem;">
<fa-icon [icon]="faCheckCircle" class="text-success me-2" />
<a href="{{buildChapterDownloadLink(download.value, chapterFile.filename)}}" target="_blank">{{
getChapterFileName(chapterFile.filename) }}</a>
</div>
</td>
<td>
@if (chapterFile.size) {
<span>{{ chapterFile.size | fileSize }}</span>
}
</td>
<td>
<div class="d-flex">
<a href="{{buildChapterDownloadLink(download.value, chapterFile.filename)}}" download
class="btn btn-link"><fa-icon [icon]="faDownload" /></a>
</div>
</td>
</tr>
}
}
} }
</tbody> </tbody>
</table> </table>

View File

@@ -48,6 +48,8 @@ export class App implements AfterViewInit, OnInit {
autoStart: boolean; autoStart: boolean;
playlistStrictMode!: boolean; playlistStrictMode!: boolean;
playlistItemLimit!: number; playlistItemLimit!: number;
splitByChapters: boolean;
chapterTemplate: string;
addInProgress = false; addInProgress = false;
themes: Theme[] = Themes; themes: Theme[] = Themes;
activeTheme: Theme | undefined; activeTheme: Theme | undefined;
@@ -103,6 +105,9 @@ export class App implements AfterViewInit, OnInit {
this.setQualities() this.setQualities()
this.quality = this.cookieService.get('metube_quality') || 'best'; this.quality = this.cookieService.get('metube_quality') || 'best';
this.autoStart = this.cookieService.get('metube_auto_start') !== 'false'; this.autoStart = this.cookieService.get('metube_auto_start') !== 'false';
this.splitByChapters = this.cookieService.get('metube_split_chapters') === 'true';
// Will be set from backend configuration, use empty string as placeholder
this.chapterTemplate = this.cookieService.get('metube_chapter_template') || '';
this.activeTheme = this.getPreferredTheme(this.cookieService); this.activeTheme = this.getPreferredTheme(this.cookieService);
@@ -221,6 +226,10 @@ export class App implements AfterViewInit, OnInit {
if (playlistItemLimit !== '0') { if (playlistItemLimit !== '0') {
this.playlistItemLimit = playlistItemLimit; this.playlistItemLimit = playlistItemLimit;
} }
// Set chapter template from backend config if not already set by cookie
if (!this.chapterTemplate) {
this.chapterTemplate = config['OUTPUT_TEMPLATE_CHAPTER'];
}
} }
}); });
} }
@@ -260,6 +269,18 @@ export class App implements AfterViewInit, OnInit {
this.cookieService.set('metube_auto_start', this.autoStart ? 'true' : 'false', { expires: 3650 }); this.cookieService.set('metube_auto_start', this.autoStart ? 'true' : 'false', { expires: 3650 });
} }
splitByChaptersChanged() {
this.cookieService.set('metube_split_chapters', this.splitByChapters ? 'true' : 'false', { expires: 3650 });
}
chapterTemplateChanged() {
// Restore default if template is cleared - get from configuration
if (!this.chapterTemplate || this.chapterTemplate.trim() === '') {
this.chapterTemplate = this.downloads.configuration['OUTPUT_TEMPLATE_CHAPTER'];
}
this.cookieService.set('metube_chapter_template', this.chapterTemplate, { expires: 3650 });
}
queueSelectionChanged(checked: number) { queueSelectionChanged(checked: number) {
this.queueDelSelected().nativeElement.disabled = checked == 0; this.queueDelSelected().nativeElement.disabled = checked == 0;
this.queueDownloadSelected().nativeElement.disabled = checked == 0; this.queueDownloadSelected().nativeElement.disabled = checked == 0;
@@ -280,7 +301,7 @@ export class App implements AfterViewInit, OnInit {
} }
} }
addDownload(url?: string, quality?: string, format?: string, folder?: string, customNamePrefix?: string, playlistStrictMode?: boolean, playlistItemLimit?: number, autoStart?: boolean) { addDownload(url?: string, quality?: string, format?: string, folder?: string, customNamePrefix?: string, playlistStrictMode?: boolean, playlistItemLimit?: number, autoStart?: boolean, splitByChapters?: boolean, chapterTemplate?: string) {
url = url ?? this.addUrl url = url ?? this.addUrl
quality = quality ?? this.quality quality = quality ?? this.quality
format = format ?? this.format format = format ?? this.format
@@ -289,10 +310,18 @@ export class App implements AfterViewInit, OnInit {
playlistStrictMode = playlistStrictMode ?? this.playlistStrictMode playlistStrictMode = playlistStrictMode ?? this.playlistStrictMode
playlistItemLimit = playlistItemLimit ?? this.playlistItemLimit playlistItemLimit = playlistItemLimit ?? this.playlistItemLimit
autoStart = autoStart ?? this.autoStart autoStart = autoStart ?? this.autoStart
splitByChapters = splitByChapters ?? this.splitByChapters
chapterTemplate = chapterTemplate ?? this.chapterTemplate
console.debug('Downloading: url='+url+' quality='+quality+' format='+format+' folder='+folder+' customNamePrefix='+customNamePrefix+' playlistStrictMode='+playlistStrictMode+' playlistItemLimit='+playlistItemLimit+' autoStart='+autoStart); // Validate chapter template if chapter splitting is enabled
if (splitByChapters && !chapterTemplate.includes('%(section_number)')) {
alert('Chapter template must include %(section_number)');
return;
}
console.debug('Downloading: url=' + url + ' quality=' + quality + ' format=' + format + ' folder=' + folder + ' customNamePrefix=' + customNamePrefix + ' playlistStrictMode=' + playlistStrictMode + ' playlistItemLimit=' + playlistItemLimit + ' autoStart=' + autoStart + ' splitByChapters=' + splitByChapters + ' chapterTemplate=' + chapterTemplate);
this.addInProgress = true; this.addInProgress = true;
this.downloads.add(url, quality, format, folder, customNamePrefix, playlistStrictMode, playlistItemLimit, autoStart).subscribe((status: Status) => { this.downloads.add(url, quality, format, folder, customNamePrefix, playlistStrictMode, playlistItemLimit, autoStart, splitByChapters, chapterTemplate).subscribe((status: Status) => {
if (status.status === 'error') { if (status.status === 'error') {
alert(`Error adding URL: ${status.msg}`); alert(`Error adding URL: ${status.msg}`);
} else { } else {
@@ -307,7 +336,7 @@ export class App implements AfterViewInit, OnInit {
} }
retryDownload(key: string, download: Download) { retryDownload(key: string, download: Download) {
this.addDownload(download.url, download.quality, download.format, download.folder, download.custom_name_prefix, download.playlist_strict_mode, download.playlist_item_limit, true); this.addDownload(download.url, download.quality, download.format, download.folder, download.custom_name_prefix, download.playlist_strict_mode, download.playlist_item_limit, true, download.split_by_chapters, download.chapter_template);
this.downloads.delById('done', [key]).subscribe(); this.downloads.delById('done', [key]).subscribe();
} }
@@ -367,6 +396,35 @@ export class App implements AfterViewInit, OnInit {
return baseDir + encodeURIComponent(download.filename); return baseDir + encodeURIComponent(download.filename);
} }
buildResultItemTooltip(download: Download) {
const parts = [];
if (download.msg) {
parts.push(download.msg);
}
if (download.error) {
parts.push(download.error);
}
return parts.join(' | ');
}
buildChapterDownloadLink(download: Download, chapterFilename: string) {
let baseDir = this.downloads.configuration["PUBLIC_HOST_URL"];
if (download.quality == 'audio' || chapterFilename.endsWith('.mp3')) {
baseDir = this.downloads.configuration["PUBLIC_HOST_AUDIO_URL"];
}
if (download.folder) {
baseDir += download.folder + '/';
}
return baseDir + encodeURIComponent(chapterFilename);
}
getChapterFileName(filepath: string) {
// Extract just the filename from the path
const parts = filepath.split('/');
return parts[parts.length - 1];
}
isNumber(event: KeyboardEvent) { isNumber(event: KeyboardEvent) {
const charCode = +event.code || event.keyCode; const charCode = +event.code || event.keyCode;
@@ -424,7 +482,7 @@ export class App implements AfterViewInit, OnInit {
this.batchImportStatus = `Importing URL ${index + 1} of ${urls.length}: ${url}`; this.batchImportStatus = `Importing URL ${index + 1} of ${urls.length}: ${url}`;
// Now pass the selected quality, format, folder, etc. to the add() method // Now pass the selected quality, format, folder, etc. to the add() method
this.downloads.add(url, this.quality, this.format, this.folder, this.customNamePrefix, this.downloads.add(url, this.quality, this.format, this.folder, this.customNamePrefix,
this.playlistStrictMode, this.playlistItemLimit, this.autoStart) this.playlistStrictMode, this.playlistItemLimit, this.autoStart, this.splitByChapters, this.chapterTemplate)
.subscribe({ .subscribe({
next: (status: Status) => { next: (status: Status) => {
if (status.status === 'error') { if (status.status === 'error') {

View File

@@ -9,6 +9,8 @@ export interface Download {
custom_name_prefix: string; custom_name_prefix: string;
playlist_strict_mode: boolean; playlist_strict_mode: boolean;
playlist_item_limit: number; playlist_item_limit: number;
split_by_chapters?: boolean;
chapter_template?: string;
status: string; status: string;
msg: string; msg: string;
percent: number; percent: number;
@@ -19,4 +21,5 @@ export interface Download {
size?: number; size?: number;
error?: string; error?: string;
deleting?: boolean; deleting?: boolean;
chapter_files?: Array<{ filename: string, size: number }>;
} }

View File

@@ -107,8 +107,8 @@ export class DownloadsService {
return of({status: 'error', msg: msg}) return of({status: 'error', msg: msg})
} }
public add(url: string, quality: string, format: string, folder: string, customNamePrefix: string, playlistStrictMode: boolean, playlistItemLimit: number, autoStart: boolean) { public add(url: string, quality: string, format: string, folder: string, customNamePrefix: string, playlistStrictMode: boolean, playlistItemLimit: number, autoStart: boolean, splitByChapters: boolean, chapterTemplate: string) {
return this.http.post<Status>('add', {url: url, quality: quality, format: format, folder: folder, custom_name_prefix: customNamePrefix, playlist_strict_mode: playlistStrictMode, playlist_item_limit: playlistItemLimit, auto_start: autoStart}).pipe( return this.http.post<Status>('add', { url: url, quality: quality, format: format, folder: folder, custom_name_prefix: customNamePrefix, playlist_strict_mode: playlistStrictMode, playlist_item_limit: playlistItemLimit, auto_start: autoStart, split_by_chapters: splitByChapters, chapter_template: chapterTemplate }).pipe(
catchError(this.handleHTTPError) catchError(this.handleHTTPError)
); );
} }
@@ -150,9 +150,11 @@ export class DownloadsService {
const defaultPlaylistStrictMode = false; const defaultPlaylistStrictMode = false;
const defaultPlaylistItemLimit = 0; const defaultPlaylistItemLimit = 0;
const defaultAutoStart = true; const defaultAutoStart = true;
const defaultSplitByChapters = false;
const defaultChapterTemplate = this.configuration['OUTPUT_TEMPLATE_CHAPTER'];
return new Promise((resolve, reject) => { return new Promise((resolve, reject) => {
this.add(url, defaultQuality, defaultFormat, defaultFolder, defaultCustomNamePrefix, defaultPlaylistStrictMode, defaultPlaylistItemLimit, defaultAutoStart) this.add(url, defaultQuality, defaultFormat, defaultFolder, defaultCustomNamePrefix, defaultPlaylistStrictMode, defaultPlaylistItemLimit, defaultAutoStart, defaultSplitByChapters, defaultChapterTemplate)
.subscribe({ .subscribe({
next: (response) => resolve(response), next: (response) => resolve(response),
error: (error) => reject(error) error: (error) => reject(error)