Videos
Download videos from various platforms.
See also
Run as user |
Instruction number |
|
1 |
|
2-13 |
install yt-dlp
pip3 install --user yt-dlp
install the dependencies
apt-get install python3-yaml aria2
Note
If you use the provided options file as-is you need to install aria2
install fpyutils. See reference
add the user to the jobs group
usermod -aG jobs myuser
create the jobs directories. See reference
mkdir -p /home/jobs/{scripts,services}/by-user/myuser chmod 700 /home/jobs/{scripts,services}/by-user/myuser
create the
script
/home/jobs/scripts/by-user/myuser/youtube_dl.py1#!/usr/bin/env python3 2# 3# youtube_dl.py 4# 5# Copyright (C) 2019-2022 Franco Masotti (franco \D\o\T masotti {-A-T-} tutanota \D\o\T com) 6# 7# This program is free software: you can redistribute it and/or modify 8# it under the terms of the GNU General Public License as published by 9# the Free Software Foundation, either version 3 of the License, or 10# (at your option) any later version. 11# 12# This program is distributed in the hope that it will be useful, 13# but WITHOUT ANY WARRANTY; without even the implied warranty of 14# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the 15# GNU General Public License for more details. 16# 17# You should have received a copy of the GNU General Public License 18# along with this program. If not, see <https://www.gnu.org/licenses/>. 19r"""Download videos, delete old ones and get a notification.""" 20 21import datetime 22import pathlib 23import shlex 24import sys 25 26import fpyutils 27import yaml 28 29if __name__ == '__main__': 30 configuration_file = shlex.quote(sys.argv[1]) 31 config = yaml.load(open(configuration_file, 'r'), Loader=yaml.SafeLoader) 32 33 pathlib.Path(shlex.quote(config['youtube_dl']['dst_dir'])).mkdir( 34 mode=0o700, parents=True, exist_ok=True) 35 pathlib.Path(shlex.quote( 36 config['youtube_dl']['archived_list_file'])).touch(mode=0o700, 37 exist_ok=True) 38 original_files = sum(1 for line in open( 39 shlex.quote(config['youtube_dl']['archived_list_file']))) 40 41 # Build binary. 42 cmd = str() 43 if 'binary' in config['youtube_dl']: 44 for c in config['youtube_dl']['binary']: 45 cmd += shlex.quote(c) + ' ' 46 47 # youtube-dl might not return 0 even if some videos are correctly downloaded. 48 command = 'pushd ' + shlex.quote( 49 config['youtube_dl']['dst_dir'] 50 ) + ' ; ' + cmd + ' --verbose --config-location ' + shlex.quote( 51 config['youtube_dl']['options_file']) + ' --batch ' + shlex.quote( 52 config['youtube_dl'] 53 ['url_list_file']) + ' --download-archive ' + shlex.quote( 54 config['youtube_dl']['archived_list_file']) + ' ; popd' 55 fpyutils.shell.execute_command_live_output(command) 56 57 # For this to work be sure to set the no-mtime option in the options file. 58 # Only the video files, infact, would retain the original modification time 59 # (not the modification time correpsponding to the download time). 60 # All the other files such as thumbnails and subtitles do not retain the 61 # original mtime. For this reason it is simpler not to consider the 62 # original mtime. 63 deleted_files = 0 64 if config['delete']['enabled']: 65 for f in sorted( 66 pathlib.Path(shlex.quote( 67 config['youtube_dl']['dst_dir'])).glob('*/*')): 68 if f.is_file() and (datetime.date.today() - 69 datetime.date.fromtimestamp(f.stat().st_mtime) 70 ).days > config['delete']['days_to_keep']: 71 f.unlink() 72 deleted_files += 1 73 74 final_files = sum(1 for line in open( 75 shlex.quote(config['youtube_dl']['archived_list_file']))) 76 77 downloaded_files = final_files - original_files 78 79 message = 'DW: ' + str(downloaded_files) + ' ; RM: ' + str(deleted_files) 80 if config['notify']['gotify']['enabled']: 81 m = config['notify']['gotify']['message'] + '\n' + message 82 fpyutils.notify.send_gotify_message( 83 config['notify']['gotify']['url'], 84 config['notify']['gotify']['token'], m, 85 config['notify']['gotify']['title'], 86 config['notify']['gotify']['priority']) 87 if config['notify']['email']['enabled']: 88 fpyutils.notify.send_email(message, 89 config['notify']['email']['smtp_server'], 90 config['notify']['email']['port'], 91 config['notify']['email']['sender'], 92 config['notify']['email']['user'], 93 config['notify']['email']['password'], 94 config['notify']['email']['receiver'], 95 config['notify']['email']['subject'])
create a
configuration file
/home/jobs/scripts/by-user/myuser/youtube_dl.some_subject.yaml1# 2# youtube_dl.some_subject.yaml 3# 4# Copyright (C) 2019-2022 Franco Masotti (franco \D\o\T masotti {-A-T-} tutanota \D\o\T com) 5# 6# This program is free software: you can redistribute it and/or modify 7# it under the terms of the GNU General Public License as published by 8# the Free Software Foundation, either version 3 of the License, or 9# (at your option) any later version. 10# 11# This program is distributed in the hope that it will be useful, 12# but WITHOUT ANY WARRANTY; without even the implied warranty of 13# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the 14# GNU General Public License for more details. 15# 16# You should have received a copy of the GNU General Public License 17# along with this program. If not, see <https://www.gnu.org/licenses/>. 18 19youtube_dl: 20 # A list of strings corresponding to binaries and options. 21 # You need to specify at least the youtube-dl binary. 22 binary: 23 # - torsocks 24 # - --isolate 25 - /home/myuser/.local/bin/yt-dlp 26 27 # The configuration file for youtube-dl. 28 options_file: '/home/jobs/scripts/by-user/myuser/youtube_dl.some_subject.options' 29 30 # The base directory where all the videos will lie. 31 dst_dir: '/home/myuser/videos/bot/some_subject' 32 33 # The base directory where all the videos will lie. 34 url_list_file: '/home/jobs/scripts/by-user/myuser/youtube_dl.some_subject.txt' 35 36 # The file containing the list of downloaded urls. 37 archived_list_file: '/home/myuser/videos/bot/some_subject/archived.txt' 38 39delete: 40 enabled: true 41 days_to_keep: 60 42 43notify: 44 email: 45 enabled: false 46 smtp_server: 'smtp.gmail.com' 47 port: 465 48 sender: 'myusername@gmail.com' 49 user: 'myusername' 50 password: 'my awesome password' 51 receiver: 'myusername@gmail.com' 52 subject: 'video bot: some_subject' 53 gotify: 54 enabled: false 55 url: '<gotify url>' 56 token: '<app token>' 57 title: 'video bot' 58 message: 'some_subject' 59 priority: 5
create the
options file
which select all the options used by yt-dlp/home/jobs/scripts/by-user/myuser/youtube_dl.some_subject.options1# 2# youtube_dl.some_subject.options 3# 4# Copyright (C) 2019 Franco Masotti (franco \D\o\T masotti {-A-T-} tutanota \D\o\T com) 5# 6# This program is free software: you can redistribute it and/or modify 7# it under the terms of the GNU General Public License as published by 8# the Free Software Foundation, either version 3 of the License, or 9# (at your option) any later version. 10# 11# This program is distributed in the hope that it will be useful, 12# but WITHOUT ANY WARRANTY; without even the implied warranty of 13# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the 14# GNU General Public License for more details. 15# 16# You should have received a copy of the GNU General Public License 17# along with this program. If not, see <https://www.gnu.org/licenses/>. 18 19# --get-id 20 21# Filter by video release date or by playlist id number, 22# up to playlist-end videos back. 23#--dateafter now-1days 24--playlist-start 1 25--playlist-end 5 26 27--add-metadata 28 29# Download data in parallel. Change or remove the max-overall-download-limit option. 30--external-downloader aria2c 31--external-downloader-args '-c -j 3 -x 3 -s 3 -k 1M --max-overall-download-limit=512K' 32 33# Avoid geographical restrictions. 34--geo-bypass 35 36--force-ipv4 37--no-color 38--ignore-errors 39--continue 40--no-cache-dir 41 42--socket-timeout 300 43 44# Prefer 720p videos and fallback to 480p. 45--format 247+bestaudio/136+bestaudio/244+bestaudio/135+bestaudio 46 47# Subtitles. 48--write-sub 49--write-auto-sub 50 51# Output file should be a mkv container. 52--merge-output-format mkv 53 54# Get the video thumbnail. 55--write-thumbnail 56 57# File path. 58# Subdirectories by yyyy-mm. 59--output "%(uploader)s/%(upload_date>%Y)s/%(upload_date>%m)s/%(upload_date)s_%(playlist_index)s_%(title)s_%(id)s" 60 61--write-description 62--write-info-json 63#--write-annotations 64--convert-subs vtt 65 66# Save the list of the previuosly downloaded videos. 67--download-archive archive.txt 68 69# Slugify file names. 70--restrict-filenames 71 72--no-overwrite 73--no-call-home 74--prefer-free-formats 75--fixup detect_or_warn 76--prefer-ffmpeg 77 78# Transform to hvec, 8-bit color. 79#--postprocessor-args "-c:v libx265 -preset veryfast -x265-params crf=28 -pix_fmt yuv420p -c:a copy" 80 81# Sleep a number of seconds between one download and the other. 82--sleep-interval 10 83 84# Use the following url list. 85--batch channels.txt 86 87# Very important if you enable automatic file deletion. 88--no-mtime 89 90# Proxy. Enable and edit this option as needed. 91# --proxy 127.0.0.1:8123
create a
text file
containing URLs of channels or playlists, one for each line/home/jobs/scripts/by-user/myuser/youtube_dl.some_subject.txt1http://a-newline 2http://separated-list 3http://of-urls.
create the
Systemd service unit file
/home/jobs/services/by-user/myuser/youtube-dl.some_subject.service1[Unit] 2Description=Download youtube-dl.some_subject videos 3Requires=network-online.target 4After=network-online.target 5 6[Service] 7Type=simple 8ExecStart=/home/jobs/scripts/by-user/myuser/youtube_dl.py /home/jobs/scripts/by-user/myuser/youtube_dl.some_subject.yaml 9User=myuser 10Group=myuser 11 12[Install] 13WantedBy=multi-user.target
create the
Systemd service timer unit file
/home/jobs/services/by-user/myuser/youtube-dl.some_subject.timer1[Unit] 2Description=Once a day download some_subject videos 3 4[Timer] 5OnCalendar=*-*-* 3:30:00 6Persistent=true 7 8[Install] 9WantedBy=timers.target
fix the permissions
chown -R myuser:myuser /home/jobs/{scripts,services}/by-user/myuser chmod 700 -R /home/jobs/{scripts,services}/by-user/myuser
run the deploy script
Footnotes
- 1
https://software.franco.net.eu.org/frnmst/automated-tasks GNU GPLv3+, copyright (c) 2019-2022, Franco Masotti
- 2
https://wiki.archlinux.org/index.php/Youtube-dl GNU Free Documentation License 1.3 or later, Copyright (c) ArchWiki contributors
- 3
https://github.com/yt-dlp/yt-dlp Unilicense