Videos

Download videos from various platforms.

See also

  • A collection of scripts I have written and/or adapted that I currently use on my systems as automated tasks 1

  • youtube-dl - ArchWiki 2

  • GitHub - yt-dlp/yt-dlp: A youtube-dl fork with additional features and fixes 3

Run as user

Instruction number

myuser

1

root

2-13

  1. install yt-dlp

    pip3 install --user yt-dlp

  2. install the dependencies

    apt-get install python3-yaml aria2

    Note

    If you use the provided options file as-is you need to install aria2

  3. install fpyutils. See reference

  4. add the user to the jobs group

    usermod -aG jobs myuser

  5. create the jobs directories. See reference

    mkdir -p /home/jobs/{scripts,services}/by-user/myuser
chmod 700 /home/jobs/{scripts,services}/by-user/myuser

  6. create the script

    /home/jobs/scripts/by-user/myuser/youtube_dl.py
     1#!/usr/bin/env python3
 2#
 3# youtube_dl.py
 4#
 5# Copyright (C) 2019-2022 Franco Masotti (franco \D\o\T masotti {-A-T-} tutanota \D\o\T com)
 6#
 7# This program is free software: you can redistribute it and/or modify
 8# it under the terms of the GNU General Public License as published by
 9# the Free Software Foundation, either version 3 of the License, or
10# (at your option) any later version.
11#
12# This program is distributed in the hope that it will be useful,
13# but WITHOUT ANY WARRANTY; without even the implied warranty of
14# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
15# GNU General Public License for more details.
16#
17# You should have received a copy of the GNU General Public License
18# along with this program.  If not, see <https://www.gnu.org/licenses/>.
19r"""Download videos, delete old ones and get a notification."""
20
21import datetime
22import pathlib
23import shlex
24import sys
25
26import fpyutils
27import yaml
28
29if __name__ == '__main__':
30    configuration_file = shlex.quote(sys.argv[1])
31    config = yaml.load(open(configuration_file, 'r'), Loader=yaml.SafeLoader)
32
33    pathlib.Path(shlex.quote(config['youtube_dl']['dst_dir'])).mkdir(
34        mode=0o700, parents=True, exist_ok=True)
35    pathlib.Path(shlex.quote(
36        config['youtube_dl']['archived_list_file'])).touch(mode=0o700,
37                                                           exist_ok=True)
38    original_files = sum(1 for line in open(
39        shlex.quote(config['youtube_dl']['archived_list_file'])))
40
41    # Build binary.
42    cmd = str()
43    if 'binary' in config['youtube_dl']:
44        for c in config['youtube_dl']['binary']:
45            cmd += shlex.quote(c) + ' '
46
47    # youtube-dl might not return 0 even if some videos are correctly downloaded.
48    command = 'pushd ' + shlex.quote(
49        config['youtube_dl']['dst_dir']
50    ) + ' ; ' + cmd + ' --verbose --config-location ' + shlex.quote(
51        config['youtube_dl']['options_file']) + ' --batch ' + shlex.quote(
52            config['youtube_dl']
53            ['url_list_file']) + ' --download-archive ' + shlex.quote(
54                config['youtube_dl']['archived_list_file']) + ' ; popd'
55    fpyutils.shell.execute_command_live_output(command)
56
57    # For this to work be sure to set the no-mtime option in the options file.
58    # Only the video files, infact, would retain the original modification time
59    # (not the modification time correpsponding to the download time).
60    # All the other files such as thumbnails and subtitles do not retain the
61    # original mtime. For this reason it is simpler not to consider the
62    # original mtime.
63    deleted_files = 0
64    if config['delete']['enabled']:
65        for f in sorted(
66                pathlib.Path(shlex.quote(
67                    config['youtube_dl']['dst_dir'])).glob('*/*')):
68            if f.is_file() and (datetime.date.today() -
69                                datetime.date.fromtimestamp(f.stat().st_mtime)
70                                ).days > config['delete']['days_to_keep']:
71                f.unlink()
72                deleted_files += 1
73
74    final_files = sum(1 for line in open(
75        shlex.quote(config['youtube_dl']['archived_list_file'])))
76
77    downloaded_files = final_files - original_files
78
79    message = 'DW: ' + str(downloaded_files) + ' ; RM: ' + str(deleted_files)
80    if config['notify']['gotify']['enabled']:
81        m = config['notify']['gotify']['message'] + '\n' + message
82        fpyutils.notify.send_gotify_message(
83            config['notify']['gotify']['url'],
84            config['notify']['gotify']['token'], m,
85            config['notify']['gotify']['title'],
86            config['notify']['gotify']['priority'])
87    if config['notify']['email']['enabled']:
88        fpyutils.notify.send_email(message,
89                                   config['notify']['email']['smtp_server'],
90                                   config['notify']['email']['port'],
91                                   config['notify']['email']['sender'],
92                                   config['notify']['email']['user'],
93                                   config['notify']['email']['password'],
94                                   config['notify']['email']['receiver'],
95                                   config['notify']['email']['subject'])

  7. create a configuration file

    /home/jobs/scripts/by-user/myuser/youtube_dl.some_subject.yaml
     1#
 2# youtube_dl.some_subject.yaml
 3#
 4# Copyright (C) 2019-2022 Franco Masotti (franco \D\o\T masotti {-A-T-} tutanota \D\o\T com)
 5#
 6# This program is free software: you can redistribute it and/or modify
 7# it under the terms of the GNU General Public License as published by
 8# the Free Software Foundation, either version 3 of the License, or
 9# (at your option) any later version.
10#
11# This program is distributed in the hope that it will be useful,
12# but WITHOUT ANY WARRANTY; without even the implied warranty of
13# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
14# GNU General Public License for more details.
15#
16# You should have received a copy of the GNU General Public License
17# along with this program.  If not, see <https://www.gnu.org/licenses/>.
18
19youtube_dl:
20    # A list of strings corresponding to binaries and options.
21    # You need to specify at least the youtube-dl binary.
22    binary:
23        # - torsocks
24        # - --isolate
25        - /home/myuser/.local/bin/yt-dlp
26
27    # The configuration file for youtube-dl.
28    options_file: '/home/jobs/scripts/by-user/myuser/youtube_dl.some_subject.options'
29
30    # The base directory where all the videos will lie.
31    dst_dir: '/home/myuser/videos/bot/some_subject'
32
33    # The base directory where all the videos will lie.
34    url_list_file: '/home/jobs/scripts/by-user/myuser/youtube_dl.some_subject.txt'
35
36    # The file containing the list of downloaded urls.
37    archived_list_file: '/home/myuser/videos/bot/some_subject/archived.txt'
38
39delete:
40    enabled: true
41    days_to_keep: 60
42
43notify:
44    email:
45        enabled: false
46        smtp_server: 'smtp.gmail.com'
47        port: 465
48        sender: 'myusername@gmail.com'
49        user: 'myusername'
50        password: 'my awesome password'
51        receiver: 'myusername@gmail.com'
52        subject: 'video bot: some_subject'
53    gotify:
54        enabled: false
55        url: '<gotify url>'
56        token: '<app token>'
57        title: 'video bot'
58        message: 'some_subject'
59        priority: 5

  8. create the options file which select all the options used by yt-dlp

    /home/jobs/scripts/by-user/myuser/youtube_dl.some_subject.options
     1#
 2# youtube_dl.some_subject.options
 3#
 4# Copyright (C) 2019 Franco Masotti (franco \D\o\T masotti {-A-T-} tutanota \D\o\T com)
 5#
 6# This program is free software: you can redistribute it and/or modify
 7# it under the terms of the GNU General Public License as published by
 8# the Free Software Foundation, either version 3 of the License, or
 9# (at your option) any later version.
10#
11# This program is distributed in the hope that it will be useful,
12# but WITHOUT ANY WARRANTY; without even the implied warranty of
13# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
14# GNU General Public License for more details.
15#
16# You should have received a copy of the GNU General Public License
17# along with this program.  If not, see <https://www.gnu.org/licenses/>.
18
19# --get-id
20
21# Filter by video release date or by playlist id number,
22# up to playlist-end videos back.
23#--dateafter now-1days
24--playlist-start 1
25--playlist-end 5
26
27--add-metadata
28
29# Download data in parallel. Change or remove the max-overall-download-limit option.
30--external-downloader aria2c
31--external-downloader-args '-c -j 3 -x 3 -s 3 -k 1M --max-overall-download-limit=512K'
32
33# Avoid geographical restrictions.
34--geo-bypass
35
36--force-ipv4
37--no-color
38--ignore-errors
39--continue
40--no-cache-dir
41
42--socket-timeout 300
43
44# Prefer 720p videos and fallback to 480p.
45--format 247+bestaudio/136+bestaudio/244+bestaudio/135+bestaudio
46
47# Subtitles.
48--write-sub
49--write-auto-sub
50
51# Output file should be a mkv container.
52--merge-output-format mkv
53
54# Get the video thumbnail.
55--write-thumbnail
56
57# File path.
58# Subdirectories by yyyy-mm.
59--output "%(uploader)s/%(upload_date>%Y)s/%(upload_date>%m)s/%(upload_date)s_%(playlist_index)s_%(title)s_%(id)s"
60
61--write-description
62--write-info-json
63#--write-annotations
64--convert-subs vtt
65
66# Save the list of the previuosly downloaded videos.
67--download-archive archive.txt
68
69# Slugify file names.
70--restrict-filenames
71
72--no-overwrite
73--no-call-home
74--prefer-free-formats
75--fixup detect_or_warn
76--prefer-ffmpeg
77
78# Transform to hvec, 8-bit color.
79#--postprocessor-args "-c:v libx265 -preset veryfast -x265-params crf=28 -pix_fmt yuv420p -c:a copy"
80
81# Sleep a number of seconds between one download and the other.
82--sleep-interval 10
83
84# Use the following url list.
85--batch channels.txt
86
87# Very important if you enable automatic file deletion.
88--no-mtime
89
90# Proxy. Enable and edit this option as needed.
91# --proxy 127.0.0.1:8123

  9. create a text file containing URLs of channels or playlists, one for each line

    /home/jobs/scripts/by-user/myuser/youtube_dl.some_subject.txt
    1http://a-newline
2http://separated-list
3http://of-urls.

  10. create the Systemd service unit file

    /home/jobs/services/by-user/myuser/youtube-dl.some_subject.service
     1[Unit]
 2Description=Download youtube-dl.some_subject videos
 3Requires=network-online.target
 4After=network-online.target
 5
 6[Service]
 7Type=simple
 8ExecStart=/home/jobs/scripts/by-user/myuser/youtube_dl.py /home/jobs/scripts/by-user/myuser/youtube_dl.some_subject.yaml
 9User=myuser
10Group=myuser
11
12[Install]
13WantedBy=multi-user.target

  11. create the Systemd service timer unit file

    /home/jobs/services/by-user/myuser/youtube-dl.some_subject.timer
    1[Unit]
2Description=Once a day download some_subject videos
3
4[Timer]
5OnCalendar=*-*-* 3:30:00
6Persistent=true
7
8[Install]
9WantedBy=timers.target

  12. fix the permissions

    chown -R myuser:myuser /home/jobs/{scripts,services}/by-user/myuser
chmod 700 -R /home/jobs/{scripts,services}/by-user/myuser

  13. run the deploy script

Footnotes

1

https://software.franco.net.eu.org/frnmst/automated-tasks GNU GPLv3+, copyright (c) 2019-2022, Franco Masotti

2

https://wiki.archlinux.org/index.php/Youtube-dl GNU Free Documentation License 1.3 or later, Copyright (c) ArchWiki contributors

3

https://github.com/yt-dlp/yt-dlp Unilicense