Videos

Download videos from various platforms.

See also

  • A collection of scripts I have written and/or adapted that I currently use on my systems as automated tasks 1

  • youtube-dl - ArchWiki 2

  • GitHub - yt-dlp/yt-dlp: A youtube-dl fork with additional features and fixes 3

  1. install the dependencies

    apt-get install python3-yaml aria2 yt-dlp/bullseye-backports
    

    Note

    If you use the provided options file as-is you need to install aria2

    Note

    You need to enable the backports before installing yt-dlp

  2. install fpyutils. See reference

  3. add the user to the jobs group

    usermod -aG jobs myuser
    
  4. create the jobs directories. See reference

    mkdir -p /home/jobs/{scripts,services}/by-user/myuser
    chmod 700 /home/jobs/{scripts,services}/by-user/myuser
    
  5. create the script

    /home/jobs/scripts/by-user/myuser/youtube_dl.py
     1#!/usr/bin/env python3
     2#
     3# youtube_dl.py
     4#
     5# Copyright (C) 2019-2022 Franco Masotti (franco \D\o\T masotti {-A-T-} tutanota \D\o\T com)
     6#
     7# This program is free software: you can redistribute it and/or modify
     8# it under the terms of the GNU General Public License as published by
     9# the Free Software Foundation, either version 3 of the License, or
    10# (at your option) any later version.
    11#
    12# This program is distributed in the hope that it will be useful,
    13# but WITHOUT ANY WARRANTY; without even the implied warranty of
    14# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
    15# GNU General Public License for more details.
    16#
    17# You should have received a copy of the GNU General Public License
    18# along with this program.  If not, see <https://www.gnu.org/licenses/>.
    19r"""Download videos, delete old ones and get a notification."""
    20
    21import datetime
    22import pathlib
    23import shlex
    24import sys
    25
    26import fpyutils
    27import yaml
    28
    29if __name__ == '__main__':
    30    configuration_file = shlex.quote(sys.argv[1])
    31    config = yaml.load(open(configuration_file, 'r'), Loader=yaml.SafeLoader)
    32
    33    pathlib.Path(shlex.quote(config['youtube_dl']['dst_dir'])).mkdir(
    34        mode=0o700, parents=True, exist_ok=True)
    35    pathlib.Path(shlex.quote(
    36        config['youtube_dl']['archived_list_file'])).touch(mode=0o700,
    37                                                           exist_ok=True)
    38    original_files = sum(1 for line in open(
    39        shlex.quote(config['youtube_dl']['archived_list_file'])))
    40
    41    # Build binary.
    42    cmd = str()
    43    if 'binary' in config['youtube_dl']:
    44        for c in config['youtube_dl']['binary']:
    45            cmd += shlex.quote(c) + ' '
    46
    47    # youtube-dl might not return 0 even if some videos are correctly downloaded.
    48    command = 'pushd ' + shlex.quote(
    49        config['youtube_dl']['dst_dir']
    50    ) + ' ; ' + cmd + ' --verbose --config-location ' + shlex.quote(
    51        config['youtube_dl']['options_file']) + ' --batch ' + shlex.quote(
    52            config['youtube_dl']
    53            ['url_list_file']) + ' --download-archive ' + shlex.quote(
    54                config['youtube_dl']['archived_list_file']) + ' ; popd'
    55    fpyutils.shell.execute_command_live_output(command)
    56
    57    # For this to work be sure to set the no-mtime option in the options file.
    58    # Only the video files, infact, would retain the original modification time
    59    # (not the modification time correpsponding to the download time).
    60    # All the other files such as thumbnails and subtitles do not retain the
    61    # original mtime. For this reason it is simpler not to consider the
    62    # original mtime.
    63    deleted_files = 0
    64    if config['delete']['enabled']:
    65        for f in sorted(
    66                pathlib.Path(shlex.quote(
    67                    config['youtube_dl']['dst_dir'])).glob('*/*')):
    68            if f.is_file() and (datetime.date.today() -
    69                                datetime.date.fromtimestamp(f.stat().st_mtime)
    70                                ).days > config['delete']['days_to_keep']:
    71                f.unlink()
    72                deleted_files += 1
    73
    74    final_files = sum(1 for line in open(
    75        shlex.quote(config['youtube_dl']['archived_list_file'])))
    76
    77    downloaded_files = final_files - original_files
    78
    79    message = 'DW: ' + str(downloaded_files) + ' ; RM: ' + str(deleted_files)
    80    if config['notify']['gotify']['enabled']:
    81        m = config['notify']['gotify']['message'] + '\n' + message
    82        fpyutils.notify.send_gotify_message(
    83            config['notify']['gotify']['url'],
    84            config['notify']['gotify']['token'], m,
    85            config['notify']['gotify']['title'],
    86            config['notify']['gotify']['priority'])
    87    if config['notify']['email']['enabled']:
    88        fpyutils.notify.send_email(message,
    89                                   config['notify']['email']['smtp_server'],
    90                                   config['notify']['email']['port'],
    91                                   config['notify']['email']['sender'],
    92                                   config['notify']['email']['user'],
    93                                   config['notify']['email']['password'],
    94                                   config['notify']['email']['receiver'],
    95                                   config['notify']['email']['subject'])
    
  6. create a configuration file

    /home/jobs/scripts/by-user/myuser/youtube_dl.some_subject.yaml
     1#
     2# youtube_dl.some_subject.yaml
     3#
     4# Copyright (C) 2019-2022 Franco Masotti (franco \D\o\T masotti {-A-T-} tutanota \D\o\T com)
     5#
     6# This program is free software: you can redistribute it and/or modify
     7# it under the terms of the GNU General Public License as published by
     8# the Free Software Foundation, either version 3 of the License, or
     9# (at your option) any later version.
    10#
    11# This program is distributed in the hope that it will be useful,
    12# but WITHOUT ANY WARRANTY; without even the implied warranty of
    13# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
    14# GNU General Public License for more details.
    15#
    16# You should have received a copy of the GNU General Public License
    17# along with this program.  If not, see <https://www.gnu.org/licenses/>.
    18
    19youtube_dl:
    20    # A list of strings corresponding to binaries and options.
    21    # You need to specify at least the youtube-dl binary.
    22    binary:
    23        # - torsocks
    24        # - --isolate
    25        - /home/myuser/.local/bin/yt-dlp
    26
    27    # The configuration file for youtube-dl.
    28    options_file: '/home/jobs/scripts/by-user/myuser/youtube_dl.some_subject.options'
    29
    30    # The base directory where all the videos will lie.
    31    dst_dir: '/home/myuser/videos/bot/some_subject'
    32
    33    # The base directory where all the videos will lie.
    34    url_list_file: '/home/jobs/scripts/by-user/myuser/youtube_dl.some_subject.txt'
    35
    36    # The file containing the list of downloaded urls.
    37    archived_list_file: '/home/myuser/videos/bot/some_subject/archived.txt'
    38
    39delete:
    40    enabled: true
    41    days_to_keep: 60
    42
    43notify:
    44    email:
    45        enabled: false
    46        smtp_server: 'smtp.gmail.com'
    47        port: 465
    48        sender: 'myusername@gmail.com'
    49        user: 'myusername'
    50        password: 'my awesome password'
    51        receiver: 'myusername@gmail.com'
    52        subject: 'video bot: some_subject'
    53    gotify:
    54        enabled: false
    55        url: '<gotify url>'
    56        token: '<app token>'
    57        title: 'video bot'
    58        message: 'some_subject'
    59        priority: 5
    
  7. create the options file which select all the options used by yt-dlp

    /home/jobs/scripts/by-user/myuser/youtube_dl.some_subject.options
     1#
     2# youtube_dl.some_subject.options
     3#
     4# Copyright (C) 2019 Franco Masotti (franco \D\o\T masotti {-A-T-} tutanota \D\o\T com)
     5#
     6# This program is free software: you can redistribute it and/or modify
     7# it under the terms of the GNU General Public License as published by
     8# the Free Software Foundation, either version 3 of the License, or
     9# (at your option) any later version.
    10#
    11# This program is distributed in the hope that it will be useful,
    12# but WITHOUT ANY WARRANTY; without even the implied warranty of
    13# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
    14# GNU General Public License for more details.
    15#
    16# You should have received a copy of the GNU General Public License
    17# along with this program.  If not, see <https://www.gnu.org/licenses/>.
    18
    19# --get-id
    20
    21# Filter by video release date or by playlist id number,
    22# up to playlist-end videos back.
    23#--dateafter now-1days
    24--playlist-start 1
    25--playlist-end 5
    26
    27--add-metadata
    28
    29# Download data in parallel. Change or remove the max-overall-download-limit option.
    30--external-downloader aria2c
    31--external-downloader-args '-c -j 3 -x 3 -s 3 -k 1M --max-overall-download-limit=512K'
    32
    33# Avoid geographical restrictions.
    34--geo-bypass
    35
    36--force-ipv4
    37--no-color
    38--ignore-errors
    39--continue
    40--no-cache-dir
    41
    42--socket-timeout 300
    43
    44# Prefer 720p videos and fallback to 480p.
    45--format 247+bestaudio/136+bestaudio/244+bestaudio/135+bestaudio
    46
    47# Subtitles.
    48--write-sub
    49--write-auto-sub
    50
    51# Output file should be a mkv container.
    52--merge-output-format mkv
    53
    54# Get the video thumbnail.
    55--write-thumbnail
    56
    57# File path.
    58# Subdirectories by yyyy-mm.
    59--output "%(uploader)s/%(upload_date>%Y)s/%(upload_date>%m)s/%(upload_date)s_%(playlist_index)s_%(title)s_%(id)s"
    60
    61--write-description
    62--write-info-json
    63#--write-annotations
    64--convert-subs vtt
    65
    66# Save the list of the previuosly downloaded videos.
    67--download-archive archive.txt
    68
    69# Slugify file names.
    70--restrict-filenames
    71
    72--no-overwrite
    73--no-call-home
    74--prefer-free-formats
    75--fixup detect_or_warn
    76--prefer-ffmpeg
    77
    78# Transform to hvec, 8-bit color.
    79#--postprocessor-args "-c:v libx265 -preset veryfast -x265-params crf=28 -pix_fmt yuv420p -c:a copy"
    80
    81# Sleep a number of seconds between one download and the other.
    82--sleep-interval 10
    83
    84# Use the following url list.
    85--batch channels.txt
    86
    87# Very important if you enable automatic file deletion.
    88--no-mtime
    89
    90# Proxy. Enable and edit this option as needed.
    91# --proxy 127.0.0.1:8123
    
  8. create a text file containing URLs of channels or playlists, one for each line

    /home/jobs/scripts/by-user/myuser/youtube_dl.some_subject.txt
    1http://a-newline
    2http://separated-list
    3http://of-urls.
    
  9. create the Systemd service unit file

    /home/jobs/services/by-user/myuser/youtube-dl.some_subject.service
     1[Unit]
     2Description=Download youtube-dl.some_subject videos
     3Requires=network-online.target
     4After=network-online.target
     5
     6[Service]
     7Type=simple
     8ExecStart=/home/jobs/scripts/by-user/myuser/youtube_dl.py /home/jobs/scripts/by-user/myuser/youtube_dl.some_subject.yaml
     9User=myuser
    10Group=myuser
    11
    12[Install]
    13WantedBy=multi-user.target
    
  10. create the Systemd service timer unit file

    /home/jobs/services/by-user/myuser/youtube-dl.some_subject.timer
    1[Unit]
    2Description=Once a day download some_subject videos
    3
    4[Timer]
    5OnCalendar=*-*-* 3:30:00
    6Persistent=true
    7
    8[Install]
    9WantedBy=timers.target
    
  11. fix the permissions

    chown -R myuser:myuser /home/jobs/{scripts,services}/by-user/myuser
    chmod 700 -R /home/jobs/{scripts,services}/by-user/myuser
    
  12. run the deploy script

Footnotes

1

https://software.franco.net.eu.org/frnmst/automated-tasks GNU GPLv3+, copyright (c) 2019-2022, Franco Masotti

2

https://wiki.archlinux.org/index.php/Youtube-dl GNU Free Documentation License 1.3 or later, Copyright (c) ArchWiki contributors

3

https://github.com/yt-dlp/yt-dlp Unilicense