QEMU

Use declarative configuration to configure and run virtual machines on a server via QEMU

Running

See also

  • A collection of scripts I have written and/or adapted that I currently use on my systems as automated tasks 1

Basic steps

  1. install these packages

    apt-get install qemu-system-x86 openssh-client python3-yaml
    
  2. install fpyutils. See reference

  3. create a new user

    useradd --system -s /bin/bash -U qvm
    passwd qvm
    usermod -aG jobs qvm
    
  4. create the jobs directories. See reference

    mkdir -p /home/jobs/{scripts,services}/by-user/qvm
    
  5. create the qvm script

    /home/jobs/scripts/by-user/qvm/qvm.py
      1#!/usr/bin/env python3
      2#
      3# qvm.py
      4#
      5# Copyright (C) 2021 Franco Masotti (franco \D\o\T masotti {-A-T-} tutanota \D\o\T com)
      6#
      7# This program is free software: you can redistribute it and/or modify
      8# it under the terms of the GNU General Public License as published by
      9# the Free Software Foundation, either version 3 of the License, or
     10# (at your option) any later version.
     11#
     12# This program is distributed in the hope that it will be useful,
     13# but WITHOUT ANY WARRANTY; without even the implied warranty of
     14# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
     15# GNU General Public License for more details.
     16#
     17# You should have received a copy of the GNU General Public License
     18# along with this program.  If not, see <http://www.gnu.org/licenses/>.
     19#
     20#
     21# Original license header:
     22#
     23# qvm - Trivial management of 64 bit virtual machines with qemu.
     24#
     25# Written in 2016 by Franco Masotti/frnmst <franco.masotti@student.unife.it>
     26#
     27# To the extent possible under law, the author(s) have dedicated all
     28# copyright and related and neighboring rights to this software to the public
     29# domain worldwide. This software is distributed without any warranty.
     30#
     31# You should have received a copy of the CC0 Public Domain Dedication along
     32# with this software. If not, see
     33# <http://creativecommons.org/publicdomain/zero/1.0/>.
     34r"""Run virtual machines."""
     35
     36import shlex
     37import sys
     38
     39import fpyutils
     40import yaml
     41
     42
     43def build_remote_command(prf: dict) -> str:
     44    if prf['system']['display']['enabled']:
     45        # See https://unix.stackexchange.com/a/83812
     46        # See also the 'TCP FORWARDING' section in man 1 ssh.
     47        ssh = '-f -p ' + prf['system']['network']['ports']['host']['ssh'] + ' -L ' + prf['system']['network']['ports']['local']['vnc'] + ':127.0.0.1:' + prf['system']['network']['ports']['host']['vnc'] + ' -l ' + prf['system']['users']['host'] + ' ' + prf['system']['network']['addresses']['host']
     48        ssh += ' sleep 10; vncviewer 127.0.0.1::' + prf['system']['network']['ports']['local']['vnc']
     49    else:
     50        ssh = '-p ' + prf['system']['network']['ports']['guest']['ssh'] + ' -l ' + prf['system']['users']['guest'] + ' ' + prf['system']['network']['addresses']['host']
     51
     52    return (
     53        prf['executables']['ssh']
     54        + ' ' + ssh
     55    )
     56
     57
     58def build_local_command(prf: dict) -> str:
     59    head = str()
     60
     61    # Memory.
     62    memory = ' -m ' + prf['system']['memory']
     63
     64    # CPU.
     65    cpu = ' -smp ' + prf['system']['cpu']['cores'] + ' -cpu ' + prf['system']['cpu']['type']
     66
     67    # Display.
     68    if prf['system']['display']['enabled']:
     69        if prf['system']['display']['vnc']['enabled']:
     70            display_number = int(prf['system']['display']['vnc']['port']) - 5900
     71            display = '-display none -monitor pty -vnc 127.0.0.1:' + str(display_number)
     72        else:
     73            display = '-display gtk'
     74    else:
     75        display = '-display none'
     76
     77    # Audio.
     78    if prf['system']['audio']['enabled']:
     79        audio = '-device ' + prf['system']['audio']['device']
     80        head += 'export QEMU_AUDIO_DRV=alsa;'
     81    else:
     82        audio = str()
     83
     84    # Network.
     85    if prf['system']['network']['enabled']:
     86        net = '-netdev user,id=user.0'
     87        i = 0
     88        for n in prf['system']['network']['ports']:
     89            for j in n:
     90                net += ',hostfwd=tcp::' + prf['system']['network']['ports'][i][j]['host'] + '-:' + prf['system']['network']['ports'][i][j]['guest']
     91                i += 1
     92        net += ' -device ' + prf['system']['network']['device'] + ',netdev=user.0'
     93    else:
     94        net = str()
     95
     96    # Mounts.
     97    if prf['system']['mount']['enabled']:
     98        mnt = str()
     99        i = 0
    100        for n in prf['system']['mount']['mountpoints']:
    101            for j in n:
    102                mnt += ' -virtfs local,path=' + prf['system']['mount']['mountpoints'][i][j]['path'] + ',security_model=passthrough,mount_tag=' + prf['system']['mount']['mountpoints'][i][j]['mount tag']
    103                i += 1
    104    else:
    105        mnt = str()
    106
    107    # CD-ROM.
    108    if prf['system']['cdrom']['enabled']:
    109        cdrom = '-cdrom ' + prf['system']['cdrom']['device'] + ' -boot order=d'
    110    else:
    111        cdrom = str()
    112
    113    # Mass memory.
    114    hdd = str()
    115    for drive in prf['system']['drives']:
    116        hdd += ' -drive file=' + drive
    117
    118    return (
    119        head
    120        + ' ' + prf['executables']['qemu']
    121        + ' ' + prf['options']
    122        + ' ' + memory
    123        + ' ' + cpu
    124        + ' ' + display
    125        + ' ' + net
    126        + ' ' + cdrom
    127        + ' ' + audio
    128        + ' ' + mnt
    129        + ' ' + hdd
    130    )
    131
    132
    133if __name__ == '__main__':
    134    configuration_file = shlex.quote(sys.argv[1])
    135    config = yaml.load(open(configuration_file, 'r'), Loader=yaml.SafeLoader)
    136    type = shlex.quote(sys.argv[2])
    137    profile = shlex.quote(sys.argv[3])
    138
    139    prf = config[type][profile]
    140    if prf['enabled']:
    141        if type == 'local':
    142            command = build_local_command(prf)
    143        elif type == 'remote':
    144            command = build_remote_command(prf)
    145
    146    fpyutils.shell.execute_command_live_output(command, dry_run=False)
    
  6. create the configuration for qvm

    /home/jobs/scripts/by-user/qvm/qvm.yaml
      1#
      2# qvm.yaml
      3#
      4# Copyright (C) 2021-2022 Franco Masotti (franco \D\o\T masotti {-A-T-} tutanota \D\o\T com)
      5#
      6# This program is free software: you can redistribute it and/or modify
      7# it under the terms of the GNU General Public License as published by
      8# the Free Software Foundation, either version 3 of the License, or
      9# (at your option) any later version.
     10#
     11# This program is distributed in the hope that it will be useful,
     12# but WITHOUT ANY WARRANTY; without even the implied warranty of
     13# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
     14# GNU General Public License for more details.
     15#
     16# You should have received a copy of the GNU General Public License
     17# along with this program.  If not, see <http://www.gnu.org/licenses/>.
     18#
     19#
     20# Original license header:
     21#
     22# qvm - Trivial management of 64 bit virtual machines with qemu.
     23#
     24# Written in 2016 by Franco Masotti/frnmst (franco \D\o\T masotti {-A-T-} student \D\o\T unife \D\o\T it)
     25#
     26# To the extent possible under law, the author(s) have dedicated all
     27# copyright and related and neighboring rights to this software to the public
     28# domain worldwide. This software is distributed without any warranty.
     29#
     30# You should have received a copy of the CC0 Public Domain Dedication along
     31# with this software. If not, see
     32# <http://creativecommons.org/publicdomain/zero/1.0/>.
     33
     34local:
     35    test:
     36        enabled: true
     37
     38        executables:
     39            qemu: '/usr/bin/qemu-system-x86_64'
     40
     41        # Genetic options.
     42        options: '-enable-kvm'
     43
     44        system:
     45            memory: '12G'
     46            cpu:
     47                cores: '6'
     48
     49                # See
     50                # $ qemu -cpu help
     51                type: 'host'
     52
     53            # Mass memory. Use device name with options.
     54            drives:
     55                # IMPORTANT: enable the following during the setup.
     56                # - '/home/user/qvm/development.qcow2'
     57
     58                # Enable the following after the setup.
     59                - '/home/user/qvm/development.qcow2.mod'
     60
     61               # Mass memory. Use device name with options.
     62               # - '/dev/sdx,format=raw'
     63               # - '/dev/sdy,format=raw'
     64
     65            # Enable this for maintenance or installation.
     66            cdrom:
     67                # IMPORTANT: set the following to true during the setup.
     68                # enabled: true
     69
     70                # Set the following to true after the setup.
     71                enabled: false
     72
     73                # Device or file name.
     74                device: debian.iso
     75
     76            # Shared data using virtfs.
     77            mount:
     78                enabled: false
     79
     80                # Enable this if you need to mount a directory using 9p.
     81                # enabled: true
     82                mountpoints:
     83                    - shared_data:
     84                        path: '/home/qvm/shares/test'
     85                        mount tag: 'shared'
     86
     87            # If enabled is false never show a display.
     88            display:
     89                enabled: true
     90                vnc:
     91                    enabled: true
     92                    port: 5900
     93
     94            audio:
     95                enabled: true
     96                device: 'AC97'
     97
     98            network:
     99                enabled: true
    100
    101                # Use this device if you use a kernel such as
    102                # the one used by Debian in the
    103                # linux-image-cloud-amd64
    104                # package:
    105                device: 'virtio-net-pci'
    106                # This device works with normal kernels:
    107                # device: 'e1000'
    108
    109                # TCP ports only.
    110                ports:
    111                    - ssh:
    112                        host:      '2222'
    113                        guest:     '22'
    114                    - other:
    115                        host:      '5555'
    116                        guest:     '3050'
    117                    - other2:
    118                        host:      '5556'
    119                        guest:     '3051'
    
  7. create the Systemd service unit file for qvm

    /home/jobs/services/by-user/qvm/qvm.local_test.service
     1[Unit]
     2Description=Run qvm local test
     3Requires=network-online.target
     4After=network-online.target
     5
     6[Service]
     7Type=simple
     8ExecStart=/home/jobs/scripts/by-user/qvm/qvm.py /home/jobs/scripts/by-user/qvm/qvm.yaml local test
     9ExecStop=/usr/bin/bash -c '/usr/bin/ssh -p 2222 powermanager@127.0.0.1 sudo poweroff; sleep 30'
    10User=qvm
    11Group=qvm
    12
    13[Install]
    14WantedBy=multi-user.target
    
  8. fix the permissions

    chmod 700 -R /home/jobs/{scripts,services}/by-user/qvm
    chown -R qvm:qvm /home/jobs/{scripts,services}/by-user/qvm
    
  9. create a new virtual hard disk

    sudo -i -u qvm
    qemu-img create -f qcow2 development.qcow2 64G
    
  10. modify the configuration to point to development.qcow2, the virtual hdd. See the commented local.test.system.drives[0] key

    Also set the cdrom device and enabled values in local.test.cdrom

  11. run the installation

    ./qvm ./qvm.py local development
    
  12. once the installation is finished power down the machine

  13. create a backup virtual hard disk

    qemu-img create -f qcow2 -b development.qcow2 development.qcow2.mod
    
  14. set local.test.system.cdrom.enabled to false. Set local.test.system.drives[0] to development.qcow2.mod

  15. run the virual machine

    ./qvm ./qvm.py local development
    
  16. quit the virtual machine and go back to the root user

    exit
    
  17. run the deploy script

  18. if you are using iptables rules on the host machine modify the rules to let data through the shared ports

  19. continue with the client configuration

Automatic shutdown

See also

  • SSH config host match port 2

To be able to shutdown automatically when the Systemd service is stopped follow these instructions

  1. connect to the guest machine

  2. create a new user

    sudo -i
    useradd -m -s /bin/bash -U powermanager
    passwd powermanager
    
  3. change the sudoers file

    visudo
    

    Add this line

    powermanager ALL=(ALL) NOPASSWD:/sbin/poweroff
    
  4. go back on the host machine and create an SSH key so that the qvm host user can connect to the powermanager guest user. Do not encrypt the key with a passphrase

    sudo -i -u qvm
    ssh-keygen -t rsa -b 16384 -C "qvm@host-2022-01-01"
    

    Save the keys as ~/.ssh/powermanager_test.

    Have a look at the ExecStop command in the Systemd service unit file

  5. in the host machine, configure the SSH config file like this

    /home/qvm/.ssh/config
    1Match host 127.0.0.1 user powermanager exec "test %p = 2222"
    2    IdentitiesOnly yes
    3    IdentityFile ~/.ssh/powermanager_test
    
  6. copy the content of /home/qvm/.ssh/powermanager_test.pub into /home/powermanager/.ssh/authorized_keys of the guest machine

Using physical partitions

Instead of using QCOW2 disk files you can use existing phisical partitions and filesystems.

Warning

Remember NOT to mount the partitions while running because data loss occurs in that case.

  1. uncomment and edit the highlighted lines and comment the original drive line

    /home/jobs/scripts/by-user/qvm/qvm.yaml
    53            # Mass memory. Use device name with options.
    54            drives:
    55                # IMPORTANT: enable the following during the setup.
    56                # - '/home/user/qvm/development.qcow2'
    57
    58                # Enable the following after the setup.
    59                - '/home/user/qvm/development.qcow2.mod'
    60
    61               # Mass memory. Use device name with options.
    62               # - '/dev/sdx,format=raw'
    63               # - '/dev/sdy,format=raw'
    
  2. run the deploy script

Share a directory

If you need to share a directory you can use a 9p filesystem if the guest kernel supports it.

  1. connect to the guest machine and run the following

    sudo -i
    modprobe 9pnet_virtio
    

    In case your kernel does not support 9p you might get a message like the following. For example this is the case for Debian’s linux-image-cloud-amd64 kernel

    modprobe: FATAL: Module 9pnet_virtio not found in directory /lib/modules/5.10.0-9-cloud-amd64
    

    If that is the case you need to find a different method such as SSHFS

  2. create the mount directory

    exit
    mkdir -p /home/qvm/shares/test
    
  3. add a shared directory in the configuration

    /home/jobs/scripts/by-user/qvm/qvm.yaml
    76            # Shared data using virtfs.
    77            mount:
    78                enabled: false
    79
    80                # Enable this if you need to mount a directory using 9p.
    81                # enabled: true
    82                mountpoints:
    83                    - shared_data:
    84                        path: '/home/qvm/shares/test'
    85                        mount tag: 'shared'
    
  4. connect to the guest machine and add an fstab entry. In this example the directory is mounted in /home/vm/shared

    /etc/fstab
    1shared    /home/vm/shared 9p auto,access=any,x-systemd.automount,msize=268435456,trans=virtio,version=9p2000.L 0 0
    
  5. create the shared directory

    mkdir /home/vm/shared
    
  6. quit the virtual machine

  7. restart the virtual machine

    systemctl restart qvm.local_test.service
    

Resize disks

See also

  • How to Expand QCOW2 3

  • Provider::VMonG5K — EnOSlib 8.0.0a12 8.0.0a12 documentation 4

  1. stop the virtual machine

  2. install this package

    apt-get install libguestfs-tools
    

    Have a look at this bug report if you have problem installing

  3. resize for example by increasing to 40G more, where virtual_hard_disk is a backing file. I always set the backing file ending in .mod.

    qemu-img resize ${virtual_hard_disk} +40G
    
  4. make a backup

    cp -aR image.qcow2.mod image.qcow2.mod.bak
    
  5. get the partition name you want to expand. Usually partition names start from /dev/sda1 (note: these partition names are not the same as the host system ones!)

    virt-filesystems --long --human-readable --all --add ${virtual_hard_disk}
    
  6. execute the actual resize operation on the virtual partition and filesystem. You can use sda1 for example as partition_name

    virt-resize --expand ${patition_name} ${virtual_hard_disk}.bak ${virtual_hard_disk}
    
  7. start the virtual machine

  8. if everything works remove the ${virtual_hard_disk}.bak file

Rename disk files

See also

  • Qemu-img Cheatsheet | Programster’s Blog 5

Rename backing file

TODO

FSCK

You must run FSCK while the virtual machine is off if you want to fix the root partition.

See also

  • kvm virtualization - How to run fsck on guest VMs from KVM - Server Fault 6

  1. identify the troubling filesystem by running the virtual machine in “display” mode, not via SSH: if the broken partition is root your virtual machine might not get to load SSHD.

  2. stop the virtual machine

  3. load the virtual hard disk

    guestfish -a ${virtual_hard_disk}
    run
    list-filesystems
    

    You get a list of partitions with the last command, for example /dev/sda1.

  4. You can run various fsck commands such as

    e2fsck-f ${partition_name}
    e2fsck ${partition_name} /dev/sda1 forceall:true
    e2fsck ${partition_name} correct:true
    fsck ${partition_name}
    
  5. Quit the program and start the virtual machine

    exit
    

Footnotes

1

https://software.franco.net.eu.org/frnmst/automated-tasks GNU GPLv3+, copyright (c) 2019-2022, Franco Masotti

2

https://superuser.com/a/870918 CC BY-SA 3.0, (c) 2015, Kenster (at superuser.com)

3

https://blog.khmersite.net/2020/11/how-to-expand-qcow2/ unknown license, © 2022

4

https://discovery.gitlabpages.inria.fr/enoslib/tutorials/vmong5k.html#changing-resource-size-of-virtual-machines GPLv3+, © Copyright 2017, Ronan-Alexandre Cherrueau, Matthieu Simonin

5

https://blog.programster.org/qemu-img-cheatsheet unknown license,

6

https://serverfault.com/a/380210 CC BY-SA 3.0, Copyright (c) 2012 mrc (at serverfault.com)