QEMU

Use declarative configuration to configure and run virtual machines on a server via QEMU

Running

Vedi anche

  • A collection of scripts I have written and/or adapted that I currently use on my systems as automated tasks 1

Basic steps

  1. install these packages

    apt-get install qemu-system-x86 openssh-client python3-yaml

  2. install fpyutils. See reference

  3. create a new user

    useradd --system -s /bin/bash -U qvm
passwd qvm
usermod -aG jobs qvm

  4. create the jobs directories. See reference

    mkdir -p /home/jobs/{scripts,services}/by-user/qvm

  5. create the qvm script

    /home/jobs/scripts/by-user/qvm/qvm.py
      1#!/usr/bin/env python3
  2#
  3# qvm.py
  4#
  5# Copyright (C) 2021 Franco Masotti (franco \D\o\T masotti {-A-T-} tutanota \D\o\T com)
  6#
  7# This program is free software: you can redistribute it and/or modify
  8# it under the terms of the GNU General Public License as published by
  9# the Free Software Foundation, either version 3 of the License, or
 10# (at your option) any later version.
 11#
 12# This program is distributed in the hope that it will be useful,
 13# but WITHOUT ANY WARRANTY; without even the implied warranty of
 14# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 15# GNU General Public License for more details.
 16#
 17# You should have received a copy of the GNU General Public License
 18# along with this program.  If not, see <http://www.gnu.org/licenses/>.
 19#
 20#
 21# Original license header:
 22#
 23# qvm - Trivial management of 64 bit virtual machines with qemu.
 24#
 25# Written in 2016 by Franco Masotti/frnmst <franco.masotti@student.unife.it>
 26#
 27# To the extent possible under law, the author(s) have dedicated all
 28# copyright and related and neighboring rights to this software to the public
 29# domain worldwide. This software is distributed without any warranty.
 30#
 31# You should have received a copy of the CC0 Public Domain Dedication along
 32# with this software. If not, see
 33# <http://creativecommons.org/publicdomain/zero/1.0/>.
 34r"""Run virtual machines."""
 35
 36import shlex
 37import sys
 38
 39import fpyutils
 40import yaml
 41
 42
 43def build_remote_command(prf: dict) -> str:
 44    if prf['system']['display']['enabled']:
 45        # See https://unix.stackexchange.com/a/83812
 46        # See also the 'TCP FORWARDING' section in man 1 ssh.
 47        ssh = '-f -p ' + prf['system']['network']['ports']['host']['ssh'] + ' -L ' + prf['system']['network']['ports']['local']['vnc'] + ':127.0.0.1:' + prf['system']['network']['ports']['host']['vnc'] + ' -l ' + prf['system']['users']['host'] + ' ' + prf['system']['network']['addresses']['host']
 48        ssh += ' sleep 10; vncviewer 127.0.0.1::' + prf['system']['network']['ports']['local']['vnc']
 49    else:
 50        ssh = '-p ' + prf['system']['network']['ports']['guest']['ssh'] + ' -l ' + prf['system']['users']['guest'] + ' ' + prf['system']['network']['addresses']['host']
 51
 52    return (
 53        prf['executables']['ssh']
 54        + ' ' + ssh
 55    )
 56
 57
 58def build_local_command(prf: dict) -> str:
 59    head = str()
 60
 61    # Memory.
 62    memory = ' -m ' + prf['system']['memory']
 63
 64    # CPU.
 65    cpu = ' -smp ' + prf['system']['cpu']['cores'] + ' -cpu ' + prf['system']['cpu']['type']
 66
 67    # Display.
 68    if prf['system']['display']['enabled']:
 69        if prf['system']['display']['vnc']['enabled']:
 70            display_number = int(prf['system']['display']['vnc']['port']) - 5900
 71            display = '-display none -monitor pty -vnc 127.0.0.1:' + str(display_number)
 72        else:
 73            display = '-display gtk'
 74    else:
 75        display = '-display none'
 76
 77    # Audio.
 78    if prf['system']['audio']['enabled']:
 79        audio = '-device ' + prf['system']['audio']['device']
 80        head += 'export QEMU_AUDIO_DRV=alsa;'
 81    else:
 82        audio = str()
 83
 84    # Network.
 85    if prf['system']['network']['enabled']:
 86        net = '-netdev user,id=user.0'
 87        i = 0
 88        for n in prf['system']['network']['ports']:
 89            for j in n:
 90                net += ',hostfwd=tcp::' + prf['system']['network']['ports'][i][j]['host'] + '-:' + prf['system']['network']['ports'][i][j]['guest']
 91                i += 1
 92        net += ' -device ' + prf['system']['network']['device'] + ',netdev=user.0'
 93    else:
 94        net = str()
 95
 96    # Mounts.
 97    if prf['system']['mount']['enabled']:
 98        mnt = str()
 99        i = 0
100        for n in prf['system']['mount']['mountpoints']:
101            for j in n:
102                mnt += ' -virtfs local,path=' + prf['system']['mount']['mountpoints'][i][j]['path'] + ',security_model=passthrough,mount_tag=' + prf['system']['mount']['mountpoints'][i][j]['mount tag']
103                i += 1
104    else:
105        mnt = str()
106
107    # CD-ROM.
108    if prf['system']['cdrom']['enabled']:
109        cdrom = '-cdrom ' + prf['system']['cdrom']['device'] + ' -boot order=d'
110    else:
111        cdrom = str()
112
113    # Mass memory.
114    hdd = str()
115    for drive in prf['system']['drives']:
116        hdd += ' -drive file=' + drive
117
118    return (
119        head
120        + ' ' + prf['executables']['qemu']
121        + ' ' + prf['options']
122        + ' ' + memory
123        + ' ' + cpu
124        + ' ' + display
125        + ' ' + net
126        + ' ' + cdrom
127        + ' ' + audio
128        + ' ' + mnt
129        + ' ' + hdd
130    )
131
132
133if __name__ == '__main__':
134    configuration_file = shlex.quote(sys.argv[1])
135    config = yaml.load(open(configuration_file, 'r'), Loader=yaml.SafeLoader)
136    type = shlex.quote(sys.argv[2])
137    profile = shlex.quote(sys.argv[3])
138
139    prf = config[type][profile]
140    if prf['enabled']:
141        if type == 'local':
142            command = build_local_command(prf)
143        elif type == 'remote':
144            command = build_remote_command(prf)
145
146    fpyutils.shell.execute_command_live_output(command, dry_run=False)

  6. create the configuration for qvm

    /home/jobs/scripts/by-user/qvm/qvm.yaml
      1#
  2# qvm.yaml
  3#
  4# Copyright (C) 2021-2022 Franco Masotti (franco \D\o\T masotti {-A-T-} tutanota \D\o\T com)
  5#
  6# This program is free software: you can redistribute it and/or modify
  7# it under the terms of the GNU General Public License as published by
  8# the Free Software Foundation, either version 3 of the License, or
  9# (at your option) any later version.
 10#
 11# This program is distributed in the hope that it will be useful,
 12# but WITHOUT ANY WARRANTY; without even the implied warranty of
 13# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 14# GNU General Public License for more details.
 15#
 16# You should have received a copy of the GNU General Public License
 17# along with this program.  If not, see <http://www.gnu.org/licenses/>.
 18#
 19#
 20# Original license header:
 21#
 22# qvm - Trivial management of 64 bit virtual machines with qemu.
 23#
 24# Written in 2016 by Franco Masotti/frnmst (franco \D\o\T masotti {-A-T-} student \D\o\T unife \D\o\T it)
 25#
 26# To the extent possible under law, the author(s) have dedicated all
 27# copyright and related and neighboring rights to this software to the public
 28# domain worldwide. This software is distributed without any warranty.
 29#
 30# You should have received a copy of the CC0 Public Domain Dedication along
 31# with this software. If not, see
 32# <http://creativecommons.org/publicdomain/zero/1.0/>.
 33
 34local:
 35    test:
 36        enabled: true
 37
 38        executables:
 39            qemu: '/usr/bin/qemu-system-x86_64'
 40
 41        # Genetic options.
 42        options: '-enable-kvm'
 43
 44        system:
 45            memory: '12G'
 46            cpu:
 47                cores: '6'
 48
 49                # See
 50                # $ qemu -cpu help
 51                type: 'host'
 52
 53            # Mass memory. Use device name with options.
 54            drives:
 55                # IMPORTANT: enable the following during the setup.
 56                # - '/home/user/qvm/development.qcow2'
 57
 58                # Enable the following after the setup.
 59                - '/home/user/qvm/development.qcow2.mod'
 60
 61               # Mass memory. Use device name with options.
 62               # - '/dev/sdx,format=raw'
 63               # - '/dev/sdy,format=raw'
 64
 65            # Enable this for maintenance or installation.
 66            cdrom:
 67                # IMPORTANT: set the following to true during the setup.
 68                # enabled: true
 69
 70                # Set the following to true after the setup.
 71                enabled: false
 72
 73                # Device or file name.
 74                device: debian.iso
 75
 76            # Shared data using virtfs.
 77            mount:
 78                enabled: false
 79
 80                # Enable this if you need to mount a directory using 9p.
 81                # enabled: true
 82                mountpoints:
 83                    - shared_data:
 84                        path: '/home/qvm/shares/test'
 85                        mount tag: 'shared'
 86
 87            # If enabled is false never show a display.
 88            display:
 89                enabled: true
 90                vnc:
 91                    enabled: true
 92                    port: 5900
 93
 94            audio:
 95                enabled: true
 96                device: 'AC97'
 97
 98            network:
 99                enabled: true
100
101                # Use this device if you use a kernel such as
102                # the one used by Debian in the
103                # linux-image-cloud-amd64
104                # package:
105                device: 'virtio-net-pci'
106                # This device works with normal kernels:
107                # device: 'e1000'
108
109                # TCP ports only.
110                ports:
111                    - ssh:
112                        host:      '2222'
113                        guest:     '22'
114                    - other:
115                        host:      '5555'
116                        guest:     '3050'
117                    - other2:
118                        host:      '5556'
119                        guest:     '3051'

  7. create the Systemd service unit file for qvm

    /home/jobs/services/by-user/qvm/qvm.local_test.service
     1[Unit]
 2Description=Run qvm local test
 3Requires=network-online.target
 4After=network-online.target
 5
 6[Service]
 7Type=simple
 8ExecStart=/home/jobs/scripts/by-user/qvm/qvm.py /home/jobs/scripts/by-user/qvm/qvm.yaml local test
 9ExecStop=/usr/bin/bash -c '/usr/bin/ssh -p 2222 powermanager@127.0.0.1 sudo poweroff; sleep 30'
10User=qvm
11Group=qvm
12
13[Install]
14WantedBy=multi-user.target

  8. fix the permissions

    chmod 700 -R /home/jobs/{scripts,services}/by-user/qvm
chown -R qvm:qvm /home/jobs/{scripts,services}/by-user/qvm

  9. create a new virtual hard disk

    sudo -i -u qvm
qemu-img create -f qcow2 development.qcow2 64G

  10. modify the configuration to point to development.qcow2, the virtual hdd. See the commented local.test.system.drives[0] key

    Also set the cdrom device and enabled values in local.test.cdrom

  11. run the installation

    ./qvm ./qvm.py local development

  12. once the installation is finished power down the machine

  13. create a backup virtual hard disk

    qemu-img create -f qcow2 -b development.qcow2 development.qcow2.mod

  14. set local.test.system.cdrom.enabled to false. Set local.test.system.drives[0] to development.qcow2.mod

  15. run the virual machine

    ./qvm ./qvm.py local development

  16. quit the virtual machine and go back to the root user

    exit

  17. run the deploy script

  18. if you are using iptables rules on the host machine modify the rules to let data through the shared ports

  19. continue with the client configuration

Automatic shutdown

Vedi anche

  • SSH config host match port 2

To be able to shutdown automatically when the Systemd service is stopped follow these instructions

  1. connect to the guest machine

  2. create a new user

    sudo -i
useradd -m -s /bin/bash -U powermanager
passwd powermanager

  3. change the sudoers file

    visudo

    Add this line

    powermanager ALL=(ALL) NOPASSWD:/sbin/poweroff

  4. go back on the host machine and create an SSH key so that the qvm host user can connect to the powermanager guest user. Do not encrypt the key with a passphrase

    sudo -i -u qvm
ssh-keygen -t rsa -b 16384 -C "qvm@host-2022-01-01"

    Save the keys as ~/.ssh/powermanager_test.

    Have a look at the ExecStop command in the Systemd service unit file

  5. in the host machine, configure the SSH config file like this

    /home/qvm/.ssh/config
    1Match host 127.0.0.1 user powermanager exec "test %p = 2222"
2    IdentitiesOnly yes
3    IdentityFile ~/.ssh/powermanager_test

  6. copy the content of /home/qvm/.ssh/powermanager_test.pub into /home/powermanager/.ssh/authorized_keys of the guest machine

Using physical partitions

Instead of using QCOW2 disk files you can use existing phisical partitions and filesystems.

Avvertimento

Remember NOT to mount the partitions while running because data loss occurs in that case.

  1. uncomment and edit the highlighted lines and comment the original drive line

    /home/jobs/scripts/by-user/qvm/qvm.yaml
    53            # Mass memory. Use device name with options.
54            drives:
55                # IMPORTANT: enable the following during the setup.
56                # - '/home/user/qvm/development.qcow2'
57
58                # Enable the following after the setup.
59                - '/home/user/qvm/development.qcow2.mod'
60
61               # Mass memory. Use device name with options.
62               # - '/dev/sdx,format=raw'
63               # - '/dev/sdy,format=raw'

  2. run the deploy script

Share a directory

If you need to share a directory you can use a 9p filesystem if the guest kernel supports it.

  1. connect to the guest machine and run the following

    sudo -i
modprobe 9pnet_virtio

    In case your kernel does not support 9p you might get a message like the following. For example this is the case for Debian’s linux-image-cloud-amd64 kernel

    modprobe: FATAL: Module 9pnet_virtio not found in directory /lib/modules/5.10.0-9-cloud-amd64

    If that is the case you need to find a different method such as SSHFS

  2. create the mount directory

    exit
mkdir -p /home/qvm/shares/test

  3. add a shared directory in the configuration

    /home/jobs/scripts/by-user/qvm/qvm.yaml
    76            # Shared data using virtfs.
77            mount:
78                enabled: false
79
80                # Enable this if you need to mount a directory using 9p.
81                # enabled: true
82                mountpoints:
83                    - shared_data:
84                        path: '/home/qvm/shares/test'
85                        mount tag: 'shared'

  4. connect to the guest machine and add an fstab entry. In this example the directory is mounted in /home/vm/shared

    /etc/fstab
    1shared    /home/vm/shared 9p auto,access=any,x-systemd.automount,msize=268435456,trans=virtio,version=9p2000.L 0 0

  5. create the shared directory

    mkdir /home/vm/shared

  6. quit the virtual machine

  7. restart the virtual machine

    systemctl restart qvm.local_test.service

Resize disks

Vedi anche

  • How to Expand QCOW2 3

  • Provider::VMonG5K — EnOSlib 8.0.0a12 8.0.0a12 documentation 4

  1. stop the virtual machine

  2. install this package

    apt-get install libguestfs-tools

    Have a look at this bug report if you have problem installing

  3. resize for example by increasing to 40G more, where virtual_hard_disk is a backing file. I always set the backing file ending in .mod.

    qemu-img resize ${virtual_hard_disk} +40G

  4. make a backup

    cp -aR image.qcow2.mod image.qcow2.mod.bak

  5. get the partition name you want to expand. Usually partition names start from /dev/sda1 (note: these partition names are not the same as the host system ones!)

    virt-filesystems --long --human-readable --all --add ${virtual_hard_disk}

  6. execute the actual resize operation on the virtual partition and filesystem. You can use sda1 for example as partition_name

    virt-resize --expand ${patition_name} ${virtual_hard_disk}.bak ${virtual_hard_disk}

  7. start the virtual machine

  8. if everything works remove the ${virtual_hard_disk}.bak file

Rename disk files

Vedi anche

  • Qemu-img Cheatsheet | Programster’s Blog 5

Rename backing file

TODO

FSCK

You must run FSCK while the virtual machine is off if you want to fix the root partition.

Vedi anche

  • kvm virtualization - How to run fsck on guest VMs from KVM - Server Fault 6

  1. identify the troubling filesystem by running the virtual machine in «display» mode, not via SSH: if the broken partition is root your virtual machine might not get to load SSHD.

  2. stop the virtual machine

  3. load the virtual hard disk

    guestfish -a ${virtual_hard_disk}
run
list-filesystems

    You get a list of partitions with the last command, for example /dev/sda1.

  4. You can run various fsck commands such as

    e2fsck-f ${partition_name}
e2fsck ${partition_name} /dev/sda1 forceall:true
e2fsck ${partition_name} correct:true
fsck ${partition_name}

  5. Quit the program and start the virtual machine

    exit

Footnotes

1

https://software.franco.net.eu.org/frnmst/automated-tasks GNU GPLv3+, copyright (c) 2019-2022, Franco Masotti

2

https://superuser.com/a/870918 CC BY-SA 3.0, (c) 2015, Kenster (at superuser.com)

3

https://blog.khmersite.net/2020/11/how-to-expand-qcow2/ unknown license, © 2022

4

https://discovery.gitlabpages.inria.fr/enoslib/tutorials/vmong5k.html#changing-resource-size-of-virtual-machines GPLv3+, © Copyright 2017, Ronan-Alexandre Cherrueau, Matthieu Simonin

5

https://blog.programster.org/qemu-img-cheatsheet unknown license,

6

https://serverfault.com/a/380210 CC BY-SA 3.0, Copyright (c) 2012 mrc (at serverfault.com)