From b93b3c571e8783a7c44af181c6489ce29ede8223 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Marek=20Marczykowski-G=C3=B3recki?= Date: Tue, 21 May 2024 20:59:46 +0200 Subject: [PATCH 1/5] Convert to RST --- .readthedocs.yaml | 25 + _dev/index.rst | 23 +- conf.py | 131 ++ developer/building/development-workflow.md | 538 ------ developer/building/development-workflow.rst | 647 +++++++ developer/building/qubes-builder-details.md | 59 - developer/building/qubes-builder-details.rst | 126 ++ developer/building/qubes-builder.md | 218 --- developer/building/qubes-builder.rst | 303 +++ developer/building/qubes-iso-building.md | 219 --- developer/building/qubes-iso-building.rst | 278 +++ developer/building/qubes-template-configs.md | 8 - developer/code/code-signing.md | 197 -- developer/code/code-signing.rst | 248 +++ developer/code/coding-style.md | 185 -- developer/code/coding-style.rst | 378 ++++ developer/code/license.md | 24 - developer/code/license.rst | 19 + developer/code/source-code.md | 82 - developer/code/source-code.rst | 102 + developer/debugging/automated-tests.md | 279 --- developer/debugging/automated-tests.rst | 355 ++++ developer/debugging/mount-lvm-image.md | 54 - developer/debugging/mount-lvm-image.rst | 60 + developer/debugging/safe-remote-ttys.md | 76 - developer/debugging/safe-remote-ttys.rst | 97 + developer/debugging/test-bench.md | 235 --- developer/debugging/test-bench.rst | 306 +++ developer/debugging/vm-interface.md | 225 --- developer/debugging/vm-interface.rst | 441 +++++ developer/debugging/windows-debugging.md | 89 - developer/debugging/windows-debugging.rst | 169 ++ developer/general/continuous-integration.md | 39 - developer/general/continuous-integration.rst | 41 + developer/general/devel-books.md | 31 - developer/general/devel-books.rst | 50 + .../general/developing-gui-applications.md | 102 - .../general/developing-gui-applications.rst | 142 ++ .../general/documentation-style-guide.md | 355 ---- .../general/documentation-style-guide.rst | 735 ++++++++ developer/general/gsoc.md | 660 ------- developer/general/gsoc.rst | 991 ++++++++++ developer/general/gsod.md | 187 -- developer/general/gsod.rst | 424 +++++ .../general/how-to-edit-the-documentation.md | 198 -- .../general/how-to-edit-the-documentation.rst | 240 +++ developer/general/package-contributions.md | 102 - developer/general/package-contributions.rst | 183 ++ developer/general/research.md | 17 - developer/general/research.rst | 66 + developer/general/usability-ux.md | 230 --- developer/general/usability-ux.rst | 420 +++++ developer/general/visual-style-guide.md | 9 - developer/general/visual-style-guide.rst | 4 + developer/general/website-style-guide.md | 78 - developer/general/website-style-guide.rst | 91 + developer/releases/1_0/release-notes.md | 50 - developer/releases/1_0/release-notes.rst | 96 + developer/releases/2_0/release-notes.md | 85 - developer/releases/2_0/release-notes.rst | 157 ++ developer/releases/3_0/release-notes.md | 62 - developer/releases/3_0/release-notes.rst | 97 + developer/releases/3_0/schedule.md | 15 - developer/releases/3_0/schedule.rst | 20 + developer/releases/3_1/release-notes.md | 65 - developer/releases/3_1/release-notes.rst | 95 + developer/releases/3_1/schedule.md | 20 - developer/releases/3_1/schedule.rst | 28 + developer/releases/3_2/release-notes.md | 61 - developer/releases/3_2/release-notes.rst | 86 + developer/releases/3_2/schedule.md | 22 - developer/releases/3_2/schedule.rst | 32 + developer/releases/4_0/release-notes.md | 107 -- developer/releases/4_0/release-notes.rst | 188 ++ developer/releases/4_0/schedule.md | 28 - developer/releases/4_0/schedule.rst | 44 + developer/releases/4_1/release-notes.md | 93 - developer/releases/4_1/release-notes.rst | 153 ++ developer/releases/4_1/schedule.md | 25 - developer/releases/4_1/schedule.rst | 38 + developer/releases/4_2/release-notes.md | 67 - developer/releases/4_2/release-notes.rst | 144 ++ developer/releases/4_2/schedule.md | 20 - developer/releases/4_2/schedule.rst | 26 + developer/releases/notes.md | 16 - developer/releases/notes.rst | 25 + developer/releases/schedules.md | 14 - developer/releases/schedules.rst | 21 + developer/releases/todo.md | 42 - developer/releases/todo.rst | 65 + developer/releases/version-scheme.md | 189 -- developer/releases/version-scheme.rst | 221 +++ developer/services/admin-api.md | 361 ---- developer/services/admin-api.rst | 845 +++++++++ .../services/disposablevm-implementation.md | 58 - .../services/disposablevm-implementation.rst | 119 ++ developer/services/dom0-secure-updates.md | 49 - developer/services/dom0-secure-updates.rst | 105 ++ developer/services/qfilecopy.md | 29 - developer/services/qfilecopy.rst | 57 + developer/services/qfileexchgd.md | 55 - developer/services/qfileexchgd.rst | 122 ++ developer/services/qmemman.md | 80 - developer/services/qmemman.rst | 175 ++ developer/services/qrexec-internals.md | 256 --- developer/services/qrexec-internals.rst | 416 +++++ developer/services/qrexec-socket-services.md | 234 --- developer/services/qrexec-socket-services.rst | 301 +++ developer/services/qrexec.md | 350 ---- developer/services/qrexec.rst | 518 ++++++ developer/services/qrexec2.md | 338 ---- developer/services/qrexec2.rst | 416 +++++ developer/system/architecture.md | 84 - developer/system/architecture.rst | 111 ++ developer/system/audio.md | 69 - developer/system/audio.rst | 112 ++ developer/system/gui.md | 462 ----- developer/system/gui.rst | 1645 +++++++++++++++++ developer/system/networking.md | 70 - developer/system/networking.rst | 201 ++ developer/system/qubes-core-admin-client.md | 8 - developer/system/qubes-core-admin.md | 8 - developer/system/qubes-core-stack.md | 8 - developer/system/security-critical-code.md | 70 - developer/system/security-critical-code.rst | 108 ++ developer/system/security-design-goals.md | 17 - developer/system/security-design-goals.rst | 26 + developer/system/system-doc.md | 13 - developer/system/template-implementation.md | 108 -- developer/system/template-implementation.rst | 236 +++ developer/system/template-manager.md | 280 --- developer/system/template-manager.rst | 443 +++++ developer/system/vm-sudo.md | 50 - developer/system/vm-sudo.rst | 63 + doc.md | 17 - .../building-archlinux-template.md | 12 - .../building-non-fedora-template.md | 12 - .../building-whonix-template.md | 10 - .../configuration-guides/change-time-zone.md | 9 - external/configuration-guides/disk-trim.md | 12 - .../configuration-guides/external-audio.md | 12 - external/configuration-guides/fetchmail.md | 12 - .../install-nvidia-driver.md | 12 - external/configuration-guides/multiboot.md | 7 - external/configuration-guides/multimedia.md | 12 - external/configuration-guides/mutt.md | 12 - .../network-bridge-support.md | 12 - .../configuration-guides/network-printer.md | 12 - external/configuration-guides/postfix.md | 12 - external/configuration-guides/rxvt.md | 12 - external/configuration-guides/vpn.md | 13 - external/configuration-guides/w3m.md | 12 - external/configuration-guides/zfs.md | 12 - external/customization-guides/dark-theme.md | 9 - .../fedora-minimal-template-customization.md | 9 - .../language-localization.md | 12 - .../removing-templatevm-packages.md | 9 - .../windows-template-customization.md | 9 - external/os-guides/centos.md | 9 - external/os-guides/gentoo.md | 9 - external/os-guides/linux-hvm-tips.md | 12 - external/os-guides/netbsd.md | 9 - external/os-guides/pentesting/blackarch.md | 10 - external/os-guides/pentesting/kali.md | 10 - external/os-guides/pentesting/ptf.md | 10 - external/os-guides/ubuntu.md | 13 - .../anonymizing-your-mac-address.md | 10 - external/privacy-guides/signal.md | 9 - external/privacy-guides/tails.md | 10 - external/privacy-guides/torvm.md | 14 - external/privacy-guides/whonix.md | 22 - .../multifactor-authentication.md | 11 - .../security-guides/security-guidelines.md | 12 - external/security-guides/split-bitcoin.md | 9 - .../application-troubleshooting.md | 9 - .../intel-igfx-troubleshooting.md | 9 - .../macbook-troubleshooting.md | 7 - .../troubleshooting/nvidia-troubleshooting.md | 10 - .../troubleshooting/sony-vaio-tinkering.md | 12 - .../troubleshooting/tails-troubleshooting.md | 9 - .../thinkpad-troubleshooting.md | 17 - index.rst | 658 +++++++ introduction/code-of-conduct.md | 125 -- introduction/code-of-conduct.rst | 151 ++ introduction/contributing.md | 60 - introduction/contributing.rst | 84 + introduction/faq.md | 809 -------- introduction/faq.rst | 1363 ++++++++++++++ introduction/getting-started.md | 256 --- introduction/getting-started.rst | 315 ++++ introduction/intro.md | 344 ---- introduction/intro.rst | 191 ++ introduction/issue-tracking.md | 233 --- introduction/issue-tracking.rst | 587 ++++++ introduction/privacy.md | 97 - introduction/privacy.rst | 96 + introduction/screenshots.md | 106 -- introduction/screenshots.rst | 249 +++ introduction/statistics.md | 59 - introduction/statistics.rst | 91 + introduction/support.md | 532 ------ introduction/support.rst | 633 +++++++ introduction/video-tours.md | 35 - introduction/video-tours.rst | 69 + project-security/security-pack.md | 240 --- project-security/security-pack.rst | 246 +++ project-security/security.md | 96 - project-security/security.rst | 105 ++ project-security/verifying-signatures.md | 908 --------- project-security/verifying-signatures.rst | 1023 ++++++++++ requirements.txt | 39 + user/advanced-topics/awesomewm.md | 38 - user/advanced-topics/awesomewm.rst | 48 + user/advanced-topics/bind-dirs.md | 114 -- user/advanced-topics/bind-dirs.rst | 196 ++ user/advanced-topics/config-files.md | 146 -- user/advanced-topics/config-files.rst | 179 ++ .../disposable-customization.md | 170 -- .../disposable-customization.rst | 273 +++ user/advanced-topics/gui-configuration.md | 57 - user/advanced-topics/gui-configuration.rst | 71 + user/advanced-topics/gui-domain.md | 155 -- user/advanced-topics/gui-domain.rst | 251 +++ .../how-to-install-software-in-dom0.md | 238 --- .../how-to-install-software-in-dom0.rst | 306 +++ user/advanced-topics/i3.md | 108 -- user/advanced-topics/i3.rst | 130 ++ .../installing-contributed-packages.md | 56 - .../installing-contributed-packages.rst | 82 + user/advanced-topics/kde.md | 86 - user/advanced-topics/kde.rst | 103 ++ user/advanced-topics/managing-vm-kernels.md | 375 ---- user/advanced-topics/managing-vm-kernels.rst | 498 +++++ user/advanced-topics/mount-from-other-os.md | 96 - user/advanced-topics/mount-from-other-os.rst | 153 ++ user/advanced-topics/qubes-service.md | 25 - user/advanced-topics/qubes-service.rst | 26 + user/advanced-topics/resize-disk-image.md | 109 -- user/advanced-topics/resize-disk-image.rst | 160 ++ user/advanced-topics/rpc-policy.md | 70 - user/advanced-topics/rpc-policy.rst | 96 + user/advanced-topics/salt.md | 628 ------- user/advanced-topics/salt.rst | 833 +++++++++ user/advanced-topics/secondary-storage.md | 114 -- user/advanced-topics/secondary-storage.rst | 159 ++ user/advanced-topics/standalones-and-hvms.md | 450 ----- user/advanced-topics/standalones-and-hvms.rst | 533 ++++++ user/advanced-topics/usb-qubes.md | 298 --- user/advanced-topics/usb-qubes.rst | 367 ++++ user/advanced-topics/volume-backup-revert.md | 50 - user/advanced-topics/volume-backup-revert.rst | 51 + .../download-mirrors.md | 27 - .../download-mirrors.rst | 36 + .../downloads.md | 11 - .../downloads.rst | 6 + .../install-security.md | 78 - .../install-security.rst | 114 ++ .../installation-guide-4.1.md | 301 --- .../installation-guide-4.1.rst | 635 +++++++ .../installation-guide.md | 302 --- .../installation-guide.rst | 612 ++++++ .../supported-releases.md | 69 - .../supported-releases.rst | 186 ++ .../testing.md | 58 - .../testing.rst | 128 ++ .../upgrade/2.md | 64 - .../upgrade/2.rst | 104 ++ .../upgrade/2b1.md | 76 - .../upgrade/2b1.rst | 120 ++ .../upgrade/2b2.md | 109 -- .../upgrade/2b2.rst | 174 ++ .../upgrade/2b3.md | 77 - .../upgrade/2b3.rst | 117 ++ .../upgrade/3_0.md | 167 -- .../upgrade/3_0.rst | 242 +++ .../upgrade/3_1.md | 124 -- .../upgrade/3_1.rst | 143 ++ .../upgrade/3_2.md | 186 -- .../upgrade/3_2.rst | 212 +++ .../upgrade/4_0.md | 118 -- .../upgrade/4_0.rst | 165 ++ .../upgrade/4_1.md | 126 -- .../upgrade/4_1.rst | 158 ++ .../upgrade/4_2.md | 118 -- .../upgrade/4_2.rst | 164 ++ .../upgrade/upgrade.md | 22 - .../upgrade/upgrade.rst | 33 + user/hardware/certified-hardware.md | 117 -- user/hardware/certified-hardware.rst | 281 +++ .../community-recommended-hardware.md | 9 - user/hardware/hcl.md | 16 - user/hardware/hcl.rst | 4 + user/hardware/how-to-use-the-hcl.md | 37 - user/hardware/how-to-use-the-hcl.rst | 69 + user/hardware/system-requirements.md | 109 -- user/hardware/system-requirements.rst | 156 ++ .../backup-emergency-restore-v2.md | 131 -- .../backup-emergency-restore-v2.rst | 166 ++ .../backup-emergency-restore-v3.md | 144 -- .../backup-emergency-restore-v3.rst | 180 ++ .../backup-emergency-restore-v4.md | 180 -- .../backup-emergency-restore-v4.rst | 247 +++ .../how-to-back-up-restore-and-migrate.md | 223 --- .../how-to-back-up-restore-and-migrate.rst | 257 +++ .../how-to-copy-and-move-files.md | 65 - .../how-to-copy-and-move-files.rst | 86 + .../how-to-copy-and-paste-text.md | 99 - .../how-to-copy-and-paste-text.rst | 149 ++ user/how-to-guides/how-to-copy-from-dom0.md | 85 - user/how-to-guides/how-to-copy-from-dom0.rst | 110 ++ .../how-to-enter-fullscreen-mode.md | 72 - .../how-to-enter-fullscreen-mode.rst | 104 ++ user/how-to-guides/how-to-install-software.md | 481 ----- .../how-to-guides/how-to-install-software.rst | 563 ++++++ .../how-to-organize-your-qubes.md | 596 ------ .../how-to-organize-your-qubes.rst | 671 +++++++ .../how-to-reinstall-a-template.md | 91 - .../how-to-reinstall-a-template.rst | 114 ++ user/how-to-guides/how-to-update.md | 88 - user/how-to-guides/how-to-update.rst | 179 ++ .../how-to-use-block-storage-devices.md | 256 --- .../how-to-use-block-storage-devices.rst | 359 ++++ user/how-to-guides/how-to-use-devices.md | 158 -- user/how-to-guides/how-to-use-devices.rst | 206 +++ user/how-to-guides/how-to-use-disposables.md | 162 -- user/how-to-guides/how-to-use-disposables.rst | 332 ++++ .../how-to-guides/how-to-use-optical-discs.md | 25 - .../how-to-use-optical-discs.rst | 29 + user/how-to-guides/how-to-use-pci-devices.md | 144 -- user/how-to-guides/how-to-use-pci-devices.rst | 218 +++ user/how-to-guides/how-to-use-usb-devices.md | 164 -- user/how-to-guides/how-to-use-usb-devices.rst | 227 +++ user/reference/glossary.md | 225 --- user/reference/glossary.rst | 295 +++ user/reference/tools.md | 31 - user/reference/tools.rst | 34 + user/security-in-qubes/anti-evil-maid.md | 61 - user/security-in-qubes/anti-evil-maid.rst | 89 + user/security-in-qubes/ctap-proxy.md | 144 -- user/security-in-qubes/ctap-proxy.rst | 274 +++ user/security-in-qubes/data-leaks.md | 56 - user/security-in-qubes/data-leaks.rst | 87 + .../device-handling-security.md | 74 - .../device-handling-security.rst | 119 ++ user/security-in-qubes/firewall.md | 473 ----- user/security-in-qubes/firewall.rst | 725 ++++++++ user/security-in-qubes/firewall_4.1.md | 513 ----- user/security-in-qubes/firewall_4.1.rst | 730 ++++++++ user/security-in-qubes/mfa.md | 393 ---- user/security-in-qubes/mfa.rst | 517 ++++++ user/security-in-qubes/split-gpg.md | 395 ---- user/security-in-qubes/split-gpg.rst | 671 +++++++ user/security-in-qubes/vm-sudo.md | 89 - user/security-in-qubes/vm-sudo.rst | 103 ++ user/templates/debian/debian-upgrade.md | 232 --- user/templates/debian/debian-upgrade.rst | 260 +++ user/templates/debian/debian.md | 111 -- user/templates/debian/debian.rst | 158 ++ user/templates/fedora/fedora-upgrade.md | 230 --- user/templates/fedora/fedora-upgrade.rst | 267 +++ user/templates/fedora/fedora.md | 47 - user/templates/fedora/fedora.rst | 73 + user/templates/minimal-templates.md | 379 ---- user/templates/minimal-templates.rst | 497 +++++ user/templates/templates.md | 436 ----- user/templates/templates.rst | 547 ++++++ user/templates/windows/migrate-to-4-1.md | 54 - user/templates/windows/migrate-to-4-1.rst | 132 ++ .../windows/qubes-windows-tools-4-0.md | 358 ---- .../windows/qubes-windows-tools-4-0.rst | 777 ++++++++ .../windows/qubes-windows-tools-4-1.md | 357 ---- .../windows/qubes-windows-tools-4-1.rst | 859 +++++++++ user/templates/windows/windows-qubes-4-0.md | 260 --- user/templates/windows/windows-qubes-4-0.rst | 401 ++++ user/templates/windows/windows-qubes-4-1.md | 272 --- user/templates/windows/windows-qubes-4-1.rst | 563 ++++++ user/templates/windows/windows.md | 19 - user/templates/windows/windows.rst | 29 + user/templates/xfce-templates.md | 48 - user/templates/xfce-templates.rst | 55 + .../app-menu-shortcut-troubleshooting.md | 121 -- .../app-menu-shortcut-troubleshooting.rst | 185 ++ .../autostart-troubleshooting.md | 36 - .../autostart-troubleshooting.rst | 47 + ...ebian-and-whonix-update-troubleshooting.md | 138 -- ...bian-and-whonix-update-troubleshooting.rst | 208 +++ user/troubleshooting/disk-troubleshooting.md | 145 -- user/troubleshooting/disk-troubleshooting.rst | 198 ++ user/troubleshooting/gui-troubleshooting.md | 76 - user/troubleshooting/gui-troubleshooting.rst | 113 ++ .../hardware-troubleshooting.md | 52 - .../hardware-troubleshooting.rst | 67 + user/troubleshooting/hvm-troubleshooting.md | 78 - user/troubleshooting/hvm-troubleshooting.rst | 112 ++ .../installation-troubleshooting.md | 125 -- .../installation-troubleshooting.rst | 200 ++ user/troubleshooting/media-troubleshooting.md | 25 - .../troubleshooting/media-troubleshooting.rst | 38 + user/troubleshooting/pci-troubleshooting.md | 154 -- user/troubleshooting/pci-troubleshooting.rst | 234 +++ .../resume-suspend-troubleshooting.md | 145 -- .../resume-suspend-troubleshooting.rst | 194 ++ user/troubleshooting/uefi-troubleshooting.md | 281 --- user/troubleshooting/uefi-troubleshooting.rst | 430 +++++ .../troubleshooting/update-troubleshooting.md | 47 - .../update-troubleshooting.rst | 67 + user/troubleshooting/usb-troubleshooting.md | 106 -- user/troubleshooting/usb-troubleshooting.rst | 176 ++ user/troubleshooting/vm-troubleshooting.md | 92 - user/troubleshooting/vm-troubleshooting.rst | 134 ++ user/troubleshooting/vpn-troubleshooting.md | 33 - user/troubleshooting/vpn-troubleshooting.rst | 48 + 412 files changed, 43831 insertions(+), 28740 deletions(-) create mode 100644 .readthedocs.yaml create mode 100644 conf.py delete mode 100644 developer/building/development-workflow.md create mode 100644 developer/building/development-workflow.rst delete mode 100644 developer/building/qubes-builder-details.md create mode 100644 developer/building/qubes-builder-details.rst delete mode 100644 developer/building/qubes-builder.md create mode 100644 developer/building/qubes-builder.rst delete mode 100644 developer/building/qubes-iso-building.md create mode 100644 developer/building/qubes-iso-building.rst delete mode 100644 developer/building/qubes-template-configs.md delete mode 100644 developer/code/code-signing.md create mode 100644 developer/code/code-signing.rst delete mode 100644 developer/code/coding-style.md create mode 100644 developer/code/coding-style.rst delete mode 100644 developer/code/license.md create mode 100644 developer/code/license.rst delete mode 100644 developer/code/source-code.md create mode 100644 developer/code/source-code.rst delete mode 100644 developer/debugging/automated-tests.md create mode 100644 developer/debugging/automated-tests.rst delete mode 100644 developer/debugging/mount-lvm-image.md create mode 100644 developer/debugging/mount-lvm-image.rst delete mode 100644 developer/debugging/safe-remote-ttys.md create mode 100644 developer/debugging/safe-remote-ttys.rst delete mode 100644 developer/debugging/test-bench.md create mode 100644 developer/debugging/test-bench.rst delete mode 100644 developer/debugging/vm-interface.md create mode 100644 developer/debugging/vm-interface.rst delete mode 100644 developer/debugging/windows-debugging.md create mode 100644 developer/debugging/windows-debugging.rst delete mode 100644 developer/general/continuous-integration.md create mode 100644 developer/general/continuous-integration.rst delete mode 100644 developer/general/devel-books.md create mode 100644 developer/general/devel-books.rst delete mode 100644 developer/general/developing-gui-applications.md create mode 100644 developer/general/developing-gui-applications.rst delete mode 100644 developer/general/documentation-style-guide.md create mode 100644 developer/general/documentation-style-guide.rst delete mode 100644 developer/general/gsoc.md create mode 100644 developer/general/gsoc.rst delete mode 100644 developer/general/gsod.md create mode 100644 developer/general/gsod.rst delete mode 100644 developer/general/how-to-edit-the-documentation.md create mode 100644 developer/general/how-to-edit-the-documentation.rst delete mode 100644 developer/general/package-contributions.md create mode 100644 developer/general/package-contributions.rst delete mode 100644 developer/general/research.md create mode 100644 developer/general/research.rst delete mode 100644 developer/general/usability-ux.md create mode 100644 developer/general/usability-ux.rst delete mode 100644 developer/general/visual-style-guide.md create mode 100644 developer/general/visual-style-guide.rst delete mode 100644 developer/general/website-style-guide.md create mode 100644 developer/general/website-style-guide.rst delete mode 100644 developer/releases/1_0/release-notes.md create mode 100644 developer/releases/1_0/release-notes.rst delete mode 100644 developer/releases/2_0/release-notes.md create mode 100644 developer/releases/2_0/release-notes.rst delete mode 100644 developer/releases/3_0/release-notes.md create mode 100644 developer/releases/3_0/release-notes.rst delete mode 100644 developer/releases/3_0/schedule.md create mode 100644 developer/releases/3_0/schedule.rst delete mode 100644 developer/releases/3_1/release-notes.md create mode 100644 developer/releases/3_1/release-notes.rst delete mode 100644 developer/releases/3_1/schedule.md create mode 100644 developer/releases/3_1/schedule.rst delete mode 100644 developer/releases/3_2/release-notes.md create mode 100644 developer/releases/3_2/release-notes.rst delete mode 100644 developer/releases/3_2/schedule.md create mode 100644 developer/releases/3_2/schedule.rst delete mode 100644 developer/releases/4_0/release-notes.md create mode 100644 developer/releases/4_0/release-notes.rst delete mode 100644 developer/releases/4_0/schedule.md create mode 100644 developer/releases/4_0/schedule.rst delete mode 100644 developer/releases/4_1/release-notes.md create mode 100644 developer/releases/4_1/release-notes.rst delete mode 100644 developer/releases/4_1/schedule.md create mode 100644 developer/releases/4_1/schedule.rst delete mode 100644 developer/releases/4_2/release-notes.md create mode 100644 developer/releases/4_2/release-notes.rst delete mode 100644 developer/releases/4_2/schedule.md create mode 100644 developer/releases/4_2/schedule.rst delete mode 100644 developer/releases/notes.md create mode 100644 developer/releases/notes.rst delete mode 100644 developer/releases/schedules.md create mode 100644 developer/releases/schedules.rst delete mode 100644 developer/releases/todo.md create mode 100644 developer/releases/todo.rst delete mode 100644 developer/releases/version-scheme.md create mode 100644 developer/releases/version-scheme.rst delete mode 100644 developer/services/admin-api.md create mode 100644 developer/services/admin-api.rst delete mode 100644 developer/services/disposablevm-implementation.md create mode 100644 developer/services/disposablevm-implementation.rst delete mode 100644 developer/services/dom0-secure-updates.md create mode 100644 developer/services/dom0-secure-updates.rst delete mode 100644 developer/services/qfilecopy.md create mode 100644 developer/services/qfilecopy.rst delete mode 100644 developer/services/qfileexchgd.md create mode 100644 developer/services/qfileexchgd.rst delete mode 100644 developer/services/qmemman.md create mode 100644 developer/services/qmemman.rst delete mode 100644 developer/services/qrexec-internals.md create mode 100644 developer/services/qrexec-internals.rst delete mode 100644 developer/services/qrexec-socket-services.md create mode 100644 developer/services/qrexec-socket-services.rst delete mode 100644 developer/services/qrexec.md create mode 100644 developer/services/qrexec.rst delete mode 100644 developer/services/qrexec2.md create mode 100644 developer/services/qrexec2.rst delete mode 100644 developer/system/architecture.md create mode 100644 developer/system/architecture.rst delete mode 100644 developer/system/audio.md create mode 100644 developer/system/audio.rst delete mode 100644 developer/system/gui.md create mode 100644 developer/system/gui.rst delete mode 100644 developer/system/networking.md create mode 100644 developer/system/networking.rst delete mode 100644 developer/system/qubes-core-admin-client.md delete mode 100644 developer/system/qubes-core-admin.md delete mode 100644 developer/system/qubes-core-stack.md delete mode 100644 developer/system/security-critical-code.md create mode 100644 developer/system/security-critical-code.rst delete mode 100644 developer/system/security-design-goals.md create mode 100644 developer/system/security-design-goals.rst delete mode 100644 developer/system/system-doc.md delete mode 100644 developer/system/template-implementation.md create mode 100644 developer/system/template-implementation.rst delete mode 100644 developer/system/template-manager.md create mode 100644 developer/system/template-manager.rst delete mode 100644 developer/system/vm-sudo.md create mode 100644 developer/system/vm-sudo.rst delete mode 100644 doc.md delete mode 100644 external/building-guides/building-archlinux-template.md delete mode 100644 external/building-guides/building-non-fedora-template.md delete mode 100644 external/building-guides/building-whonix-template.md delete mode 100644 external/configuration-guides/change-time-zone.md delete mode 100644 external/configuration-guides/disk-trim.md delete mode 100644 external/configuration-guides/external-audio.md delete mode 100644 external/configuration-guides/fetchmail.md delete mode 100644 external/configuration-guides/install-nvidia-driver.md delete mode 100644 external/configuration-guides/multiboot.md delete mode 100644 external/configuration-guides/multimedia.md delete mode 100644 external/configuration-guides/mutt.md delete mode 100644 external/configuration-guides/network-bridge-support.md delete mode 100644 external/configuration-guides/network-printer.md delete mode 100644 external/configuration-guides/postfix.md delete mode 100644 external/configuration-guides/rxvt.md delete mode 100644 external/configuration-guides/vpn.md delete mode 100644 external/configuration-guides/w3m.md delete mode 100644 external/configuration-guides/zfs.md delete mode 100644 external/customization-guides/dark-theme.md delete mode 100644 external/customization-guides/fedora-minimal-template-customization.md delete mode 100644 external/customization-guides/language-localization.md delete mode 100644 external/customization-guides/removing-templatevm-packages.md delete mode 100644 external/customization-guides/windows-template-customization.md delete mode 100644 external/os-guides/centos.md delete mode 100644 external/os-guides/gentoo.md delete mode 100644 external/os-guides/linux-hvm-tips.md delete mode 100644 external/os-guides/netbsd.md delete mode 100644 external/os-guides/pentesting/blackarch.md delete mode 100644 external/os-guides/pentesting/kali.md delete mode 100644 external/os-guides/pentesting/ptf.md delete mode 100644 external/os-guides/ubuntu.md delete mode 100644 external/privacy-guides/anonymizing-your-mac-address.md delete mode 100644 external/privacy-guides/signal.md delete mode 100644 external/privacy-guides/tails.md delete mode 100644 external/privacy-guides/torvm.md delete mode 100644 external/privacy-guides/whonix.md delete mode 100644 external/security-guides/multifactor-authentication.md delete mode 100644 external/security-guides/security-guidelines.md delete mode 100644 external/security-guides/split-bitcoin.md delete mode 100644 external/troubleshooting/application-troubleshooting.md delete mode 100644 external/troubleshooting/intel-igfx-troubleshooting.md delete mode 100644 external/troubleshooting/macbook-troubleshooting.md delete mode 100644 external/troubleshooting/nvidia-troubleshooting.md delete mode 100644 external/troubleshooting/sony-vaio-tinkering.md delete mode 100644 external/troubleshooting/tails-troubleshooting.md delete mode 100644 external/troubleshooting/thinkpad-troubleshooting.md create mode 100644 index.rst delete mode 100644 introduction/code-of-conduct.md create mode 100644 introduction/code-of-conduct.rst delete mode 100644 introduction/contributing.md create mode 100644 introduction/contributing.rst delete mode 100644 introduction/faq.md create mode 100644 introduction/faq.rst delete mode 100644 introduction/getting-started.md create mode 100644 introduction/getting-started.rst delete mode 100644 introduction/intro.md create mode 100644 introduction/intro.rst delete mode 100644 introduction/issue-tracking.md create mode 100644 introduction/issue-tracking.rst delete mode 100644 introduction/privacy.md create mode 100644 introduction/privacy.rst delete mode 100644 introduction/screenshots.md create mode 100644 introduction/screenshots.rst delete mode 100644 introduction/statistics.md create mode 100644 introduction/statistics.rst delete mode 100644 introduction/support.md create mode 100644 introduction/support.rst delete mode 100644 introduction/video-tours.md create mode 100644 introduction/video-tours.rst delete mode 100644 project-security/security-pack.md create mode 100644 project-security/security-pack.rst delete mode 100644 project-security/security.md create mode 100644 project-security/security.rst delete mode 100644 project-security/verifying-signatures.md create mode 100644 project-security/verifying-signatures.rst create mode 100644 requirements.txt delete mode 100644 user/advanced-topics/awesomewm.md create mode 100644 user/advanced-topics/awesomewm.rst delete mode 100644 user/advanced-topics/bind-dirs.md create mode 100644 user/advanced-topics/bind-dirs.rst delete mode 100644 user/advanced-topics/config-files.md create mode 100644 user/advanced-topics/config-files.rst delete mode 100644 user/advanced-topics/disposable-customization.md create mode 100644 user/advanced-topics/disposable-customization.rst delete mode 100644 user/advanced-topics/gui-configuration.md create mode 100644 user/advanced-topics/gui-configuration.rst delete mode 100644 user/advanced-topics/gui-domain.md create mode 100644 user/advanced-topics/gui-domain.rst delete mode 100644 user/advanced-topics/how-to-install-software-in-dom0.md create mode 100644 user/advanced-topics/how-to-install-software-in-dom0.rst delete mode 100644 user/advanced-topics/i3.md create mode 100644 user/advanced-topics/i3.rst delete mode 100644 user/advanced-topics/installing-contributed-packages.md create mode 100644 user/advanced-topics/installing-contributed-packages.rst delete mode 100644 user/advanced-topics/kde.md create mode 100644 user/advanced-topics/kde.rst delete mode 100644 user/advanced-topics/managing-vm-kernels.md create mode 100644 user/advanced-topics/managing-vm-kernels.rst delete mode 100644 user/advanced-topics/mount-from-other-os.md create mode 100644 user/advanced-topics/mount-from-other-os.rst delete mode 100644 user/advanced-topics/qubes-service.md create mode 100644 user/advanced-topics/qubes-service.rst delete mode 100644 user/advanced-topics/resize-disk-image.md create mode 100644 user/advanced-topics/resize-disk-image.rst delete mode 100644 user/advanced-topics/rpc-policy.md create mode 100644 user/advanced-topics/rpc-policy.rst delete mode 100644 user/advanced-topics/salt.md create mode 100644 user/advanced-topics/salt.rst delete mode 100644 user/advanced-topics/secondary-storage.md create mode 100644 user/advanced-topics/secondary-storage.rst delete mode 100644 user/advanced-topics/standalones-and-hvms.md create mode 100644 user/advanced-topics/standalones-and-hvms.rst delete mode 100644 user/advanced-topics/usb-qubes.md create mode 100644 user/advanced-topics/usb-qubes.rst delete mode 100644 user/advanced-topics/volume-backup-revert.md create mode 100644 user/advanced-topics/volume-backup-revert.rst delete mode 100644 user/downloading-installing-upgrading/download-mirrors.md create mode 100644 user/downloading-installing-upgrading/download-mirrors.rst delete mode 100644 user/downloading-installing-upgrading/downloads.md create mode 100644 user/downloading-installing-upgrading/downloads.rst delete mode 100644 user/downloading-installing-upgrading/install-security.md create mode 100644 user/downloading-installing-upgrading/install-security.rst delete mode 100644 user/downloading-installing-upgrading/installation-guide-4.1.md create mode 100644 user/downloading-installing-upgrading/installation-guide-4.1.rst delete mode 100644 user/downloading-installing-upgrading/installation-guide.md create mode 100644 user/downloading-installing-upgrading/installation-guide.rst delete mode 100644 user/downloading-installing-upgrading/supported-releases.md create mode 100644 user/downloading-installing-upgrading/supported-releases.rst delete mode 100644 user/downloading-installing-upgrading/testing.md create mode 100644 user/downloading-installing-upgrading/testing.rst delete mode 100644 user/downloading-installing-upgrading/upgrade/2.md create mode 100644 user/downloading-installing-upgrading/upgrade/2.rst delete mode 100644 user/downloading-installing-upgrading/upgrade/2b1.md create mode 100644 user/downloading-installing-upgrading/upgrade/2b1.rst delete mode 100644 user/downloading-installing-upgrading/upgrade/2b2.md create mode 100644 user/downloading-installing-upgrading/upgrade/2b2.rst delete mode 100644 user/downloading-installing-upgrading/upgrade/2b3.md create mode 100644 user/downloading-installing-upgrading/upgrade/2b3.rst delete mode 100644 user/downloading-installing-upgrading/upgrade/3_0.md create mode 100644 user/downloading-installing-upgrading/upgrade/3_0.rst delete mode 100644 user/downloading-installing-upgrading/upgrade/3_1.md create mode 100644 user/downloading-installing-upgrading/upgrade/3_1.rst delete mode 100644 user/downloading-installing-upgrading/upgrade/3_2.md create mode 100644 user/downloading-installing-upgrading/upgrade/3_2.rst delete mode 100644 user/downloading-installing-upgrading/upgrade/4_0.md create mode 100644 user/downloading-installing-upgrading/upgrade/4_0.rst delete mode 100644 user/downloading-installing-upgrading/upgrade/4_1.md create mode 100644 user/downloading-installing-upgrading/upgrade/4_1.rst delete mode 100644 user/downloading-installing-upgrading/upgrade/4_2.md create mode 100644 user/downloading-installing-upgrading/upgrade/4_2.rst delete mode 100644 user/downloading-installing-upgrading/upgrade/upgrade.md create mode 100644 user/downloading-installing-upgrading/upgrade/upgrade.rst delete mode 100644 user/hardware/certified-hardware.md create mode 100644 user/hardware/certified-hardware.rst delete mode 100644 user/hardware/community-recommended-hardware.md delete mode 100644 user/hardware/hcl.md create mode 100644 user/hardware/hcl.rst delete mode 100644 user/hardware/how-to-use-the-hcl.md create mode 100644 user/hardware/how-to-use-the-hcl.rst delete mode 100644 user/hardware/system-requirements.md create mode 100644 user/hardware/system-requirements.rst delete mode 100644 user/how-to-guides/backup-emergency-restore-v2.md create mode 100644 user/how-to-guides/backup-emergency-restore-v2.rst delete mode 100644 user/how-to-guides/backup-emergency-restore-v3.md create mode 100644 user/how-to-guides/backup-emergency-restore-v3.rst delete mode 100644 user/how-to-guides/backup-emergency-restore-v4.md create mode 100644 user/how-to-guides/backup-emergency-restore-v4.rst delete mode 100644 user/how-to-guides/how-to-back-up-restore-and-migrate.md create mode 100644 user/how-to-guides/how-to-back-up-restore-and-migrate.rst delete mode 100644 user/how-to-guides/how-to-copy-and-move-files.md create mode 100644 user/how-to-guides/how-to-copy-and-move-files.rst delete mode 100644 user/how-to-guides/how-to-copy-and-paste-text.md create mode 100644 user/how-to-guides/how-to-copy-and-paste-text.rst delete mode 100644 user/how-to-guides/how-to-copy-from-dom0.md create mode 100644 user/how-to-guides/how-to-copy-from-dom0.rst delete mode 100644 user/how-to-guides/how-to-enter-fullscreen-mode.md create mode 100644 user/how-to-guides/how-to-enter-fullscreen-mode.rst delete mode 100644 user/how-to-guides/how-to-install-software.md create mode 100644 user/how-to-guides/how-to-install-software.rst delete mode 100644 user/how-to-guides/how-to-organize-your-qubes.md create mode 100644 user/how-to-guides/how-to-organize-your-qubes.rst delete mode 100644 user/how-to-guides/how-to-reinstall-a-template.md create mode 100644 user/how-to-guides/how-to-reinstall-a-template.rst delete mode 100644 user/how-to-guides/how-to-update.md create mode 100644 user/how-to-guides/how-to-update.rst delete mode 100644 user/how-to-guides/how-to-use-block-storage-devices.md create mode 100644 user/how-to-guides/how-to-use-block-storage-devices.rst delete mode 100644 user/how-to-guides/how-to-use-devices.md create mode 100644 user/how-to-guides/how-to-use-devices.rst delete mode 100644 user/how-to-guides/how-to-use-disposables.md create mode 100644 user/how-to-guides/how-to-use-disposables.rst delete mode 100644 user/how-to-guides/how-to-use-optical-discs.md create mode 100644 user/how-to-guides/how-to-use-optical-discs.rst delete mode 100644 user/how-to-guides/how-to-use-pci-devices.md create mode 100644 user/how-to-guides/how-to-use-pci-devices.rst delete mode 100644 user/how-to-guides/how-to-use-usb-devices.md create mode 100644 user/how-to-guides/how-to-use-usb-devices.rst delete mode 100644 user/reference/glossary.md create mode 100644 user/reference/glossary.rst delete mode 100644 user/reference/tools.md create mode 100644 user/reference/tools.rst delete mode 100644 user/security-in-qubes/anti-evil-maid.md create mode 100644 user/security-in-qubes/anti-evil-maid.rst delete mode 100644 user/security-in-qubes/ctap-proxy.md create mode 100644 user/security-in-qubes/ctap-proxy.rst delete mode 100644 user/security-in-qubes/data-leaks.md create mode 100644 user/security-in-qubes/data-leaks.rst delete mode 100644 user/security-in-qubes/device-handling-security.md create mode 100644 user/security-in-qubes/device-handling-security.rst delete mode 100644 user/security-in-qubes/firewall.md create mode 100644 user/security-in-qubes/firewall.rst delete mode 100644 user/security-in-qubes/firewall_4.1.md create mode 100644 user/security-in-qubes/firewall_4.1.rst delete mode 100644 user/security-in-qubes/mfa.md create mode 100644 user/security-in-qubes/mfa.rst delete mode 100644 user/security-in-qubes/split-gpg.md create mode 100644 user/security-in-qubes/split-gpg.rst delete mode 100644 user/security-in-qubes/vm-sudo.md create mode 100644 user/security-in-qubes/vm-sudo.rst delete mode 100644 user/templates/debian/debian-upgrade.md create mode 100644 user/templates/debian/debian-upgrade.rst delete mode 100644 user/templates/debian/debian.md create mode 100644 user/templates/debian/debian.rst delete mode 100644 user/templates/fedora/fedora-upgrade.md create mode 100644 user/templates/fedora/fedora-upgrade.rst delete mode 100644 user/templates/fedora/fedora.md create mode 100644 user/templates/fedora/fedora.rst delete mode 100644 user/templates/minimal-templates.md create mode 100644 user/templates/minimal-templates.rst delete mode 100644 user/templates/templates.md create mode 100644 user/templates/templates.rst delete mode 100644 user/templates/windows/migrate-to-4-1.md create mode 100644 user/templates/windows/migrate-to-4-1.rst delete mode 100644 user/templates/windows/qubes-windows-tools-4-0.md create mode 100644 user/templates/windows/qubes-windows-tools-4-0.rst delete mode 100644 user/templates/windows/qubes-windows-tools-4-1.md create mode 100644 user/templates/windows/qubes-windows-tools-4-1.rst delete mode 100644 user/templates/windows/windows-qubes-4-0.md create mode 100644 user/templates/windows/windows-qubes-4-0.rst delete mode 100644 user/templates/windows/windows-qubes-4-1.md create mode 100644 user/templates/windows/windows-qubes-4-1.rst delete mode 100644 user/templates/windows/windows.md create mode 100644 user/templates/windows/windows.rst delete mode 100644 user/templates/xfce-templates.md create mode 100644 user/templates/xfce-templates.rst delete mode 100644 user/troubleshooting/app-menu-shortcut-troubleshooting.md create mode 100644 user/troubleshooting/app-menu-shortcut-troubleshooting.rst delete mode 100644 user/troubleshooting/autostart-troubleshooting.md create mode 100644 user/troubleshooting/autostart-troubleshooting.rst delete mode 100644 user/troubleshooting/debian-and-whonix-update-troubleshooting.md create mode 100644 user/troubleshooting/debian-and-whonix-update-troubleshooting.rst delete mode 100644 user/troubleshooting/disk-troubleshooting.md create mode 100644 user/troubleshooting/disk-troubleshooting.rst delete mode 100644 user/troubleshooting/gui-troubleshooting.md create mode 100644 user/troubleshooting/gui-troubleshooting.rst delete mode 100644 user/troubleshooting/hardware-troubleshooting.md create mode 100644 user/troubleshooting/hardware-troubleshooting.rst delete mode 100644 user/troubleshooting/hvm-troubleshooting.md create mode 100644 user/troubleshooting/hvm-troubleshooting.rst delete mode 100644 user/troubleshooting/installation-troubleshooting.md create mode 100644 user/troubleshooting/installation-troubleshooting.rst delete mode 100644 user/troubleshooting/media-troubleshooting.md create mode 100644 user/troubleshooting/media-troubleshooting.rst delete mode 100644 user/troubleshooting/pci-troubleshooting.md create mode 100644 user/troubleshooting/pci-troubleshooting.rst delete mode 100644 user/troubleshooting/resume-suspend-troubleshooting.md create mode 100644 user/troubleshooting/resume-suspend-troubleshooting.rst delete mode 100644 user/troubleshooting/uefi-troubleshooting.md create mode 100644 user/troubleshooting/uefi-troubleshooting.rst delete mode 100644 user/troubleshooting/update-troubleshooting.md create mode 100644 user/troubleshooting/update-troubleshooting.rst delete mode 100644 user/troubleshooting/usb-troubleshooting.md create mode 100644 user/troubleshooting/usb-troubleshooting.rst delete mode 100644 user/troubleshooting/vm-troubleshooting.md create mode 100644 user/troubleshooting/vm-troubleshooting.rst delete mode 100644 user/troubleshooting/vpn-troubleshooting.md create mode 100644 user/troubleshooting/vpn-troubleshooting.rst diff --git a/.readthedocs.yaml b/.readthedocs.yaml new file mode 100644 index 0000000000..f952c7afa1 --- /dev/null +++ b/.readthedocs.yaml @@ -0,0 +1,25 @@ +version: 2 + +build: + os: "ubuntu-22.04" + tools: + python: "3.10" + +submodules: + include: all + recursive: true + +sphinx: + builder: html + configuration: conf.py + fail_on_warning: false + +python: + install: + - requirements: requirements.txt + +formats: +- pdf +- epub + + diff --git a/_dev/index.rst b/_dev/index.rst index 39488eb68f..cc737d8ada 100644 --- a/_dev/index.rst +++ b/_dev/index.rst @@ -1,9 +1,26 @@ +============================================== Welcome to Qubes OS developer's documentation! ============================================== + This is documentation for the source code. Please choose specific repostitory: -* `core-admin `_ -* `core-admin-client `_ +.. toctree:: + :maxdepth: 1 + + core-admin <> +.. _core-admin: /projects/core-admin + + + + core-admin-client <> +.. _core-admin-client: /projects/core-admin-client + + + + + +Or see the main Qubes OS documentation +.. _the main qubes os documentation: https://www.qubes-os.org/doc/ -Or see `the main Qubes OS documentation `_. +. diff --git a/conf.py b/conf.py new file mode 100644 index 0000000000..383076a97a --- /dev/null +++ b/conf.py @@ -0,0 +1,131 @@ +""" + ReST directive for embedding Youtube and Vimeo videos. + There are two directives added: ``youtube`` and ``vimeo``. The only + argument is the video id of the video to include. + Both directives have three optional arguments: ``height``, ``width`` + and ``align``. Default height is 281 and default width is 500. + Example:: + .. youtube:: anwy2MPT5RE + :height: 315 + :width: 560 + :align: left + :copyright: (c) 2012 by Danilo Bargen. + :license: BSD 3-clause +""" +from __future__ import absolute_import +from docutils import nodes +from docutils.parsers.rst import Directive, directives + +# -- Project information ----------------------------------------------------- + +project = 'Qubes OS' +copyright = '2024, Qubes OS Project' +author = 'Qubes OS Project' + +title = "Qubes Docs" +html_title = "Qubes Docs" + +# The full version, including alpha/beta/rc tags +release = '4.2' + +# -- General configuration --------------------------------------------------- + +html_static_path = ['attachment/doc'] +extensions = [ + 'sphinx.ext.autosectionlabel', + 'sphinxnotes.strike', + 'sphinx_reredirects' +] + +redirects = { + "user/hardware/hcl": "https://www.qubes-os.org/hcl/", + "user/downloading-installing-upgrading/downloads:mirrors":"https://www.qubes-os.org/downloads/mirrors/", + "developer/general/visual-style-guide": "https://www.qubes-os.org/doc/visual-style-guide/", + "developer/general/website-style-guide":"https://www.qubes-os.org/doc/website-style-guide/", + "user/downloading-installing-upgrading/downloads":"https://www.qubes-os.org/downloads/", + "developer/general/how-to-edit-the-documentation":"https://www.qubes-os.org/doc/how-to-edit-the-documentation/" +} + +autosectionlabel_prefix_document = True + +source_suffix = { + '.rst': 'restructuredtext', +} + +root_doc = "index" +exclude_patterns = [ + '_dev/*', + 'attachment/*', + '**/*.txt' +] + +html_theme = 'sphinx_rtd_theme' +# html_theme = 'default' +# html_theme = 'classic' + +html_theme_options = { + 'externalrefs': True, + 'bgcolor': 'white', + 'linkcolor': '#99bfff', + 'textcolor': '#000000', + 'visitedlinkcolor': '#7b7b7b', + 'bodyfont': '"Open Sans", Arial, sans-serif', + 'codebgcolor': 'grey', + 'body_min_width': '50%', + 'body_max_width': '90%', + 'collapse_navigation': True, +} + +gettext_uuid = True +gettext_compact = False + +# epub_show_urls = 'footnote' +# latex_show_urls ='footnote' + + +locale_dirs = ['_translated'] + +# from https://gist.github.com/ehles/bed012d78aad5d3cd6c35a49bef32f9f +def align(argument): + """Conversion function for the "align" option.""" + return directives.choice(argument, ('left', 'center', 'right')) + + +class IframeVideo(Directive): + has_content = False + required_arguments = 1 + optional_arguments = 0 + final_argument_whitespace = False + option_spec = { + 'height': directives.nonnegative_int, + 'width': directives.nonnegative_int, + 'align': align, + } + default_width = 500 + default_height = 281 + + def run(self): + self.options['video_id'] = directives.uri(self.arguments[0]) + if not self.options.get('width'): + self.options['width'] = self.default_width + if not self.options.get('height'): + self.options['height'] = self.default_height + if not self.options.get('align'): + self.options['align'] = 'left' + return [nodes.raw('', self.html % self.options, format='html')] + + +class GeneralVid(IframeVideo): + html = '' + + +class Youtube(IframeVideo): + html = '' + + +def setup(builder): + directives.register_directive('youtube', Youtube) + directives.register_directive('generalvid', GeneralVid) diff --git a/developer/building/development-workflow.md b/developer/building/development-workflow.md deleted file mode 100644 index dd72699aba..0000000000 --- a/developer/building/development-workflow.md +++ /dev/null @@ -1,538 +0,0 @@ ---- -lang: en -layout: doc -permalink: /doc/development-workflow/ -redirect_from: -- /en/doc/development-workflow/ -- /doc/DevelopmentWorkflow/ -- /wiki/DevelopmentWorkflow/ -ref: 66 -title: Development workflow ---- - -A workflow for developing Qubes OS+ - -First things first, setup [QubesBuilder](/doc/qubes-builder/). This guide -assumes you're using qubes-builder to build Qubes. - -## Repositories and committing Code - -Qubes is split into a bunch of git repos. These are all contained in the -`qubes-src` directory under qubes-builder. Subdirectories there are separate -components, stored in separate git repositories. - -The best way to write and contribute code is to create a git repo somewhere -(e.g., github) for the repo you are interested in editing (e.g., -`qubes-manager`, `core-agent-linux`, etc). To integrate your repo with the rest -of Qubes, cd to the repo directory and add your repository as a remote in git - -**Example:** - -~~~ -$ cd qubes-builder/qubes-src/qubes-manager -$ git remote add abel git@github.com:abeluck/qubes-manager.git -~~~ - -You can then proceed to easily develop in your own branches, pull in new -commits from the dev branches, merge them, and eventually push to your own repo -on github. - -When you are ready to submit your changes to Qubes to be merged, push your -changes, then create a signed git tag (using `git tag -s`). Finally, send a -letter to the Qubes listserv describing the changes and including the link to -your repository. You can also create pull request on github. Don't forget to -include your public PGP key you use to sign your tags. - -### Kernel-specific notes - -#### Prepare fresh version of kernel sources, with Qubes-specific patches applied - -In qubes-builder/qubes-src/linux-kernel: - -~~~ -make prep -~~~ - -The resulting tree will be in kernel-\/linux-\: - -~~~ -ls -ltrd kernel*/linux* -~~~ - -~~~ -drwxr-xr-x 23 user user 4096 Nov 5 09:50 kernel-3.4.18/linux-3.4.18 -drwxr-xr-x 6 user user 4096 Nov 21 20:48 kernel-3.4.18/linux-obj -~~~ - -#### Go to the kernel tree and update the version - -In qubes-builder/qubes-src/linux-kernel: - -~~~ -cd kernel-3.4.18/linux-3.4.18 -~~~ - -#### Changing the config - -In kernel-3.4.18/linux-3.4.18: - -~~~ -cp ../../config .config -make oldconfig -~~~ - -Now change the configuration. For example, in kernel-3.4.18/linux-3.4.18: - -~~~ -make menuconfig -~~~ - -Copy the modified config back into the kernel tree: - -~~~ -cp .config ../../../config -~~~ - -#### Patching the code - -TODO: describe the workflow for patching the code, below are some random notes, not working well - -~~~ -ln -s ../../patches.xen -export QUILT_PATCHES=patches.xen -export QUILT_REFRESH_ARGS="-p ab --no-timestamps --no-index" -export QUILT_SERIES=../../series-pvops.conf - -quilt new patches.xen/pvops-3.4-0101-usb-xen-pvusb-driver-bugfix.patch -quilt add drivers/usb/host/Kconfig drivers/usb/host/Makefile \ - drivers/usb/host/xen-usbback/* drivers/usb/host/xen-usbfront.c \ - include/xen/interface/io/usbif.h - -*edit something* - -quilt refresh -cd ../.. -vi series.conf -~~~ - -#### Building RPMs - -TODO: Is this step generic for all subsystems? - -Now it is a good moment to make sure you have changed kernel release name in -rel file. For example, if you change it to '1debug201211116c' the -resulting RPMs will be named -'kernel-3.4.18-1debug20121116c.pvops.qubes.x86\_64.rpm'. This will help -distinguish between different versions of the same package. - -You might want to take a moment here to review (git diff, git status), commit -your changes locally. - -To actually build RPMs, in qubes-builder: - -~~~ -make linux-kernel -~~~ - -RPMs will appear in qubes-src/linux-kernel/pkgs/fc20/x86\_64: - -~~~ --rw-rw-r-- 1 user user 42996126 Nov 17 04:08 kernel-3.4.18-1debug20121116c.pvops.qubes.x86_64.rpm --rw-rw-r-- 1 user user 43001450 Nov 17 05:36 kernel-3.4.18-1debug20121117a.pvops.qubes.x86_64.rpm --rw-rw-r-- 1 user user 8940138 Nov 17 04:08 kernel-devel-3.4.18-1debug20121116c.pvops.qubes.x86_64.rpm --rw-rw-r-- 1 user user 8937818 Nov 17 05:36 kernel-devel-3.4.18-1debug20121117a.pvops.qubes.x86_64.rpm --rw-rw-r-- 1 user user 54490741 Nov 17 04:08 kernel-qubes-vm-3.4.18-1debug20121116c.pvops.qubes.x86_64.rpm --rw-rw-r-- 1 user user 54502117 Nov 17 05:37 kernel-qubes-vm-3.4.18-1debug20121117a.pvops.qubes.x86_64.rpm -~~~ - -### Useful [QubesBuilder](/doc/qubes-builder/) commands - -1. `make check` - will check if all the code was committed into repository and -if all repository are tagged with signed tag. -2. `make show-vtags` - show version of each component (based on git tags) - -mostly useful just before building ISO. **Note:** this will not show version -for components containing changes since last version tag. -3. `make push` - push change from **all** repositories to git server. You must -set proper remotes (see above) for all repositories first. -4. `make prepare-merge` - fetch changes from remote repositories (can be -specified on commandline via GIT\_SUBDIR or GIT\_REMOTE vars), (optionally) -verify tags and show the changes. This do not merge the changes - there are -left for review as FETCH\_HEAD ref. You can merge them using `git merge -FETCH_HEAD` (in each repo directory). Or `make do-merge` to merge all of them. - -## Copying Code to dom0 - -When developing it is convenient to be able to rapidly test changes. Assuming -you're developing Qubes on Qubes, you should be working in a special VM for -Qubes and occasionally you will want to transfer code or RPMs back to dom0 for -testing. - -Here are some handy scripts Marek has shared to facilitate this. - -You may also like to run your [test environment on separate -machine](/doc/test-bench/). - -### Syncing dom0 files - -TODO: edit this script to be more generic - -~~~ -#!/bin/sh - -set -x -set -e - -QUBES_PY_DIR=/usr/lib64/python2.6/site-packages/qubes -QUBES_PY=$QUBES_PY_DIR/qubes.py -QUBESUTILS_PY=$QUBES_PY_DIR/qubesutils.py - -qvm-run -p qubes-devel 'cd qubes-builder/qubes-src/core/dom0; tar c qmemman/qmemman*.py qvm-core/*.py qvm-tools/* misc/vm-template-hvm.conf misc/qubes-start.desktop ../misc/block-snapshot aux-tools ../qrexec' |tar xv -cp $QUBES_PY qubes.py.bak$$ -cp $QUBESUTILS_PY qubesutils.py.bak$$ -cp /etc/xen/scripts/block-snapshot block-snapshot.bak$$ -sudo cp qvm-core/qubes.py $QUBES_PY -sudo cp qvm-core/qubesutils.py $QUBESUTILS_PY -sudo cp qvm-core/guihelpers.py $QUBES_PY_DIR/ -sudo cp qmemman/qmemman*.py $QUBES_PY_DIR/ -sudo cp misc/vm-template-hvm.conf /usr/share/qubes/ -sudo cp misc/qubes-start.desktop /usr/share/qubes/ -sudo cp misc/block-snapshot /etc/xen/scripts/ -sudo cp aux-tools/qubes-dom0-updates.cron /etc/cron.daily/ -# FIXME(Abel Luck): I hope to -~~~ - -### Apply qvm-tools - -TODO: make it more generic - -~~~ -#!/bin/sh - -BAK=qvm-tools.bak$$ -mkdir -p $BAK -cp -a /usr/bin/qvm-* /usr/bin/qubes-* $BAK/ -sudo cp qvm-tools/qvm-* qvm-tools/qubes-* /usr/bin/ -~~~ - -### Copy from dom0 to an appvm - -~~~ -#/bin/sh -# -# usage ./cp-domain -# -domain=$1 -file=$2 -fname=`basename $file` - -qvm-run $domain 'mkdir /home/user/incoming/dom0 -p' -cat $file| qvm-run --pass-io $domain "cat > /home/user/incoming/dom0/$fname" -~~~ - -## Git connection between VMs - -Sometimes it's useful to transfer git commits between VMs. You can use `git -format-patch` for that and simply copy the files. But you can also setup -custom qrexec service for it. - -Below example assumes that you use `builder-RX` directory in target VM to -store sources in qubes-builder layout (where `X` is some number). Make sure that -all the scripts are executable. - -Service file (save in `/usr/local/etc/qubes-rpc/local.Git` in target VM): - -~~~ -#!/bin/sh - -exec 2>/tmp/log2 - -read service rel repo -echo "Params: $service $rel $repo" >&2 -# Adjust regexps if needed -echo "$repo" | grep -q '^[A-Za-z0-9-]\+$' || exit 1 -echo "$rel" | grep -q '^[0-9.]\+$' || exit 1 -path="/home/user/builder-R$rel/qubes-src/$repo" -if [ "$repo" = "builder" ]; then - path="/home/user/builder-R$rel" -fi -case $service in - git-receive-pack|git-upload-pack) - echo "starting $service $path" >&2 - exec $service $path - ;; - *) - echo "Unsupported service: $service" >&2 - ;; -esac -~~~ - -Client script (save in `~/bin/git-qrexec` in source VM): - -~~~ -#!/bin/sh - -VMNAME=$1 - -(echo $GIT_EXT_SERVICE $2 $3; exec cat) | qrexec-client-vm $VMNAME local.Git -~~~ - -You will also need to setup qrexec policy in dom0 (`/etc/qubes-rpc/policy/local.Git`). - -Usage: - -~~~ -[user@source core-agent-linux]$ git remote add testbuilder "ext::git-qrexec testbuilder 3 core-agent-linux" -[user@source core-agent-linux]$ git push testbuilder master -~~~ - -You can create `~/bin/add-remote` script to ease adding remotes: - -~~~ -#!/bin/sh - -[ -n "$1" ] || exit 1 - -if [ "$1" = "tb" ]; then - git remote add $1 "ext::git-qrexec testbuilder 3 `basename $PWD`" - exit $? -fi - -git remote add $1 git@github.com:$1/qubes-`basename $PWD` -~~~ - -It should be executed from component top level directory. This script takes one -argument - remote name. If it is `tb`, then it creates qrexec-based git remote -to `testbuilder` VM. Otherwise it creates remote pointing at github account of -the same name. In any case it points at repository matching current directory -name. - -## Sending packages to different VM - -Other useful script(s) can be used to setup local package repository hosted in -some VM. This way you can keep your development VM behind firewall, while -having an option to expose some yum/apt repository to the local network (to -have them installed on test machine). - -To achieve this goal, a dummy repository can be created, which instead of -populating metadata locally, will upload the packages to some other VM and -trigger repository update there (using qrexec). You can use `unstable` -repository flavor, because there is no release managing rules bundled (unlike -current and current-testing). - -### RPM packages - yum repo - -In source VM, grab [linux-yum](https://github.com/QubesOS/qubes-linux-yum) repository (below is assumed you've made it in -`~/repo-yum-upload` directory) and replace `update_repo.sh` script with: - -~~~ -#!/bin/sh - -VMNAME=repo-vm - -set -e -qvm-copy-to-vm $VMNAME $1 -# remove only files, leave directory structure -find -type f -name '*.rpm' -delete -# trigger repo update -qrexec-client-vm $VMNAME local.UpdateYum -~~~ - -In target VM, setup actual yum repository (also based on [linux-yum](https://github.com/QubesOS/qubes-linux-yum), this time -without modifications). You will also need to setup some gpg key for signing -packages (it is possible to force yum to install unsigned packages, but it -isn't possible for `qubes-dom0-update` tool). Fill `~/.rpmmacros` with -key description: - -~~~ -%_gpg_name Test packages signing key -~~~ - -Then setup `local.UpdateYum` qrexec service (`/usr/local/etc/qubes-rpc/local.UpdateYum`): - -~~~ -#!/bin/sh - -if [ -z "$QREXEC_REMOTE_DOMAIN" ]; then - exit 1 -fi - -real_repository=/home/user/linux-yum -incoming=/home/user/QubesIncoming/$QREXEC_REMOTE_DOMAIN - -find $incoming -name '*.rpm' |xargs rpm -K |grep -iv pgp |cut -f1 -d: |xargs -r setsid -w rpm --addsign 2>&1 - -rsync -lr --remove-source-files $incoming/ $real_repository -cd $real_repository -export SKIP_REPO_CHECK=1 -if [ -d $incoming/r3.1 ]; then - ./update_repo-unstable.sh r3.1 -fi - -if [ -d $incoming/r3.0 ]; then - ./update_repo-unstable.sh r3.0 -fi - -if [ -d $incoming/r2 ]; then - ./update_repo-unstable.sh r2 -fi -find $incoming -type d -empty -delete -exit 0 -~~~ - -Of course you will also need to setup qrexec policy in dom0 -`/etc/qubes-rpc/policy/local.UpdateYum`. - -If you want to access the repository from network, you need to setup HTTP -server serving it, and configure the system to let other machines actually -reach this HTTP server. You can use for example using [port -forwarding](/doc/firewall/#port-forwarding-to-a-qube-from-the-outside-world) or setting up Tor hidden service. Configuration -details of those services are outside of the scope of this page. - -Usage: setup `builder.conf` in source VM to use your dummy-uploader repository: - -~~~ -LINUX_REPO_BASEDIR = ../../repo-yum-upload/r3.1 -~~~ - -Then use `make update-repo-unstable` to upload the packages. You can also -specify selected components on command line, then build them and upload to the -repository: - -~~~ -make COMPONENTS="core-agent-linux gui-agent-linux linux-utils" qubes update-repo-unstable -~~~ - -On the test machine, add yum repository (`/etc/yum.repos.d`) pointing at just -configured HTTP server. For example: - -~~~ -[local-test] -name=Test -baseurl=http://local-test.lan/linux-yum/r$releasever/unstable/dom0/fc20 -~~~ - -Remember to also import gpg public key using `rpm --import`. - -### Deb packages - Apt repo - -Steps are mostly the same as in the case of yum repo. The only details that differ: - -- use [linux-deb](https://github.com/QubesOS/qubes-linux-deb) instead of [linux-yum](https://github.com/QubesOS/qubes-linux-yum) as a base - both in source and target VM -- use different `update_repo.sh` script in source VM (below) -- use `local.UpdateApt` qrexec service in target VM (code below) -- in target VM additionally place `update-local-repo.sh` script in repository dir (code below) - -`update_repo.sh` script: - -~~~ -#!/bin/sh - -set -e - -current_release=$1 -VMNAME=repo-vm - -qvm-copy-to-vm $VMNAME $1 -find $current_release -type f -name '*.deb' -delete -rm -f $current_release/vm/db/* -qrexec-client-vm $VMNAME local.UpdateApt -~~~ - -`local.UpdateApt` service code (`/usr/local/etc/qubes-rpc/local.UpdateApt` in repo-serving VM): - -~~~ -#!/bin/sh - -if [ -z "$QREXEC_REMOTE_DOMAIN" ]; then - exit 1 -fi - -incoming=/home/user/QubesIncoming/$QREXEC_REMOTE_DOMAIN - -rsync -lr --remove-source-files $incoming/ /home/user/linux-deb/ -cd /home/user/linux-deb -export SKIP_REPO_CHECK=1 -if [ -d $incoming/r3.1 ]; then - for dist in `ls r3.1/vm/dists`; do - ./update-local-repo.sh r3.1/vm $dist - done -fi - -if [ -d $incoming/r3.0 ]; then - for dist in `ls r3.0/vm/dists`; do - ./update-local-repo.sh r3.0/vm $dist - done -fi - -if [ -d $incoming/r2 ]; then - for dist in `ls r2/vm/dists`; do - ./update-local-repo.sh r2/vm $dist - done -fi -find $incoming -type d -empty -delete -exit 0 -~~~ - -`update-local-repo.sh`: - -~~~ -#!/bin/sh - -set -e - -# Set this to your local repository signing key -SIGN_KEY=01ABCDEF - -[ -z "$1" ] && { echo "Usage: $0 "; exit 1; } - -REPO_DIR=$1 -DIST=$2 - -if [ "$DIST" = "wheezy-unstable" ]; then - DIST_TAG=deb7 -elif [ "$DIST" = "jessie-unstable" ]; then - DIST_TAG=deb8 -elif [ "$DIST" = "stretch-unstable" ]; then - DIST_TAG=deb9 -fi - -pushd $REPO_DIR -mkdir -p dists/$DIST/main/binary-amd64 -dpkg-scanpackages --multiversion --arch "*$DIST_TAG*" . > dists/$DIST/main/binary-amd64/Packages -gzip -9c dists/$DIST/main/binary-amd64/Packages > dists/$DIST/main/binary-amd64/Packages.gz -cat > dists/$DIST/Release <> dists/$DIST/Release - -rm -f $DIST/Release.gpg -rm -f $DIST/InRelease -gpg -abs -u "$SIGN_KEY" \ - < dists/$DIST/Release > dists/$DIST/Release.gpg -gpg -a -s --clearsign -u "$SIGN_KEY" \ - < dists/$DIST/Release > dists/$DIST/InRelease -popd - -if [ `id -u` -eq 0 ]; then - chown -R --reference=$REPO_DIR $REPO_DIR -fi -~~~ - -Usage: add this line to `/etc/apt/sources.list` on test machine (adjust host and path): - -~~~ -deb http://local-test.lan/linux-deb/r3.1 jessie-unstable main -~~~ diff --git a/developer/building/development-workflow.rst b/developer/building/development-workflow.rst new file mode 100644 index 0000000000..c8d9ecccf0 --- /dev/null +++ b/developer/building/development-workflow.rst @@ -0,0 +1,647 @@ +==================== +Development workflow +==================== + + +A workflow for developing Qubes OS+ + +First things first, setup :doc:`QubesBuilder `. This +guide assumes you’re using qubes-builder to build Qubes. + +Repositories and committing Code +-------------------------------- + + +Qubes is split into a bunch of git repos. These are all contained in the +``qubes-src`` directory under qubes-builder. Subdirectories there are +separate components, stored in separate git repositories. + +The best way to write and contribute code is to create a git repo +somewhere (e.g., github) for the repo you are interested in editing +(e.g., ``qubes-manager``, ``core-agent-linux``, etc). To integrate your +repo with the rest of Qubes, cd to the repo directory and add your +repository as a remote in git + +**Example:** + +.. code:: bash + + $ cd qubes-builder/qubes-src/qubes-manager + $ git remote add abel git@github.com:abeluck/qubes-manager.git + + + +You can then proceed to easily develop in your own branches, pull in new +commits from the dev branches, merge them, and eventually push to your +own repo on github. + +When you are ready to submit your changes to Qubes to be merged, push +your changes, then create a signed git tag (using ``git tag -s``). +Finally, send a letter to the Qubes listserv describing the changes and +including the link to your repository. You can also create pull request +on github. Don’t forget to include your public PGP key you use to sign +your tags. + +Kernel-specific notes +^^^^^^^^^^^^^^^^^^^^^ + + +Prepare fresh version of kernel sources, with Qubes-specific patches applied +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + + +In qubes-builder/qubes-src/linux-kernel: + +.. code:: bash + + make prep + + + +The resulting tree will be in kernel-/linux-: + +.. code:: bash + + ls -ltrd kernel*/linux* + + + +.. code:: bash + + drwxr-xr-x 23 user user 4096 Nov 5 09:50 kernel-3.4.18/linux-3.4.18 + drwxr-xr-x 6 user user 4096 Nov 21 20:48 kernel-3.4.18/linux-obj + + + +Go to the kernel tree and update the version +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + + +In qubes-builder/qubes-src/linux-kernel: + +.. code:: bash + + cd kernel-3.4.18/linux-3.4.18 + + + +Changing the config +^^^^^^^^^^^^^^^^^^^ + + +In kernel-3.4.18/linux-3.4.18: + +.. code:: bash + + cp ../../config .config + make oldconfig + + + +Now change the configuration. For example, in +kernel-3.4.18/linux-3.4.18: + +.. code:: bash + + make menuconfig + + + +Copy the modified config back into the kernel tree: + +.. code:: bash + + cp .config ../../../config + + + +Patching the code +^^^^^^^^^^^^^^^^^ + + +TODO: describe the workflow for patching the code, below are some random +notes, not working well + +.. code:: bash + + ln -s ../../patches.xen + export QUILT_PATCHES=patches.xen + export QUILT_REFRESH_ARGS="-p ab --no-timestamps --no-index" + export QUILT_SERIES=../../series-pvops.conf + + quilt new patches.xen/pvops-3.4-0101-usb-xen-pvusb-driver-bugfix.patch + quilt add drivers/usb/host/Kconfig drivers/usb/host/Makefile \ + drivers/usb/host/xen-usbback/* drivers/usb/host/xen-usbfront.c \ + include/xen/interface/io/usbif.h + + *edit something* + + quilt refresh + cd ../.. + vi series.conf + + + +Building RPMs +^^^^^^^^^^^^^ + + +TODO: Is this step generic for all subsystems? + +Now it is a good moment to make sure you have changed kernel release +name in rel file. For example, if you change it to ‘1debug201211116c’ +the resulting RPMs will be named +‘kernel-3.4.18-1debug20121116c.pvops.qubes.x86_64.rpm’. This will help +distinguish between different versions of the same package. + +You might want to take a moment here to review (git diff, git status), +commit your changes locally. + +To actually build RPMs, in qubes-builder: + +.. code:: bash + + make linux-kernel + + + +RPMs will appear in qubes-src/linux-kernel/pkgs/fc20/x86_64: + +.. code:: bash + + -rw-rw-r-- 1 user user 42996126 Nov 17 04:08 kernel-3.4.18-1debug20121116c.pvops.qubes.x86_64.rpm + -rw-rw-r-- 1 user user 43001450 Nov 17 05:36 kernel-3.4.18-1debug20121117a.pvops.qubes.x86_64.rpm + -rw-rw-r-- 1 user user 8940138 Nov 17 04:08 kernel-devel-3.4.18-1debug20121116c.pvops.qubes.x86_64.rpm + -rw-rw-r-- 1 user user 8937818 Nov 17 05:36 kernel-devel-3.4.18-1debug20121117a.pvops.qubes.x86_64.rpm + -rw-rw-r-- 1 user user 54490741 Nov 17 04:08 kernel-qubes-vm-3.4.18-1debug20121116c.pvops.qubes.x86_64.rpm + -rw-rw-r-- 1 user user 54502117 Nov 17 05:37 kernel-qubes-vm-3.4.18-1debug20121117a.pvops.qubes.x86_64.rpm + + + +Useful :doc:`QubesBuilder ` commands +^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + + +1. ``make check`` - will check if all the code was committed into + repository and if all repository are tagged with signed tag. + +2. ``make show-vtags`` - show version of each component (based on git + tags) - mostly useful just before building ISO. **Note:** this will + not show version for components containing changes since last version + tag. + +3. ``make push`` - push change from **all** repositories to git server. + You must set proper remotes (see above) for all repositories first. + +4. ``make prepare-merge`` - fetch changes from remote repositories (can + be specified on commandline via GIT_SUBDIR or GIT_REMOTE vars), + (optionally) verify tags and show the changes. This do not merge the + changes - there are left for review as FETCH_HEAD ref. You can merge + them using ``git merge FETCH_HEAD`` (in each repo directory). Or + ``make do-merge`` to merge all of them. + + + +Copying Code to dom0 +-------------------- + + +When developing it is convenient to be able to rapidly test changes. +Assuming you’re developing Qubes on Qubes, you should be working in a +special VM for Qubes and occasionally you will want to transfer code or +RPMs back to dom0 for testing. + +Here are some handy scripts Marek has shared to facilitate this. + +You may also like to run your :doc:`test environment on separate machine `. + +Syncing dom0 files +^^^^^^^^^^^^^^^^^^ + + +TODO: edit this script to be more generic + +.. code:: bash + + #!/bin/sh + + set -x + set -e + + QUBES_PY_DIR=/usr/lib64/python2.6/site-packages/qubes + QUBES_PY=$QUBES_PY_DIR/qubes.py + QUBESUTILS_PY=$QUBES_PY_DIR/qubesutils.py + + qvm-run -p qubes-devel 'cd qubes-builder/qubes-src/core/dom0; tar c qmemman/qmemman*.py qvm-core/*.py qvm-tools/* misc/vm-template-hvm.conf misc/qubes-start.desktop ../misc/block-snapshot aux-tools ../qrexec' |tar xv + cp $QUBES_PY qubes.py.bak$$ + cp $QUBESUTILS_PY qubesutils.py.bak$$ + cp /etc/xen/scripts/block-snapshot block-snapshot.bak$$ + sudo cp qvm-core/qubes.py $QUBES_PY + sudo cp qvm-core/qubesutils.py $QUBESUTILS_PY + sudo cp qvm-core/guihelpers.py $QUBES_PY_DIR/ + sudo cp qmemman/qmemman*.py $QUBES_PY_DIR/ + sudo cp misc/vm-template-hvm.conf /usr/share/qubes/ + sudo cp misc/qubes-start.desktop /usr/share/qubes/ + sudo cp misc/block-snapshot /etc/xen/scripts/ + sudo cp aux-tools/qubes-dom0-updates.cron /etc/cron.daily/ + # FIXME(Abel Luck): I hope to + + + +Apply qvm-tools +^^^^^^^^^^^^^^^ + + +TODO: make it more generic + +.. code:: bash + + #!/bin/sh + + BAK=qvm-tools.bak$$ + mkdir -p $BAK + cp -a /usr/bin/qvm-* /usr/bin/qubes-* $BAK/ + sudo cp qvm-tools/qvm-* qvm-tools/qubes-* /usr/bin/ + + + +Copy from dom0 to an appvm +^^^^^^^^^^^^^^^^^^^^^^^^^^ + + +.. code:: bash + + #/bin/sh + # + # usage ./cp-domain + # + domain=$1 + file=$2 + fname=`basename $file` + + qvm-run $domain 'mkdir /home/user/incoming/dom0 -p' + cat $file| qvm-run --pass-io $domain "cat > /home/user/incoming/dom0/$fname" + + + +Git connection between VMs +-------------------------- + + +Sometimes it’s useful to transfer git commits between VMs. You can use +``git format-patch`` for that and simply copy the files. But you can +also setup custom qrexec service for it. + +Below example assumes that you use ``builder-RX`` directory in target VM +to store sources in qubes-builder layout (where ``X`` is some number). +Make sure that all the scripts are executable. + +Service file (save in ``/usr/local/etc/qubes-rpc/local.Git`` in target +VM): + +.. code:: bash + + #!/bin/sh + + exec 2>/tmp/log2 + + read service rel repo + echo "Params: $service $rel $repo" >&2 + # Adjust regexps if needed + echo "$repo" | grep -q '^[A-Za-z0-9-]\+$' || exit 1 + echo "$rel" | grep -q '^[0-9.]\+$' || exit 1 + path="/home/user/builder-R$rel/qubes-src/$repo" + if [ "$repo" = "builder" ]; then + path="/home/user/builder-R$rel" + fi + case $service in + git-receive-pack|git-upload-pack) + echo "starting $service $path" >&2 + exec $service $path + ;; + *) + echo "Unsupported service: $service" >&2 + ;; + esac + + + +Client script (save in ``~/bin/git-qrexec`` in source VM): + +.. code:: bash + + #!/bin/sh + + VMNAME=$1 + + (echo $GIT_EXT_SERVICE $2 $3; exec cat) | qrexec-client-vm $VMNAME local.Git + + + +You will also need to setup qrexec policy in dom0 +(``/etc/qubes-rpc/policy/local.Git``). + +Usage: + +.. code:: bash + + [user@source core-agent-linux]$ git remote add testbuilder "ext::git-qrexec testbuilder 3 core-agent-linux" + [user@source core-agent-linux]$ git push testbuilder master + + + +You can create ``~/bin/add-remote`` script to ease adding remotes: + +.. code:: bash + + #!/bin/sh + + [ -n "$1" ] || exit 1 + + if [ "$1" = "tb" ]; then + git remote add $1 "ext::git-qrexec testbuilder 3 `basename $PWD`" + exit $? + fi + + git remote add $1 git@github.com:$1/qubes-`basename $PWD` + + + +It should be executed from component top level directory. This script +takes one argument - remote name. If it is ``tb``, then it creates +qrexec-based git remote to ``testbuilder`` VM. Otherwise it creates +remote pointing at github account of the same name. In any case it +points at repository matching current directory name. + +Sending packages to different VM +-------------------------------- + + +Other useful script(s) can be used to setup local package repository +hosted in some VM. This way you can keep your development VM behind +firewall, while having an option to expose some yum/apt repository to +the local network (to have them installed on test machine). + +To achieve this goal, a dummy repository can be created, which instead +of populating metadata locally, will upload the packages to some other +VM and trigger repository update there (using qrexec). You can use +``unstable`` repository flavor, because there is no release managing +rules bundled (unlike current and current-testing). + +RPM packages - yum repo +^^^^^^^^^^^^^^^^^^^^^^^ + + +In source VM, grab +`linux-yum `__ repository +(below is assumed you’ve made it in ``~/repo-yum-upload`` directory) and +replace ``update_repo.sh`` script with: + +.. code:: bash + + #!/bin/sh + + VMNAME=repo-vm + + set -e + qvm-copy-to-vm $VMNAME $1 + # remove only files, leave directory structure + find -type f -name '*.rpm' -delete + # trigger repo update + qrexec-client-vm $VMNAME local.UpdateYum + + + +In target VM, setup actual yum repository (also based on +`linux-yum `__, this time +without modifications). You will also need to setup some gpg key for +signing packages (it is possible to force yum to install unsigned +packages, but it isn’t possible for ``qubes-dom0-update`` tool). Fill +``~/.rpmmacros`` with key description: + +.. code:: bash + + %_gpg_name Test packages signing key + + + +Then setup ``local.UpdateYum`` qrexec service +(``/usr/local/etc/qubes-rpc/local.UpdateYum``): + +.. code:: bash + + #!/bin/sh + + if [ -z "$QREXEC_REMOTE_DOMAIN" ]; then + exit 1 + fi + + real_repository=/home/user/linux-yum + incoming=/home/user/QubesIncoming/$QREXEC_REMOTE_DOMAIN + + find $incoming -name '*.rpm' |xargs rpm -K |grep -iv pgp |cut -f1 -d: |xargs -r setsid -w rpm --addsign 2>&1 + + rsync -lr --remove-source-files $incoming/ $real_repository + cd $real_repository + export SKIP_REPO_CHECK=1 + if [ -d $incoming/r3.1 ]; then + ./update_repo-unstable.sh r3.1 + fi + + if [ -d $incoming/r3.0 ]; then + ./update_repo-unstable.sh r3.0 + fi + + if [ -d $incoming/r2 ]; then + ./update_repo-unstable.sh r2 + fi + find $incoming -type d -empty -delete + exit 0 + + + +Of course you will also need to setup qrexec policy in dom0 +``/etc/qubes-rpc/policy/local.UpdateYum``. + +If you want to access the repository from network, you need to setup +HTTP server serving it, and configure the system to let other machines +actually reach this HTTP server. You can use for example using :ref:`port forwarding ` +or setting up Tor hidden service. Configuration details of those +services are outside of the scope of this page. + +Usage: setup ``builder.conf`` in source VM to use your dummy-uploader +repository: + +.. code:: bash + + LINUX_REPO_BASEDIR = ../../repo-yum-upload/r3.1 + + + +Then use ``make update-repo-unstable`` to upload the packages. You can +also specify selected components on command line, then build them and +upload to the repository: + +.. code:: bash + + make COMPONENTS="core-agent-linux gui-agent-linux linux-utils" qubes update-repo-unstable + + + +On the test machine, add yum repository (``/etc/yum.repos.d``) pointing +at just configured HTTP server. For example: + +.. code:: bash + + [local-test] + name=Test + baseurl=http://local-test.lan/linux-yum/r$releasever/unstable/dom0/fc20 + + + +Remember to also import gpg public key using ``rpm --import``. + +Deb packages - Apt repo +^^^^^^^^^^^^^^^^^^^^^^^ + + +Steps are mostly the same as in the case of yum repo. The only details +that differ: + +- use `linux-deb `__ + instead of `linux-yum `__ + as a base - both in source and target VM + +- use different ``update_repo.sh`` script in source VM (below) + +- use ``local.UpdateApt`` qrexec service in target VM (code below) + +- in target VM additionally place ``update-local-repo.sh`` script in + repository dir (code below) + + + +``update_repo.sh`` script: + +.. code:: bash + + #!/bin/sh + + set -e + + current_release=$1 + VMNAME=repo-vm + + qvm-copy-to-vm $VMNAME $1 + find $current_release -type f -name '*.deb' -delete + rm -f $current_release/vm/db/* + qrexec-client-vm $VMNAME local.UpdateApt + + + +``local.UpdateApt`` service code +(``/usr/local/etc/qubes-rpc/local.UpdateApt`` in repo-serving VM): + +.. code:: bash + + #!/bin/sh + + if [ -z "$QREXEC_REMOTE_DOMAIN" ]; then + exit 1 + fi + + incoming=/home/user/QubesIncoming/$QREXEC_REMOTE_DOMAIN + + rsync -lr --remove-source-files $incoming/ /home/user/linux-deb/ + cd /home/user/linux-deb + export SKIP_REPO_CHECK=1 + if [ -d $incoming/r3.1 ]; then + for dist in `ls r3.1/vm/dists`; do + ./update-local-repo.sh r3.1/vm $dist + done + fi + + if [ -d $incoming/r3.0 ]; then + for dist in `ls r3.0/vm/dists`; do + ./update-local-repo.sh r3.0/vm $dist + done + fi + + if [ -d $incoming/r2 ]; then + for dist in `ls r2/vm/dists`; do + ./update-local-repo.sh r2/vm $dist + done + fi + find $incoming -type d -empty -delete + exit 0 + + + +``update-local-repo.sh``: + +.. code:: bash + + #!/bin/sh + + set -e + + # Set this to your local repository signing key + SIGN_KEY=01ABCDEF + + [ -z "$1" ] && { echo "Usage: $0 "; exit 1; } + + REPO_DIR=$1 + DIST=$2 + + if [ "$DIST" = "wheezy-unstable" ]; then + DIST_TAG=deb7 + elif [ "$DIST" = "jessie-unstable" ]; then + DIST_TAG=deb8 + elif [ "$DIST" = "stretch-unstable" ]; then + DIST_TAG=deb9 + fi + + pushd $REPO_DIR + mkdir -p dists/$DIST/main/binary-amd64 + dpkg-scanpackages --multiversion --arch "*$DIST_TAG*" . > dists/$DIST/main/binary-amd64/Packages + gzip -9c dists/$DIST/main/binary-amd64/Packages > dists/$DIST/main/binary-amd64/Packages.gz + cat > dists/$DIST/Release <> dists/$DIST/Release + + rm -f $DIST/Release.gpg + rm -f $DIST/InRelease + gpg -abs -u "$SIGN_KEY" \ + < dists/$DIST/Release > dists/$DIST/Release.gpg + gpg -a -s --clearsign -u "$SIGN_KEY" \ + < dists/$DIST/Release > dists/$DIST/InRelease + popd + + if [ `id -u` -eq 0 ]; then + chown -R --reference=$REPO_DIR $REPO_DIR + fi + + + +Usage: add this line to ``/etc/apt/sources.list`` on test machine +(adjust host and path): + +.. code:: bash + + deb http://local-test.lan/linux-deb/r3.1 jessie-unstable main + + diff --git a/developer/building/qubes-builder-details.md b/developer/building/qubes-builder-details.md deleted file mode 100644 index 1f5e9f3f2a..0000000000 --- a/developer/building/qubes-builder-details.md +++ /dev/null @@ -1,59 +0,0 @@ ---- -lang: en -layout: doc -permalink: /doc/qubes-builder-details/ -redirect_from: -- /en/doc/qubes-builder-details/ -- /doc/QubesBuilderDetails/ -- /wiki/QubesBuilderDetails/ -ref: 65 -title: Qubes builder details ---- - - - - -Components Makefile.builder file --------------------------------- - -[QubesBuilder](/doc/qubes-builder/) expects that each component have *Makefile.builder* file in its root directory. This file specifies what should be done to build the package. As name suggests, this is normal makefile, which is included by builder as its configuration. Its main purpose is to set some variables. Generally all available variables/settings are described as comments at the beginning of Makefile.\* in [QubesBuilder](/doc/qubes-builder/). - -Variables for Linux build: - -- `RPM_SPEC_FILES` List (space separated) of spec files for RPM package build. Path should be relative to component root directory. [QubesBuilder](/doc/qubes-builder/) will install all BuildRequires (in chroot environment) before the build. In most Qubes components all spec files are kept in *rpm\_spec* directory. This is mainly used for Fedora packages build. -- `ARCH_BUILD_DIRS` List (space separated) of directories with PKGBUILD files for Archlinux package build. Similar to RPM build, [QubesBuilder](/doc/qubes-builder/) will install all makedepends, then build the package. - -Most components uses *archlinux* directory for this purpose, so its good to keep this style. - -Variables for Windows build: - -- `WIN_COMPILER` Choose which compiler should be used for this component, thus which build scripts. Currently two options available: - - `WDK` - Windows Driver Kit (default). Command used to build: *build -cZg*. - - `mingw` - MinGW (Windows gcc port). Command used to build: *make all* -- `WIN_SOURCE_SUBDIRS` List of directories in which above command should be run. In most cases it will be only one entry: current directory (*.*). -- `WIN_PREBUILD_CMD` Command to run before build, mostly useful for WDK build (in mingw case, you can use makefile for this purpose). Can be used to set some variables, preprocess some files etc. -- `WIN_SIGN_CMD` Command used to sign resulting binaries. Note that default value is *sign.bat*. If you don't want to sign binaries, specify some placeholder here (eg. *true*). Check existing components (eg. vmm-xen-windows-pvdrivers) for example scripts. This command will be run with certain environment variables: - - `CERT_FILENAME` Path to key file (pfx format) - - `CERT_PASSWORD` Key password - - `CERT_PUBLIC_FILENAME` Certificate path in the case of self-signed cert - - `CERT_CROSS_CERT_FILENAME` Certificate path in the case of correct autheticode cert - - `SIGNTOOL` Path to signtool -- `WIN_PACKAGE_CMD` Command used to produce installation package (msi or msm). Default value is *wix.bat*, similar to above - use *true* if you don't want this command. -- `WIN_OUTPUT_HEADERS` Directory (relative to `WIN_SOURCE_SUBDIRS` element) with public headers of the package - for use in other components. -- `WIN_OUTPUT_LIBS` Directory (relative to `WIN_SOURCE_SUBDIRS` element) with libraries (both DLL and implib) of the package - for use in other components. Note that [QubesBuilder](/doc/qubes-builder/) will copy files specified as *\$(WIN\_OUTPUT\_LIBS)/\*/\** to match WDK directory layout (*\/\/\*), so you in mingw build you need to place libraries in some additional subdirectory. -- `WIN_BUILD_DEPS` List of components required to build this one. [QubesBuilder](/doc/qubes-builder/) will copy files specified with `WIN_OUTPUT_HEADERS` and `WIN_OUTPUT_LIBS` of those components to some directory and provide its path with `QUBES_INCLUDES` and `QUBES_LIBS` variables. Use those variables in your build scripts (*sources* or *Makefile* - depending on selected compiler). You can assume that the variables are always set and directories always exists, even if empty. - -builder.conf settings ---------------------- - -Most settings are documented in *builder.conf.default* file, which can be used as template the actual configuration. - -**TODO** - -Notes ------ - -* For a list of custom TemplateVMs available in QubesBuilder look at [Supported Versions page](/doc/supported-releases/). diff --git a/developer/building/qubes-builder-details.rst b/developer/building/qubes-builder-details.rst new file mode 100644 index 0000000000..ea2d96553e --- /dev/null +++ b/developer/building/qubes-builder-details.rst @@ -0,0 +1,126 @@ +===================== +Qubes builder details +===================== + + +.. warning:: + Note: This information concerns the old Qubes builder (v1). It + supports only building Qubes 4.1 or earlier.The build process has + been completely rewritten in qubes-builder v2. This can be used for + building Qubes R4.1 and later, and all related components. + + +Components Makefile.builder file +-------------------------------- + + +:doc:`QubesBuilder ` expects that each component have +*Makefile.builder* file in its root directory. This file specifies what +should be done to build the package. As name suggests, this is normal +makefile, which is included by builder as its configuration. Its main +purpose is to set some variables. Generally all available +variables/settings are described as comments at the beginning of +Makefile.* in :doc:`QubesBuilder `. + +Variables for Linux build: + +- ``RPM_SPEC_FILES`` List (space separated) of spec files for RPM + package build. Path should be relative to component root directory. + :doc:`QubesBuilder ` will install all BuildRequires + (in chroot environment) before the build. In most Qubes components + all spec files are kept in *rpm_spec* directory. This is mainly used + for Fedora packages build. + +- ``ARCH_BUILD_DIRS`` List (space separated) of directories with + PKGBUILD files for Archlinux package build. Similar to RPM build, + :doc:`QubesBuilder ` will install all makedepends, + then build the package. + + + +Most components uses *archlinux* directory for this purpose, so its good +to keep this style. + +Variables for Windows build: + +- ``WIN_COMPILER`` Choose which compiler should be used for this + component, thus which build scripts. Currently two options available: + + - ``WDK`` - Windows Driver Kit (default). Command used to build: + *build -cZg*. + + - ``mingw`` - MinGW (Windows gcc port). Command used to build: *make all* + + + +- ``WIN_SOURCE_SUBDIRS`` List of directories in which above command + should be run. In most cases it will be only one entry: current + directory (*.*). + +- ``WIN_PREBUILD_CMD`` Command to run before build, mostly useful for + WDK build (in mingw case, you can use makefile for this purpose). Can + be used to set some variables, preprocess some files etc. + +- ``WIN_SIGN_CMD`` Command used to sign resulting binaries. Note that + default value is *sign.bat*. If you don’t want to sign binaries, + specify some placeholder here (eg. *true*). Check existing components + (eg. vmm-xen-windows-pvdrivers) for example scripts. This command + will be run with certain environment variables: + + - ``CERT_FILENAME`` Path to key file (pfx format) + + - ``CERT_PASSWORD`` Key password + + - ``CERT_PUBLIC_FILENAME`` Certificate path in the case of + self-signed cert + + - ``CERT_CROSS_CERT_FILENAME`` Certificate path in the case of + correct autheticode cert + + - ``SIGNTOOL`` Path to signtool + + + +- ``WIN_PACKAGE_CMD`` Command used to produce installation package (msi + or msm). Default value is *wix.bat*, similar to above - use *true* if + you don’t want this command. + +- ``WIN_OUTPUT_HEADERS`` Directory (relative to ``WIN_SOURCE_SUBDIRS`` + element) with public headers of the package - for use in other + components. + +- ``WIN_OUTPUT_LIBS`` Directory (relative to ``WIN_SOURCE_SUBDIRS`` + element) with libraries (both DLL and implib) of the package - for + use in other components. Note that + :doc:`QubesBuilder ` will copy files specified as + *$(WIN_OUTPUT_LIBS)/*/** to match WDK directory layout (*//*), so you in mingw + build you need to place libraries in some additional subdirectory. + +- ``WIN_BUILD_DEPS`` List of components required to build this one. + :doc:`QubesBuilder ` will copy files specified with + ``WIN_OUTPUT_HEADERS`` and ``WIN_OUTPUT_LIBS`` of those components to + some directory and provide its path with ``QUBES_INCLUDES`` and + ``QUBES_LIBS`` variables. Use those variables in your build scripts + (*sources* or *Makefile* - depending on selected compiler). You can + assume that the variables are always set and directories always + exists, even if empty. + + + +builder.conf settings +--------------------- + + +Most settings are documented in *builder.conf.default* file, which can +be used as template the actual configuration. + +**TODO** + +Notes +----- + + +- For a list of custom TemplateVMs available in QubesBuilder look at + :doc:`Supported Versions page `. + + diff --git a/developer/building/qubes-builder.md b/developer/building/qubes-builder.md deleted file mode 100644 index 31755705be..0000000000 --- a/developer/building/qubes-builder.md +++ /dev/null @@ -1,218 +0,0 @@ ---- -lang: en -layout: doc -permalink: /doc/qubes-builder/ -redirect_from: -- /en/doc/qubes-builder/ -- /doc/QubesBuilder/ -- /wiki/QubesBuilder/ -ref: 64 -title: Qubes builder ---- - - - -**Note: See [ISO building instructions](/doc/qubes-iso-building/) for a streamlined overview on how to use the build system.** - - -We have a fully automated build system for Qubes, that downloads, builds and -packages all the Qubes components, and finally should spit out a ready-to-use -installation ISO, all in a [secure](/news/2016/05/30/build-security/) way. - -In order to use it, you should use an rpm-based distro, like Fedora :), and should ensure the following packages are installed: - -- sudo -- gnupg -- git -- createrepo -- rpm-build -- dnf-plugins-core -- make -- wget -- rpmdevtools -- python3-sh -- dialog -- rpm-sign -- dpkg-dev -- debootstrap -- python3-pyyaml -- devscripts -- perl-Digest-MD5 -- perl-Digest-SHA - -Usually you can install those packages by just issuing: - -```shell -sudo dnf install gnupg git createrepo rpm-build make wget rpmdevtools python3-sh dialog rpm-sign dpkg-dev debootstrap python3-pyyaml devscripts perl-Digest-MD5 perl-Digest-SHA -``` - -The build system creates build environments in chroots and so no other packages are needed on the host. -All files created by the build system are contained within the qubes-builder directory. -The full build requires some 25GB of free space, so keep that in mind when deciding where to place this directory. - -The build system is configured via builder.conf file. -You can use the setup.sh script to create and modify this file. -Alternatively, you can copy the provided default builder.conf, and modify it as needed, e.g.: - -```shell -cp example-configs/qubes-os-master.conf builder.conf -# edit the builder.conf file and set the following variables: -NO_SIGN=1 -``` - -One additional useful requirement is that 'sudo root' must work without any prompt, which is default on most distros (e.g. 'sudo bash' brings you the root shell without asking for any password). -This is important as the builder needs to switch to root and then back to user several times during the build process. - -Additionally, if building with signing enabled (NO\_SIGN is not set), you must adjust `\~/.rpmmacros` file so that it points to the GPG key used for package signing, e.g.: - -```bash -%_signature gpg -%_gpg_path /home/user/.gnupg -%_gpg_name AC1BF9B3 # <-- Key ID used for signing -``` - -It is also recommended that you use an empty passphrase for the private key used for signing. -Contrary to a popular belief, this doesn't affect your key or sources security -- if somebody compromised your system, then the game is over anyway, whether you have used an additional passphrase for the key or not. - -So, to build Qubes you would do: - -```shell -# Import the Qubes master key -gpg --recv-keys 0xDDFA1A3E36879494 - -# Verify its fingerprint, set as 'trusted'. -# This is described here: -# https://www.qubes-os.org/doc/VerifyingSignatures - -wget https://keys.qubes-os.org/keys/qubes-developers-keys.asc -gpg --import qubes-developers-keys.asc - -git clone https://github.com/QubesOS/qubes-builder.git qubes-builder -cd qubes-builder - -# Verify its integrity: -git tag -v `git describe` - -cp example-configs/qubes-os-master.conf builder.conf -# edit the builder.conf file and set the following variables: -# NO_SIGN="1" - -# Download all components: - -make get-sources - -# And now to build all Qubes RPMs (this will take a few hours): - -make qubes - -# ... and then to build the ISO - -make iso -``` - -And this should produce a shiny new ISO. - -You can also build selected component separately. Eg. to compile only gui virtualization agent/daemon: - -```shell -make gui-daemon -``` - -You can get a full list from make help. - -## Making customized build - -### Manual source modification - -You can also modify sources somehow if you wish. -Here are some basic steps: - -1. Download qubes-builder as described above (if you want to use marmarek's branches, you should also download qubes-builder from his repo - replace 'QubesOS' with 'marmarek' in above git clone command) -2. Edit builder.conf (still the same as above), some useful additions: - - You can also set GIT\_PREFIX="marmarek/qubes-" to use marmarek's repo instead of "mainstream" - it contains newer (but less tested) versions -3. Download unmodified sources - - ```shell - make get-sources - ``` - -4. **Make your modifications here** - -5. Build the Qubes - `make qubes` actually is just meta target which builds all required - components in correct order. The list of components is configured in - builder.conf. You can also check the current value at the end of `make - help`, or using `make build-info`. - -6. `get-sources` is already done, so continue with the next one. You can skip `sign-all` if you've disabled signing - - ```shell - make vmm-xen core-admin linux-kernel gui-daemon template desktop-linux-kde installer-qubes-os manager linux-dom0-updates - ``` - -7. build iso installation image - - ```shell - make iso - ``` - -### Use pre-built Qubes packages - -For building just a few selected packages, it's very useful to download pre-built qubes-specific dependencies from `{yum,deb}.qubes-os.org`. -This is especially true for `gcc`, which takes several hours to build. - -Before creating the `chroot`, add this to your `builder.conf`: - -``` -USE_QUBES_REPO_VERSION = $(RELEASE) -``` - -It will add the 'current' Qubes repository to your `chroot` environment. -Next, specify which components (`gcc`, for example) you want to download instead of compiling: - -``` -COMPONENTS := $(filter-out gcc,$(COMPONENTS)) -``` - -Alternatively, edit the actual COMPONENTS list which is defined in the included version-dependent config from example-configs (see series of include directives near the beginning of `builder.conf`). -This way, you can build only the packages in which you are interested. - -If you also want to use the 'current-testing' repository, add this to your configuration: - -``` -USE_QUBES_REPO_TESTING = 1 -``` - -In the case of an existing `chroot`, for mock-enabled builds, this works immediately because `chroot` is constructed each time separately. -For legacy builds, it will not add the necessary configuration into the build environment unless a specific builder change or configuration would force rebuilding chroot. - -Also, once enabled, disabling this setting will not disable repositories in relevant chroots. -And even if it did, there could be some leftover packages installed from those repos (which may or may not be desirable). - -**Note** -If you are building Ubuntu templates, you cannot use this option. -This is because Qubes does not provide official packages for Ubuntu templates. - -## Code verification keys management - -[QubesBuilder](/doc/qubes-builder/) by default verifies signed tags on every downloaded code. -Public keys used for that are stored in `keyrings/git`. -By default Qubes developers' keys are imported automatically, but if you need some additional keys (for example your own), you can add them using: - -```shell -GNUPGHOME=$PWD/keyrings/git gpg --import /path/to/key.asc -GNUPGHOME=$PWD/keyrings/git gpg --edit-key ID_OF_JUST_IMPORTED_KEY -# here use "trust" command to set key fully or ultimately trusted - only those keys are accepted by QubesBuilder -``` - -All Qubes developers' keys are signed by the Qubes Master Signing Key (which is set as ultimately trusted key), so are trusted automatically. - -If you are the owner of Master key and want to revoke such signature, use the `revsig` gpg key edit command and update the key in qubes-developers-keys.asc - now the key will be no longer trusted (unless manually set as such). - -## Further information - -For advanced [QubesBuilder](/doc/qubes-builder/) use, and preparing sources, take a look at [QubesBuilderDetails](/doc/qubes-builder-details/) page, or [QubesBuilder's doc directory](https://github.com/marmarek/qubes-builder/tree/master/doc). diff --git a/developer/building/qubes-builder.rst b/developer/building/qubes-builder.rst new file mode 100644 index 0000000000..b489148262 --- /dev/null +++ b/developer/building/qubes-builder.rst @@ -0,0 +1,303 @@ +============= +Qubes builder +============= + + +.. warning:: + Note: These instructions concern the older Qubes builder (v1). It + supports only building Qubes 4.1 or earlier.The build process has + been completely rewritten in qubes-builder v2. This can be used for + building Qubes R4.1 and later, and all related components. + + +**Note: See** :doc:`ISO building instructions ` +**for a streamlined overview on how to use the build system.** + +We have a fully automated build system for Qubes, that downloads, builds +and packages all the Qubes components, and finally should spit out a +ready-to-use installation ISO, all in a +`secure `__ way. + +In order to use it, you should use an rpm-based distro, like Fedora :), +and should ensure the following packages are installed: + +- sudo + +- gnupg + +- git + +- createrepo + +- rpm-build + +- dnf-plugins-core + +- make + +- wget + +- rpmdevtools + +- python3-sh + +- dialog + +- rpm-sign + +- dpkg-dev + +- debootstrap + +- python3-pyyaml + +- devscripts + +- perl-Digest-MD5 + +- perl-Digest-SHA + + + +Usually you can install those packages by just issuing: + +.. code:: bash + + sudo dnf install gnupg git createrepo rpm-build make wget rpmdevtools python3-sh dialog rpm-sign dpkg-dev debootstrap python3-pyyaml devscripts perl-Digest-MD5 perl-Digest-SHA + + +The build system creates build environments in chroots and so no other +packages are needed on the host. All files created by the build system +are contained within the qubes-builder directory. The full build +requires some 25GB of free space, so keep that in mind when deciding +where to place this directory. + +The build system is configured via builder.conf file. You can use the +setup.sh script to create and modify this file. Alternatively, you can +copy the provided default builder.conf, and modify it as needed, e.g.: + +.. code:: bash + + cp example-configs/qubes-os-master.conf builder.conf + # edit the builder.conf file and set the following variables: + NO_SIGN=1 + + +One additional useful requirement is that ‘sudo root’ must work without +any prompt, which is default on most distros (e.g. ‘sudo bash’ brings +you the root shell without asking for any password). This is important +as the builder needs to switch to root and then back to user several +times during the build process. + +Additionally, if building with signing enabled (NO_SIGN is not set), you +must adjust ``\~/.rpmmacros`` file so that it points to the GPG key used +for package signing, e.g.: + +.. code:: bash + + %_signature gpg + %_gpg_path /home/user/.gnupg + %_gpg_name AC1BF9B3 # <-- Key ID used for signing + + +It is also recommended that you use an empty passphrase for the private +key used for signing. Contrary to a popular belief, this doesn’t affect +your key or sources security – if somebody compromised your system, then +the game is over anyway, whether you have used an additional passphrase +for the key or not. + +So, to build Qubes you would do: + +.. code:: bash + + # Import the Qubes master key + gpg --recv-keys 0xDDFA1A3E36879494 + + # Verify its fingerprint, set as 'trusted'. + # This is described here: + # https://www.qubes-os.org/doc/VerifyingSignatures + + wget https://keys.qubes-os.org/keys/qubes-developers-keys.asc + gpg --import qubes-developers-keys.asc + + git clone https://github.com/QubesOS/qubes-builder.git qubes-builder + cd qubes-builder + + # Verify its integrity: + git tag -v `git describe` + + cp example-configs/qubes-os-master.conf builder.conf + # edit the builder.conf file and set the following variables: + # NO_SIGN="1" + + # Download all components: + + make get-sources + + # And now to build all Qubes RPMs (this will take a few hours): + + make qubes + + # ... and then to build the ISO + + make iso + + +And this should produce a shiny new ISO. + +You can also build selected component separately. Eg. to compile only +gui virtualization agent/daemon: + +.. code:: bash + + make gui-daemon + + +You can get a full list from make help. + +Making customized build +----------------------- + + +Manual source modification +^^^^^^^^^^^^^^^^^^^^^^^^^^ + + +You can also modify sources somehow if you wish. Here are some basic +steps: + +1. Download qubes-builder as described above (if you want to use + marmarek’s branches, you should also download qubes-builder from his + repo - replace ‘QubesOS’ with ‘marmarek’ in above git clone command) + +2. Edit builder.conf (still the same as above), some useful additions: + + + +- You can also set GIT_PREFIX=“marmarek/qubes-” to use marmarek’s repo + instead of “mainstream” - it contains newer (but less tested) + versions + + + +3. Download unmodified sources + + .. code:: bash + + make get-sources + + +4. **Make your modifications here** + +5. Build the Qubes ``make qubes`` actually is just meta target which + builds all required components in correct order. The list of + components is configured in builder.conf. You can also check the + current value at the end of ``make help``, or using + ``make build-info``. + +6. ``get-sources`` is already done, so continue with the next one. You + can skip ``sign-all`` if you’ve disabled signing + + .. code:: bash + + make vmm-xen core-admin linux-kernel gui-daemon template desktop-linux-kde installer-qubes-os manager linux-dom0-updates + + +7. build iso installation image + + .. code:: bash + + make iso + + + + + + +Use pre-built Qubes packages +^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + + +For building just a few selected packages, it’s very useful to download +pre-built qubes-specific dependencies from ``{yum,deb}.qubes-os.org``. +This is especially true for ``gcc``, which takes several hours to build. + +Before creating the ``chroot``, add this to your ``builder.conf``: + +.. code:: bash + + USE_QUBES_REPO_VERSION = $(RELEASE) + + + +It will add the ‘current’ Qubes repository to your ``chroot`` +environment. Next, specify which components (``gcc``, for example) you +want to download instead of compiling: + +.. code:: bash + + COMPONENTS := $(filter-out gcc,$(COMPONENTS)) + + + +Alternatively, edit the actual COMPONENTS list which is defined in the +included version-dependent config from example-configs (see series of +include directives near the beginning of ``builder.conf``). This way, +you can build only the packages in which you are interested. + +If you also want to use the ‘current-testing’ repository, add this to +your configuration: + +.. code:: bash + + USE_QUBES_REPO_TESTING = 1 + + + +In the case of an existing ``chroot``, for mock-enabled builds, this +works immediately because ``chroot`` is constructed each time +separately. For legacy builds, it will not add the necessary +configuration into the build environment unless a specific builder +change or configuration would force rebuilding chroot. + +Also, once enabled, disabling this setting will not disable repositories +in relevant chroots. And even if it did, there could be some leftover +packages installed from those repos (which may or may not be desirable). + +**Note** If you are building Ubuntu templates, you cannot use this +option. This is because Qubes does not provide official packages for +Ubuntu templates. + +Code verification keys management +--------------------------------- + + +:doc:`QubesBuilder ` by default verifies signed tags +on every downloaded code. Public keys used for that are stored in +``keyrings/git``. By default Qubes developers’ keys are imported +automatically, but if you need some additional keys (for example your +own), you can add them using: + +.. code:: bash + + GNUPGHOME=$PWD/keyrings/git gpg --import /path/to/key.asc + GNUPGHOME=$PWD/keyrings/git gpg --edit-key ID_OF_JUST_IMPORTED_KEY + # here use "trust" command to set key fully or ultimately trusted - only those keys are accepted by QubesBuilder + + +All Qubes developers’ keys are signed by the Qubes Master Signing Key +(which is set as ultimately trusted key), so are trusted automatically. + +If you are the owner of Master key and want to revoke such signature, +use the ``revsig`` gpg key edit command and update the key in +qubes-developers-keys.asc - now the key will be no longer trusted +(unless manually set as such). + +Further information +------------------- + + +For advanced :doc:`QubesBuilder ` use, and preparing +sources, take a look at +:doc:`QubesBuilderDetails ` page, or +`QubesBuilder’s doc directory `__. diff --git a/developer/building/qubes-iso-building.md b/developer/building/qubes-iso-building.md deleted file mode 100644 index a8d6cd5766..0000000000 --- a/developer/building/qubes-iso-building.md +++ /dev/null @@ -1,219 +0,0 @@ ---- -lang: en -layout: doc -permalink: /doc/qubes-iso-building/ -redirect_from: -- /doc/qubes-r3-building/ -- /en/doc/qubes-r3-building/ -- /en/doc/qubes-iso-building/ -- /doc/QubesR3Building/ -- /wiki/QubesR3Building/ -ref: 63 -title: Qubes ISO building ---- - - - -Build Environment ------------------ - -Fedora 36 (and 37) has been successfully used to build Qubes R4.1 with the below steps. -Other rpm-based operating systems may also work. -Travis-CI uses Ubuntu 18.04 to perform test builds, except it can not test the `./setup` script. - -**Notes:** On modern Fedora system (like Fedora 37) SeLinux is enforced by -default and is blocking the build system. You would get error like -"can't create transaction lock on /.../rpm/.rpm.lock (Permission denied)". -You can set SeLinux to permissive mode with - -~~~bash -sudo setenforce 0 -~~~ - - -In `dom0`, install the Fedora 36 (or 37) template if you don't already have it. - -~~~ -sudo qubes-dom0-update qubes-template-fedora-36 -~~~ - -Create a standalone AppVM from the Fedora template. -Set private storage to at least 60 GB if you will be building only the default templates; 100 GB or more if you plan on additional. -It's not required, but if you allocate additional CPU cores, the build process can utilize them at some steps such as the kernel build. -Likewise, more memory (up to 16 GB) can help. -Last, you may want to disable memory balancing, but keep in mind the impact on your other qubes. - -Once you've built the development AppVM, open a Terminal window to it and install the necessary dependencies (see [QubesBuilder](/doc/qubes-builder/) for more info): - -~~~ -$ sudo dnf install git createrepo rpm-build rpm-sign make python3-sh rpmdevtools rpm-sign dialog perl-open python3-pyyaml perl-Digest-MD5 perl-Digest-SHA -~~~ - -Get the necessary keys to verify the sources (run these and other commands below as a regular user, not root): - -~~~ -wget https://keys.qubes-os.org/keys/qubes-master-signing-key.asc -gpg --import qubes-master-signing-key.asc -gpg --edit-key 36879494 -fpr -# Verify fingerprint! See Note below! -# Once verified, set trust to *ultimate* -# (Typical sequence is trust, 5, Y, q) -wget https://keys.qubes-os.org/keys/qubes-developers-keys.asc -gpg --import qubes-developers-keys.asc -~~~ - -**Note** In the above process, we do *not* rely on the security of our server (keys.qubes-os.org) nor the connection (ssl, cert) -- we only rely on you getting the Qubes Master Signing Key fingerprint *somehow* and ensuring they match! -See [verifying signatures](/security/verifying-signatures/#how-to-import-and-authenticate-the-qubes-master-signing-key) for verification sources. - -Now let's bootstrap the builder. Unfortunately, the builder cannot verify itself (the classic Chicken and Egg problem), so we need to verify the signature manually: - -~~~ -git clone https://github.com/QubesOS/qubes-builder.git -cd qubes-builder -git tag -v `git describe` -~~~ - -**Note** It's very important to check if the verification message contains "Good signature from ..." and does not contain "WARNING: This key is not certified with a trusted signature!". - -Assuming the verification went fine, we're good to go with all the rest without ever thinking more about verifying digital signatures on all the rest of the components, apart from an additional step if doing a non-scripted build. -The builder will do that for us for each component, every time we build, even for all auxiliary files (e.g. Xen or Linux kernel sources). - -Build using setup script ------------------ - -Let's configure the builder first (see [procedure](/doc/qubes-iso-building/#build-using-manual-steps) at bottom if you would prefer to manually configure): - -~~~ -cd ~/qubes-builder -./setup -# Select Yes to add Qubes Master Signing Key -# Select Yes to add Qubes OS Signing Key -# Select 4.1 for version -# Stable -# Select Yes for fast Git cloning -# Select Current (if you want the option to use pre-built packages) -# Select No (we want a full build) -# Select fc36 and bullseye (for the currently shipping templates) -# Select builder-rpm, builder-debian, template-whonix, mgmt-salt -# Select Yes to add adrelanos's third party key -# Select Yes (to download) -~~~ - -Once it completes downloading, re-run `setup` to add the Whonix templates: - -~~~ -./setup -# Choose the same options as above, except at templates select: -# fc36, bullseye, whonix-gateway-16, whonix-workstation-16 -~~~ - -Continue the build process with: - -~~~ -make install-deps -make get-sources -~~~ - -When building the Whonix templates, you will often need to add/update the `WHONIX_TBB_VERSION` variable in `builder.conf` at this stage to specify the currently shipping Tor Browser version. -See the related note under [Extra Whonix Build Options](/doc/building-whonix-template/). - -You may also want to add `COMPONENTS := $(filter-out gcc,$(COMPONENTS))` to bypass a multiple hour compile step. -See [QubesBuilder](/doc/qubes-builder/#use-pre-built-qubes-packages) for more detail. - -Finally, if you are making a test build, use: - -~~~ -make qubes -make iso -~~~ - -Or for a fully signed build (this requires setting `SIGN_KEY` in `builder.conf`): - -~~~ -make qubes -make sign-all -make iso -~~~ - -Enjoy your new ISO! - -Build using manual steps ------------------ - -Instead of using `./setup`, you can manually configure the build. -The script takes care of a lot of the keyring preparation for us, so we first need to set that up. - -If you will be building Whonix templates: - -~~~ -cd ~ -gpg --keyserver pgp.mit.edu --recv-keys 916B8D99C38EAF5E8ADC7A2A8D66066A2EEACCDA -gpg --fingerprint 916B8D99C38EAF5E8ADC7A2A8D66066A2EEACCDA -~~~ - -**Note:** It's very important to check the fingerprint displayed against multiple sources such as the [Whonix web site](https://www.whonix.org/wiki/Whonix_Signing_Key), etc. -It should look something like this: - -~~~ -pub rsa4096 2014-01-16 [SC] [expires: 2026-01-23] - Key fingerprint = 916B 8D99 C38E AF5E 8ADC 7A2A 8D66 066A 2EEA CCDA -uid [ unknown] Patrick Schleizer -uid [ unknown] Patrick Schleizer -uid [ unknown] Patrick Schleizer -sub rsa4096 2014-01-16 [E] [expires: 2026-01-23] -sub rsa4096 2014-01-16 [A] [expires: 2026-01-23] -sub rsa4096 2014-01-16 [S] [expires: 2026-01-23] - -~~~ - -Next, prepare the Git keyring directory and copy them in: - -~~~ -export GNUPGHOME=~/qubes-builder/keyrings/git -mkdir --parents "$GNUPGHOME" -cp ~/.gnupg/pubring.gpg "$GNUPGHOME" -cp ~/.gnupg/trustdb.gpg "$GNUPGHOME" -chmod --recursive 700 "$GNUPGHOME" -~~~ - -Copy one of the example configurations: - -~~~ -cd ~/qubes-builder -cp example-configs/qubes-os-master.conf builder.conf -~~~ - -Edit `builder.conf`, referring to `doc/Configuration.md` for a description of all available options. - -Continue the build process with: - -~~~ -make install-deps -make get-sources -unset GNUPGHOME -~~~ - -When building the Whonix templates, you will often need to add/update the `WHONIX_TBB_VERSION` variable at this stage to specify the currently shipping Tor Browser version. -See the related note under [Extra Whonix Build Options](/doc/building-whonix-template/). - -Finally, if you are making a test build, use: - -~~~ -make qubes -make iso -~~~ - -Or for a fully signed build (this requires setting `SIGN_KEY` in `builder.conf`): - -~~~ -make qubes -make sign-all -make iso -~~~ - -Enjoy your new ISO! diff --git a/developer/building/qubes-iso-building.rst b/developer/building/qubes-iso-building.rst new file mode 100644 index 0000000000..2ce8fcccc2 --- /dev/null +++ b/developer/building/qubes-iso-building.rst @@ -0,0 +1,278 @@ +================== +Qubes ISO building +================== + + +.. warning:: + Note: These instructions concern the older Qubes builder (v1). It + supports only building Qubes 4.1 or earlier.The build process has + been completely rewritten in qubes-builder v2. This can be used for + building Qubes R4.1 and later, and all related components. + + +Build Environment +----------------- + + +Fedora 36 (and 37) has been successfully used to build Qubes R4.1 with +the below steps. Other rpm-based operating systems may also work. +Travis-CI uses Ubuntu 18.04 to perform test builds, except it can not +test the ``./setup`` script. + +**Notes:** On modern Fedora system (like Fedora 37) SeLinux is enforced +by default and is blocking the build system. You would get error like +“can’t create transaction lock on /…/rpm/.rpm.lock (Permission denied)”. +You can set SeLinux to permissive mode with + +.. code:: bash + + sudo setenforce 0 + + +In ``dom0``, install the Fedora 36 (or 37) template if you don’t already +have it. + +.. code:: bash + + sudo qubes-dom0-update qubes-template-fedora-36 + + + +Create a standalone AppVM from the Fedora template. Set private storage +to at least 60 GB if you will be building only the default templates; +100 GB or more if you plan on additional. It’s not required, but if you +allocate additional CPU cores, the build process can utilize them at +some steps such as the kernel build. Likewise, more memory (up to 16 GB) +can help. Last, you may want to disable memory balancing, but keep in +mind the impact on your other qubes. + +Once you’ve built the development AppVM, open a Terminal window to it +and install the necessary dependencies (see +:doc:`QubesBuilder ` for more info): + +.. code:: bash + + $ sudo dnf install git createrepo rpm-build rpm-sign make python3-sh rpmdevtools rpm-sign dialog perl-open python3-pyyaml perl-Digest-MD5 perl-Digest-SHA + + + +Get the necessary keys to verify the sources (run these and other +commands below as a regular user, not root): + +.. code:: bash + + wget https://keys.qubes-os.org/keys/qubes-master-signing-key.asc + gpg --import qubes-master-signing-key.asc + gpg --edit-key 36879494 + fpr + # Verify fingerprint! See Note below! + # Once verified, set trust to *ultimate* + # (Typical sequence is trust, 5, Y, q) + wget https://keys.qubes-os.org/keys/qubes-developers-keys.asc + gpg --import qubes-developers-keys.asc + + + +**Note** In the above process, we do *not* rely on the security of our +server (keys.qubes-os.org) nor the connection (ssl, cert) – we only rely +on you getting the Qubes Master Signing Key fingerprint *somehow* and +ensuring they match! See :ref:`verifying signatures ` +for verification sources. + +Now let’s bootstrap the builder. Unfortunately, the builder cannot +verify itself (the classic Chicken and Egg problem), so we need to +verify the signature manually: + +.. code:: bash + + git clone https://github.com/QubesOS/qubes-builder.git + cd qubes-builder + git tag -v `git describe` + + + +**Note** It’s very important to check if the verification message +contains “Good signature from …” and does not contain “WARNING: This key +is not certified with a trusted signature!”. + +Assuming the verification went fine, we’re good to go with all the rest +without ever thinking more about verifying digital signatures on all the +rest of the components, apart from an additional step if doing a +non-scripted build. The builder will do that for us for each component, +every time we build, even for all auxiliary files (e.g. Xen or Linux +kernel sources). + +Build using setup script +------------------------ + + +Let’s configure the builder first (see +:ref:`procedure ` at +bottom if you would prefer to manually configure): + +.. code:: bash + + cd ~/qubes-builder + ./setup + # Select Yes to add Qubes Master Signing Key + # Select Yes to add Qubes OS Signing Key + # Select 4.1 for version + # Stable + # Select Yes for fast Git cloning + # Select Current (if you want the option to use pre-built packages) + # Select No (we want a full build) + # Select fc36 and bullseye (for the currently shipping templates) + # Select builder-rpm, builder-debian, template-whonix, mgmt-salt + # Select Yes to add adrelanos's third party key + # Select Yes (to download) + + + +Once it completes downloading, re-run ``setup`` to add the Whonix +templates: + +.. code:: bash + + ./setup + # Choose the same options as above, except at templates select: + # fc36, bullseye, whonix-gateway-16, whonix-workstation-16 + + + +Continue the build process with: + +.. code:: bash + + make install-deps + make get-sources + + + +When building the Whonix templates, you will often need to add/update +the ``WHONIX_TBB_VERSION`` variable in ``builder.conf`` at this stage to +specify the currently shipping Tor Browser version. See the related note +under `Extra Whonix Build Options `__. + +You may also want to add +``COMPONENTS := $(filter-out gcc,$(COMPONENTS))`` to bypass a multiple +hour compile step. See +:ref:`QubesBuilder ` for +more detail. + +Finally, if you are making a test build, use: + +.. code:: bash + + make qubes + make iso + + + +Or for a fully signed build (this requires setting ``SIGN_KEY`` in +``builder.conf``): + +.. code:: bash + + make qubes + make sign-all + make iso + + + +Enjoy your new ISO! + +Build using manual steps +------------------------ + + +Instead of using ``./setup``, you can manually configure the build. The +script takes care of a lot of the keyring preparation for us, so we +first need to set that up. + +If you will be building Whonix templates: + +.. code:: bash + + cd ~ + gpg --keyserver pgp.mit.edu --recv-keys 916B8D99C38EAF5E8ADC7A2A8D66066A2EEACCDA + gpg --fingerprint 916B8D99C38EAF5E8ADC7A2A8D66066A2EEACCDA + + + +**Note:** It’s very important to check the fingerprint displayed against +multiple sources such as the `Whonix web site `__, etc. It should +look something like this: + +.. code:: bash + + pub rsa4096 2014-01-16 [SC] [expires: 2026-01-23] + Key fingerprint = 916B 8D99 C38E AF5E 8ADC 7A2A 8D66 066A 2EEA CCDA + uid [ unknown] Patrick Schleizer + uid [ unknown] Patrick Schleizer + uid [ unknown] Patrick Schleizer + sub rsa4096 2014-01-16 [E] [expires: 2026-01-23] + sub rsa4096 2014-01-16 [A] [expires: 2026-01-23] + sub rsa4096 2014-01-16 [S] [expires: 2026-01-23] + + + +Next, prepare the Git keyring directory and copy them in: + +.. code:: bash + + export GNUPGHOME=~/qubes-builder/keyrings/git + mkdir --parents "$GNUPGHOME" + cp ~/.gnupg/pubring.gpg "$GNUPGHOME" + cp ~/.gnupg/trustdb.gpg "$GNUPGHOME" + chmod --recursive 700 "$GNUPGHOME" + + + +Copy one of the example configurations: + +.. code:: bash + + cd ~/qubes-builder + cp example-configs/qubes-os-master.conf builder.conf + + + +Edit ``builder.conf``, referring to ``doc/Configuration.md`` for a +description of all available options. + +Continue the build process with: + +.. code:: bash + + make install-deps + make get-sources + unset GNUPGHOME + + + +When building the Whonix templates, you will often need to add/update +the ``WHONIX_TBB_VERSION`` variable at this stage to specify the +currently shipping Tor Browser version. See the related note under +`Extra Whonix Build Options `__. + +Finally, if you are making a test build, use: + +.. code:: bash + + make qubes + make iso + + + +Or for a fully signed build (this requires setting ``SIGN_KEY`` in +``builder.conf``): + +.. code:: bash + + make qubes + make sign-all + make iso + + + +Enjoy your new ISO! diff --git a/developer/building/qubes-template-configs.md b/developer/building/qubes-template-configs.md deleted file mode 100644 index 71139116d7..0000000000 --- a/developer/building/qubes-template-configs.md +++ /dev/null @@ -1,8 +0,0 @@ ---- -lang: en -layout: doc -permalink: /doc/qubes-template-configs/ -redirect_to: https://github.com/QubesOS/qubes-template-configs -ref: 248 -title: Qubes template configs ---- diff --git a/developer/code/code-signing.md b/developer/code/code-signing.md deleted file mode 100644 index c94b9e1bb4..0000000000 --- a/developer/code/code-signing.md +++ /dev/null @@ -1,197 +0,0 @@ ---- -lang: en -layout: doc -permalink: /doc/code-signing/ -ref: 51 -title: Code signing ---- - -All contributions to the Qubes OS [source code](/doc/source-code/) must be cryptographically signed by the author's PGP key. - -## Generating a Key - -(Note: If you already have a PGP key, you may skip this step.) - -Alex Cabal has written an excellent [guide](https://alexcabal.com/creating-the-perfect-gpg-keypair/) on creating a PGP keypair. -Below, we reproduce just the minimum steps in generating a keypair using GnuPG. -Please read Cabal's full guide for further important details. - -~~~ -$ gpg --gen-key -gpg (GnuPG) 2.2.27; Copyright (C) 2021 Free Software Foundation, Inc. -This is free software: you are free to change and redistribute it. -There is NO WARRANTY, to the extent permitted by law. - -gpg: directory '/home/user/.gnupg' created -gpg: keybox '/home/user/.gnupg/pubring.kbx' created -Note: Use "gpg --full-generate-key" for a full featured key generation dialog. - -GnuPG needs to construct a user ID to identify your key. - -Real name: Bilbo Baggins -Email address: bilbo@shire.org -You selected this USER-ID: - "Bilbo Baggins " - -Change (N)ame, (E)mail, or (O)kay/(Q)uit? O -We need to generate a lot of random bytes. It is a good idea to perform -some other action (type on the keyboard, move the mouse, utilize the -disks) during the prime generation; this gives the random number -generator a better chance to gain enough entropy. - - - -We need to generate a lot of random bytes. It is a good idea to perform -some other action (type on the keyboard, move the mouse, utilize the -disks) during the prime generation; this gives the random number -generator a better chance to gain enough entropy. -gpg: /home/user/.gnupg/trustdb.gpg: trustdb created -gpg: key 6E2F4E7AF50A5827 marked as ultimately trusted -gpg: directory '/home/user/.gnupg/openpgp-revocs.d' created -gpg: revocation certificate stored as '/home/user/.gnupg/openpgp-revocs.d/87975838063F97A968D503266E2F4E7AF50A5827.rev' -public and secret key created and signed. - -pub rsa3072 2021-12-30 [SC] [expires: 2023-12-30] - 87975838063F97A968D503266E2F4E7AF50A5827 -uid Bilbo Baggins -sub rsa3072 2021-12-30 [E] [expires: 2023-12-30] -~~~ - -## Upload the Key - -For others to find the public key, please upload it to a server. - -Currently, [these](https://github.com/marmarek/signature-checker/blob/master/check-git-signature#L133-L135) are the recognized servers. - -In the example below, we will use `keyserver.ubuntu.com`. - -Replace 6E2F4E7AF50A5827 with your key ID, preferably the **long keyID** -which is the last 16 hex digits of the long number in the second line -of the output above: -``` -pub rsa3072 2021-12-30 [SC] [expires: 2023-12-30] - 87975838063F97A968D503266E2F4E7AF50A5827 -``` - -```shell_session -$ gpg --send-keys --keyserver hkps://keyserver.ubuntu.com 6E2F4E7AF50A5827 -gpg: sending key 6E2F4E7AF50A5827 to hkps://keyserver.ubuntu.com -``` - -## Using PGP with Git - -If you're submitting a patch via GitHub (or a similar Git server), please sign -your Git commits. - -1. Set up Git to use your key: - - ~~~ - git config --global user.signingkey - ~~~ - -2. Set up Git to sign your commits with your key: - - ~~~ - git config --global commit.gpgsign true - ~~~ - - Alternatively, manually specify when a commit is to be signed: - - ~~~ - git commit -S - ~~~ - -3. (Optional) Create signed tags. - Signed commits are totally sufficient to contribute to Qubes OS. - However, if you have commits which are not signed and you do not want to change them, - you can create a signed tag for the commit and push it before the check. - - This is useful for example, if you have a commit back in the git history which - you like to sign now without rewriting the history. - - ~~~ - git tag -s -m "" - ~~~ - - You can also create an alias to make this easier. - Edit your `~/.gitconfig` file. - In the `[alias]` section, add `stag` to create signed tags and `spush` to create signed tags and push them. - - ~~~ - [alias] - stag = "!bash -c 'id=\"`git rev-parse --verify HEAD`\"; tag_name="signed_tag_for_${id:0:8}"; git tag -s "$tag_name" -m \"Tag for commit $id\"; echo \"$tag_name\"'" - spush = "!bash -c 'git push origin `git stag`'" - ~~~ - - You may also find it convenient to have an alias for verifying the tag on the - latest commit: - - ~~~ - vtag = !git tag -v `git describe` - ~~~ - -## GitHub Signature Verification (optional) - -GitHub shows a green `Verified` label indicating that the GPG signature could be -verified using any of the contributor’s GPG keys uploaded to GitHub. You can -upload your public key on GitHub by adding your public GPG key on the [New GPG -key](https://github.com/settings/gpg/new) under the [SSH GPG keys page](https://github.com/settings/keys). - -## Code Signature Checks - -The [signature-checker](https://github.com/marmarek/signature-checker) checks if code contributions are signed. -Although GitHub adds a little green `Verified` button next to the commit, the [signature-checker](https://github.com/marmarek/signature-checker) uses this algorithm to check if a commit is correctly signed: - -1. Is the commit signed? - If the commit is not signed, you can see the message - > policy/qubesos/code-signing — No signature found -2. If the commit is signed, the key is downloaded from a GPG key server. - If you can see the following error message, please check if you have uploaded the key to a key server. - > policy/qubesos/code-signing — Unable to verify (no valid key found) - -### No Signature Found - -> policy/qubesos/code-signing — No signature found - -In this case, you have several options to sign the commit: - -1. Amend the commit and replace it with a signed commit. - You can use this command to create a new signed commit: - - ``` - git commit --amend -S - ``` - - This also rewrites the commit so you need to push it forcefully: - - ``` - git push -f - ``` - -2. Create a signed tag for the unsigned commit. - If the commit is back in history and you do not want to change it, - you can create a signed tag for this commit and push the signature. - You can use the alias from above: - - ``` - git checkout - git spush - ``` - - Now, the signature checker needs to re-check the signature. - Please comment on the pull request that you would like to have the signatures checked again. - -### Unable To Verify - -> policy/qubesos/code-signing — Unable to verify (no valid key found) - -This means that the [signature-checker](https://github.com/marmarek/signature-checker) has found a signature for the commit -but is not able to verify it using the any key available. -This might be that you forgot to upload the key to a key server. -Please upload it. - -## Using PGP with Email - -If you're submitting a patch by emailing the [developer mailing list](/support/#qubes-devel), simply sign your email with your PGP key. -One good way to do this is with a program like [Enigmail](https://www.enigmail.net/). -Enigmail is a security addon for the Mozilla Thunderbird email client that allows you to easily digitally encrypt and sign your emails. diff --git a/developer/code/code-signing.rst b/developer/code/code-signing.rst new file mode 100644 index 0000000000..cd5aede90b --- /dev/null +++ b/developer/code/code-signing.rst @@ -0,0 +1,248 @@ +============ +Code signing +============ + + +All contributions to the Qubes OS :doc:`source code ` +must be cryptographically signed by the author’s PGP key. + +Generating a Key +---------------- + + +(Note: If you already have a PGP key, you may skip this step.) + +Alex Cabal has written an excellent +`guide `__ on +creating a PGP keypair. Below, we reproduce just the minimum steps in +generating a keypair using GnuPG. Please read Cabal’s full guide for +further important details. + +.. code:: bash + + $ gpg --gen-key + gpg (GnuPG) 2.2.27; Copyright (C) 2021 Free Software Foundation, Inc. + This is free software: you are free to change and redistribute it. + There is NO WARRANTY, to the extent permitted by law. + + gpg: directory '/home/user/.gnupg' created + gpg: keybox '/home/user/.gnupg/pubring.kbx' created + Note: Use "gpg --full-generate-key" for a full featured key generation dialog. + + GnuPG needs to construct a user ID to identify your key. + + Real name: Bilbo Baggins + Email address: bilbo@shire.org + You selected this USER-ID: + "Bilbo Baggins " + + Change (N)ame, (E)mail, or (O)kay/(Q)uit? O + We need to generate a lot of random bytes. It is a good idea to perform + some other action (type on the keyboard, move the mouse, utilize the + disks) during the prime generation; this gives the random number + generator a better chance to gain enough entropy. + + + + We need to generate a lot of random bytes. It is a good idea to perform + some other action (type on the keyboard, move the mouse, utilize the + disks) during the prime generation; this gives the random number + generator a better chance to gain enough entropy. + gpg: /home/user/.gnupg/trustdb.gpg: trustdb created + gpg: key 6E2F4E7AF50A5827 marked as ultimately trusted + gpg: directory '/home/user/.gnupg/openpgp-revocs.d' created + gpg: revocation certificate stored as '/home/user/.gnupg/openpgp-revocs.d/87975838063F97A968D503266E2F4E7AF50A5827.rev' + public and secret key created and signed. + + pub rsa3072 2021-12-30 [SC] [expires: 2023-12-30] + 87975838063F97A968D503266E2F4E7AF50A5827 + uid Bilbo Baggins + sub rsa3072 2021-12-30 [E] [expires: 2023-12-30] + + + +Upload the Key +-------------- + + +For others to find the public key, please upload it to a server. + +Currently, +`these `__ +are the recognized servers. + +In the example below, we will use ``keyserver.ubuntu.com``. + +Replace 6E2F4E7AF50A5827 with your key ID, preferably the **long keyID** +which is the last 16 hex digits of the long number in the second line of +the output above: + +.. code:: bash + + pub rsa3072 2021-12-30 [SC] [expires: 2023-12-30] + 87975838063F97A968D503266E2F4E7AF50A5827 + + + +.. code:: bash + + $ gpg --send-keys --keyserver hkps://keyserver.ubuntu.com 6E2F4E7AF50A5827 + gpg: sending key 6E2F4E7AF50A5827 to hkps://keyserver.ubuntu.com + + +Using PGP with Git +------------------ + + +If you’re submitting a patch via GitHub (or a similar Git server), +please sign your Git commits. + +1. Set up Git to use your key: + + .. code:: bash + + git config --global user.signingkey + + + +2. Set up Git to sign your commits with your key: + + .. code:: bash + + git config --global commit.gpgsign true + + + Alternatively, manually specify when a commit is to be signed: + + .. code:: bash + + git commit -S + + + +3. (Optional) Create signed tags. Signed commits are totally sufficient + to contribute to Qubes OS. However, if you have commits which are not + signed and you do not want to change them, you can create a signed + tag for the commit and push it before the check. + This is useful for example, if you have a commit back in the git + history which you like to sign now without rewriting the history. + + .. code:: bash + + git tag -s -m "" + + + You can also create an alias to make this easier. Edit your + ``~/.gitconfig`` file. In the ``[alias]`` section, add ``stag`` to + create signed tags and ``spush`` to create signed tags and push them. + + .. code:: bash + + [alias] + stag = "!bash -c 'id=\"`git rev-parse --verify HEAD`\"; tag_name="signed_tag_for_${id:0:8}"; git tag -s "$tag_name" -m \"Tag for commit $id\"; echo \"$tag_name\"'" + spush = "!bash -c 'git push origin `git stag`'" + + + You may also find it convenient to have an alias for verifying the + tag on the latest commit: + + .. code:: bash + + vtag = !git tag -v `git describe` + + + + + +GitHub Signature Verification (optional) +---------------------------------------- + + +GitHub shows a green ``Verified`` label indicating that the GPG +signature could be verified using any of the contributor’s GPG keys +uploaded to GitHub. You can upload your public key on GitHub by adding +your public GPG key on the `New GPG key `__ under the `SSH GPG keys page `__. + +Code Signature Checks +--------------------- + + +The +`signature-checker `__ +checks if code contributions are signed. Although GitHub adds a little +green ``Verified`` button next to the commit, the +`signature-checker `__ +uses this algorithm to check if a commit is correctly signed: + +1. Is the commit signed? If the commit is not signed, you can see the + message > policy/qubesos/code-signing — No signature found + +2. If the commit is signed, the key is downloaded from a GPG key server. + If you can see the following error message, please check if you have + uploaded the key to a key server. > policy/qubesos/code-signing — + Unable to verify (no valid key found) + + + +No Signature Found +^^^^^^^^^^^^^^^^^^ + + + policy/qubesos/code-signing — No signature found + +In this case, you have several options to sign the commit: + +1. Amend the commit and replace it with a signed commit. You can use + this command to create a new signed commit: + + .. code:: bash + + git commit --amend -S + + + This also rewrites the commit so you need to push it forcefully: + + .. code:: bash + + git push -f + + + +2. Create a signed tag for the unsigned commit. If the commit is back in + history and you do not want to change it, you can create a signed tag + for this commit and push the signature. You can use the alias from + above: + + .. code:: bash + + git checkout + git spush + + + Now, the signature checker needs to re-check the signature. Please + comment on the pull request that you would like to have the + signatures checked again. + + + +Unable To Verify +^^^^^^^^^^^^^^^^ + + + policy/qubesos/code-signing — Unable to verify (no valid key found) + +This means that the +`signature-checker `__ +has found a signature for the commit but is not able to verify it using +the any key available. This might be that you forgot to upload the key +to a key server. Please upload it. + +Using PGP with Email +-------------------- + + +If you’re submitting a patch by emailing the :ref:`developer mailing list `, simply sign your email with your PGP +key. One good way to do this is with a program like +`Enigmail `__. Enigmail is a security addon +for the Mozilla Thunderbird email client that allows you to easily +digitally encrypt and sign your emails. diff --git a/developer/code/coding-style.md b/developer/code/coding-style.md deleted file mode 100644 index beb616a251..0000000000 --- a/developer/code/coding-style.md +++ /dev/null @@ -1,185 +0,0 @@ ---- -lang: en -layout: doc -permalink: /doc/coding-style/ -redirect_from: -- /en/doc/coding-style/ -- /doc/CodingStyle/ -- /wiki/CodingStyle/ -- /trac/wiki/CodingStyle/ -ref: 53 -title: Coding style ---- - -Rationale ---------- - -Maintaining proper coding style is very important for any large software project, such as Qubes. Here's why: - -- It eases maintenance tasks, such as adding new functionality or generalizing code later, -- It allows others (as well as the future you!) to easily understand fragments of code and what they were supposed to do, and thus makes it easier to later extend them with newer functionality or bug fixes, -- It allows others to easily review the code and catch various bugs, -- It provides for an aesthetically pleasing experience when one reads the code... - -Often, developers, usually smart ones, undersell the value of proper coding style, thinking that it's much more important how their code works. These developers expect that if their code solves some problem using a nice and neat trick, then that's all that is really required. Such thinking shows, however, immaturity and is a signal that the developer, no matter how bright and smart, might not be a good fit for larger projects. Writing a clever exploit for a Black Hat show is one thing - writing useful software supposed to be used and maintained for years is quite a different story. If you want to show off what a smart programmer you are, then you should become a researcher and write exploits. If, on the other hand, you want to be part of a team that makes real, useful software, you should ensure your coding style is impeccable. At Qubes project, we often took shortcuts and wrote nasty code, and this has always back fired at us, sometime months, sometime years later, the net result being we had to spend time fixing code, rather than implementing new functionality. - -And here's a [link to the real case](https://groups.google.com/forum/#!msg/qubes-devel/XgTo6L8-5XA/JLOadvBqnqMJ) (one Qubes Security Bulletin) demonstrating how the lackadaisical coding style lead to a real security bug. Never assume you're smart enough to disregard clean and rigorous coding! - -General typographic conventions -------------------------------- - -- **Use space-expanded tabs that equal 4 spaces.** Yes, we know, there are many arguments for using "real" tabs instead of space-expanded tabs, but we need to pick one convention to make the project consistent. One argument for using space-expanded tabs is that this way the programmer is in control of how the code will look like, despite how other users have configured their editors to visualize the tabs (of course, we assume any sane person uses a fixed-width font for viewing the source code). If it makes you feel any better, assume this is just an arbitrary choice made to enforce a unified style. - -- **Maintain max. line length of 80 characters**. Even though today's monitors often are very wide and it's often not a problem to have 120 characters displayed in an editor, maintaining shorter line lengths improves readability. It also allows others to have two parallel windows open, side by side, each with different parts of the source code. - -- **Naming conventions for any OS *other than Windows***: - - `ClassName` - - `some_variable`, `some_function`, `some_argument` - -- **Naming convention *for Windows OS*** -- exceptionally to preserve Windows conventions please use the following: - - `ClassName`, `FunctionName` - - `pszArgumentOne`, `hPipe` -- use Hungarian notation for argument and variables - -- **Maintain a decent amount of horizontal spacing**, e.g. add a space after `if` or before `{` in C, and similar in other languages. Whether and where to also use spaces within expressions, such as (x\*2+5) vs. (x \* 2 + 5) is left to the developer's judgment. Do not put spaces immediately after or before the brackets in expressions, so avoid constructs like this: `if ( condition )` and use ones like this: `if (condition)` instead. - -- **Use single new lines** ('\\n' aka LF) in any non-Windows source code. On Windows, exceptionally, use the CRLF line endings (--). This will allow the source code to be easily viewable in various Windows-based programs. - -- **Use descriptive names for variables and functions**! Really, at a time when most editors have auto-completion features, there is no excuse for using short variable names. - -- **Comments should be indented together with the code**, e.g. like this: - - ~~~ - for (...) { - // The following code finds PGP private key matching the given public key in O(1) - while (key_found) { - (...) - } - } - ~~~ - -File naming conventions ------------------------ - -- All file names written with small letters, use dash to separate words, rather than underscores, e.g. `qubes-dom0-update`. Never use spaces! - -**File naming in Linux/Unix-like systems:** - -- User commands that operate on particular VMs (also those accessible in VMs): `/usr/bin/qvm-*` -- User commands that apply to the whole system (Dom0 only): `/usr/bin/qubes-*` -- Auxiliary executables and scripts in `/usr/libexec/qubes/` (Note: previously we used `/usr/lib/qubes` but decided to change that) -- Helper, non-executable files in `/usr/share/qubes/` -- Various config files in `/etc/qubes` -- Qubes RPC services in `/etc/qubes-rpc`. Qubes RPC Policy definitions (only in Dom0) in `/etc/qubes-rpc/policy/` -- All VM-related configs, images, and other files in `/var/lib/qubes/` -- System-wide temporary files which reflect the current state of system in `/var/run/qubes` -- Logs: either log to the system-wide messages, or to `/var/log/qubes/` - -**File naming in Windows systems:** - -- All base qubes-related files in `C:\Program Files\Invisible Things Lab\Qubes\` (Exceptionally spaces are allowed here to adhere to Windows naming conventions) -- Other, third-party files, not Qubes-specific, such as e.g. Xen PV drivers might be in different vendor subdirs, e.g. `C:\Program Files\Xen PV Drivers` - -General programming style guidelines ------------------------------------- - -- Do not try to impress with your coding kung-fu, do not use tricks to save 2 lines of code, always prefer readability over trickiness! -- Make sure your code compiles and builds without warnings. -- Always think first about interfaces (e.g. function arguments, or class methods) and data structures before you start writing the actual code. -- Use comments to explain non-trivial code fragments, or expected behavior of more complex functions, if it is not clear from their name. -- Do **not** use comments for code fragments where it is immediately clear what the code does. E.g. avoid constructs like this: - - ~~~ - // Return window id - int get_window_id (...) { - (...) - return id; - } - ~~~ - -- Do **not** use comments to disable code fragments. In production code there should really be no commented or disabled code fragments. If you really, really have a good reason to retain some fragment of unused code, use \#if or \#ifdef to disable it, e.g.: - - ~~~ - #if 0 - (...) // Some unused code here - #endif - ~~~ - - ... and preferably use some descriptive macro instead of just `0`, e.g.: - - ~~~ - #if USE_OLD_WINDOW_TRAVERSING - (...) // Some unused code here - #endif - ~~~ - - Not sure how to do similar thing in Python... Anyone? - -> But generally, there is little excuse to keep old, unused code fragments in the code. One should really use the functionality provided by the source code management system, such as git, instead. E.g. create a special branch for storing the old, unused code -- this way you will always be able to merge this code into upstream in the future. - -- Do not hardcode values in the code! The only three numbers that are an exception here and for which it is acceptable to hardcode them are: `0`, `1` and `-1`, and frankly the last two are still controversial... - -Source Code management (Git) guidelines ---------------------------------------- - -- Use git to maintain all code for Qubes project. - -- Before you start using git, make sure you understand that git is a decentralized Source Code Management system, and that it doesn't behave like traditional, centralized source code management systems, such as SVN. Here's a good [introductory book on git](https://git-scm.com/book). Read it. - -- Qubes code is divided into many git repositories. There are several reasons for that: - - This creates natural boundaries between different code blocks, enforcing proper interfaces, and easing independent development to be conducted on various code parts at the same time, without the fear of running into conflicts. - - By maintaining relatively small git repositories, it is easy for new developers to understand the code and contribute new patches, without the need to understand all the other code. - - Code repositories represent also licensing boundaries. So, e.g. because `core-agent-linux` and `core-agent-windows` are maintained in two different repositories, it is possible to have the latter under a proprietary, non-GPL license, while keeping the former fully open source. - - We have drastically changed the layout and naming of the code repositories shortly after Qubes OS R2 Beta 2 release. For details on the current code layout, please read [this article](https://blog.invisiblethings.org/2013/03/21/introducing-qubes-odyssey-framework.html). - -Commit message guidelines -------------------------- - -Please attempt to follow these conventions when writing your Git commit messages: - -- Separate the subject line from the body with a blank line. -- Limit the subject line to approximately 50 characters. -- Capitalize the subject line. -- Do not end the subject line with a period. -- Use the imperative mood in the subject line. -- Wrap the body at 72 characters. -- Use the body to explain *what* and *why* rather than *how*. - -For details, examples, and the rationale behind each of these conventions, please see [this blog post](https://chris.beams.io/posts/git-commit/), which is the source of this list. - -Security coding guidelines --------------------------- - -- As a general rule: **untrusted input** is our \#1 enemy! -- Any input that comes from untrusted, or less trusted, or just differently-trusted, entity should always be considered as malicious and should always be sanitized and verified. So, if your software runs in Dom0 and processes some input from any of the VMs, this input should be considered to be malicious. Even if your software runs in a VM, and processes input from some other VM, you should also assume that the input is malicious and verify it. -- Use `untrusted_` prefix for all variables that hold values read from untrusted party and which have not yet been verified to be decent, e.g.: - - ~~~ - read_struct(untrusted_conf); - /* sanitize start */ - if (untrusted_conf.width > MAX_WINDOW_WIDTH) - untrusted_conf.width = MAX_WINDOW_WIDTH; - if (untrusted_conf.height > MAX_WINDOW_HEIGHT) - untrusted_conf.height = MAX_WINDOW_HEIGHT; - width = untrusted_conf.width; - height = untrusted_conf.height; - ~~~ - -- Use others variables, without the `untrusted_` prefix to hold the sanitized values, as shown above. - -Python-specific guidelines --------------------------- - -- Please follow the guidelines [here](https://peps.python.org/pep-0008/), unless they were in conflict with what is written on this page. - -C and C++ specific guidelines ------------------------------ - -- Do not place code in `*.h` files. -- Use `const` whenever possible, e.g. in function arguments passed via pointers. -- Do not mix procedural and objective code together -- if you write in C++, use classes and objects. -- Think about classes hierarchy, before starting to implement specific methods. - -Bash-specific guidelines ------------------------- - -- Avoid writing scripts in bash whenever possible. Use python instead. Bash-scripts are Unix-specific and will not work under Windows VMs, or in Windows admin domain, or Windows gui domain. diff --git a/developer/code/coding-style.rst b/developer/code/coding-style.rst new file mode 100644 index 0000000000..dd5faf05b2 --- /dev/null +++ b/developer/code/coding-style.rst @@ -0,0 +1,378 @@ +============ +Coding style +============ + + +Rationale +--------- + + +Maintaining proper coding style is very important for any large software +project, such as Qubes. Here’s why: + +- It eases maintenance tasks, such as adding new functionality or + generalizing code later, + +- It allows others (as well as the future you!) to easily understand + fragments of code and what they were supposed to do, and thus makes + it easier to later extend them with newer functionality or bug fixes, + +- It allows others to easily review the code and catch various bugs, + +- It provides for an aesthetically pleasing experience when one reads + the code… + + + +Often, developers, usually smart ones, undersell the value of proper +coding style, thinking that it’s much more important how their code +works. These developers expect that if their code solves some problem +using a nice and neat trick, then that’s all that is really required. +Such thinking shows, however, immaturity and is a signal that the +developer, no matter how bright and smart, might not be a good fit for +larger projects. Writing a clever exploit for a Black Hat show is one +thing - writing useful software supposed to be used and maintained for +years is quite a different story. If you want to show off what a smart +programmer you are, then you should become a researcher and write +exploits. If, on the other hand, you want to be part of a team that +makes real, useful software, you should ensure your coding style is +impeccable. At Qubes project, we often took shortcuts and wrote nasty +code, and this has always back fired at us, sometime months, sometime +years later, the net result being we had to spend time fixing code, +rather than implementing new functionality. + +And here’s a `link to the real case `__ +(one Qubes Security Bulletin) demonstrating how the lackadaisical coding +style lead to a real security bug. Never assume you’re smart enough to +disregard clean and rigorous coding! + +General typographic conventions +------------------------------- + + +- **Use space-expanded tabs that equal 4 spaces.** Yes, we know, there + are many arguments for using “real” tabs instead of space-expanded + tabs, but we need to pick one convention to make the project + consistent. One argument for using space-expanded tabs is that this + way the programmer is in control of how the code will look like, + despite how other users have configured their editors to visualize + the tabs (of course, we assume any sane person uses a fixed-width + font for viewing the source code). If it makes you feel any better, + assume this is just an arbitrary choice made to enforce a unified + style. + +- **Maintain max. line length of 80 characters**. Even though today’s + monitors often are very wide and it’s often not a problem to have 120 + characters displayed in an editor, maintaining shorter line lengths + improves readability. It also allows others to have two parallel + windows open, side by side, each with different parts of the source + code. + +- **Naming conventions for any OS other than Windows**: + + - ``ClassName`` + + - ``some_variable``, ``some_function``, ``some_argument`` + + + +- **Naming convention for Windows OS** – exceptionally to preserve + Windows conventions please use the following: + + - ``ClassName``, ``FunctionName`` + + - ``pszArgumentOne``, ``hPipe`` – use Hungarian notation for + argument and variables + + + +- **Maintain a decent amount of horizontal spacing**, e.g. add a space + after ``if`` or before ``{`` in C, and similar in other languages. + Whether and where to also use spaces within expressions, such as + (x*2+5) vs. (x * 2 + 5) is left to the developer’s judgment. Do not + put spaces immediately after or before the brackets in expressions, + so avoid constructs like this: ``if ( condition )`` and use ones like + this: ``if (condition)`` instead. + +- **Use single new lines** (‘\n’ aka LF) in any non-Windows source + code. On Windows, exceptionally, use the CRLF line endings (–). This + will allow the source code to be easily viewable in various + Windows-based programs. + +- **Use descriptive names for variables and functions**! Really, at a + time when most editors have auto-completion features, there is no + excuse for using short variable names. + +- **Comments should be indented together with the code**, e.g. like + this: + + .. code:: c + + for (...) { + // The following code finds PGP private key matching the given public key in O(1) + while (key_found) { + (...) + } + } + + + + + +File naming conventions +----------------------- + + +- All file names written with small letters, use dash to separate + words, rather than underscores, e.g. ``qubes-dom0-update``. Never use + spaces! + + + +**File naming in Linux/Unix-like systems:** + +- User commands that operate on particular VMs (also those accessible + in VMs): ``/usr/bin/qvm-*`` + +- User commands that apply to the whole system (Dom0 only): + ``/usr/bin/qubes-*`` + +- Auxiliary executables and scripts in ``/usr/libexec/qubes/`` (Note: + previously we used ``/usr/lib/qubes`` but decided to change that) + +- Helper, non-executable files in ``/usr/share/qubes/`` + +- Various config files in ``/etc/qubes`` + +- Qubes RPC services in ``/etc/qubes-rpc``. Qubes RPC Policy + definitions (only in Dom0) in ``/etc/qubes-rpc/policy/`` + +- All VM-related configs, images, and other files in + ``/var/lib/qubes/`` + +- System-wide temporary files which reflect the current state of system + in ``/var/run/qubes`` + +- Logs: either log to the system-wide messages, or to + ``/var/log/qubes/`` + + + +**File naming in Windows systems:** + +- All base qubes-related files in + ``C:\Program Files\Invisible Things Lab\Qubes\`` (Exceptionally + spaces are allowed here to adhere to Windows naming conventions) + +- Other, third-party files, not Qubes-specific, such as e.g. Xen PV + drivers might be in different vendor subdirs, + e.g. ``C:\Program Files\Xen PV Drivers`` + + + +General programming style guidelines +------------------------------------ + + +- Do not try to impress with your coding kung-fu, do not use tricks to + save 2 lines of code, always prefer readability over trickiness! + +- Make sure your code compiles and builds without warnings. + +- Always think first about interfaces (e.g. function arguments, or + class methods) and data structures before you start writing the + actual code. + +- Use comments to explain non-trivial code fragments, or expected + behavior of more complex functions, if it is not clear from their + name. + +- Do **not** use comments for code fragments where it is immediately + clear what the code does. E.g. avoid constructs like this: + + .. code:: c + + // Return window id + int get_window_id (...) { + (...) + return id; + } + + + +- Do **not** use comments to disable code fragments. In production code + there should really be no commented or disabled code fragments. If + you really, really have a good reason to retain some fragment of + unused code, use #if or #ifdef to disable it, e.g.: + + .. code:: c + + #if 0 + (...) // Some unused code here + #endif + + + … and preferably use some descriptive macro instead of just ``0``, + e.g.: + + .. code:: c + + #if USE_OLD_WINDOW_TRAVERSING + (...) // Some unused code here + #endif + + + Not sure how to do similar thing in Python… Anyone? + + + + But generally, there is little excuse to keep old, unused code + fragments in the code. One should really use the functionality + provided by the source code management system, such as git, instead. + E.g. create a special branch for storing the old, unused code – this + way you will always be able to merge this code into upstream in the + future. + +- Do not hardcode values in the code! The only three numbers that are + an exception here and for which it is acceptable to hardcode them + are: ``0``, ``1`` and ``-1``, and frankly the last two are still + controversial… + + + +Source Code management (Git) guidelines +--------------------------------------- + + +- Use git to maintain all code for Qubes project. + +- Before you start using git, make sure you understand that git is a + decentralized Source Code Management system, and that it doesn’t + behave like traditional, centralized source code management systems, + such as SVN. Here’s a good `introductory book on git `__. Read it. + +- Qubes code is divided into many git repositories. There are several + reasons for that: + + - This creates natural boundaries between different code blocks, + enforcing proper interfaces, and easing independent development to + be conducted on various code parts at the same time, without the + fear of running into conflicts. + + - By maintaining relatively small git repositories, it is easy for + new developers to understand the code and contribute new patches, + without the need to understand all the other code. + + - Code repositories represent also licensing boundaries. So, + e.g. because ``core-agent-linux`` and ``core-agent-windows`` are + maintained in two different repositories, it is possible to have + the latter under a proprietary, non-GPL license, while keeping the + former fully open source. + + - We have drastically changed the layout and naming of the code + repositories shortly after Qubes OS R2 Beta 2 release. For details + on the current code layout, please read `this article `__. + + + + + +Commit message guidelines +------------------------- + + +Please attempt to follow these conventions when writing your Git commit +messages: + +- Separate the subject line from the body with a blank line. + +- Limit the subject line to approximately 50 characters. + +- Capitalize the subject line. + +- Do not end the subject line with a period. + +- Use the imperative mood in the subject line. + +- Wrap the body at 72 characters. + +- Use the body to explain *what* and *why* rather than *how*. + + + +For details, examples, and the rationale behind each of these +conventions, please see `this blog post `__, which is the source +of this list. + +Security coding guidelines +-------------------------- + + +- As a general rule: **untrusted input** is our #1 enemy! + +- Any input that comes from untrusted, or less trusted, or just + differently-trusted, entity should always be considered as malicious + and should always be sanitized and verified. So, if your software + runs in Dom0 and processes some input from any of the VMs, this input + should be considered to be malicious. Even if your software runs in a + VM, and processes input from some other VM, you should also assume + that the input is malicious and verify it. + +- Use ``untrusted_`` prefix for all variables that hold values read + from untrusted party and which have not yet been verified to be + decent, e.g.: + + .. code:: c + + read_struct(untrusted_conf); + /* sanitize start */ + if (untrusted_conf.width > MAX_WINDOW_WIDTH) + untrusted_conf.width = MAX_WINDOW_WIDTH; + if (untrusted_conf.height > MAX_WINDOW_HEIGHT) + untrusted_conf.height = MAX_WINDOW_HEIGHT; + width = untrusted_conf.width; + height = untrusted_conf.height; + + + +- Use others variables, without the ``untrusted_`` prefix to hold the + sanitized values, as shown above. + + + +Python-specific guidelines +-------------------------- + + +- Please follow the guidelines + `here `__, unless they were in + conflict with what is written on this page. + + + +C and C++ specific guidelines +----------------------------- + + +- Do not place code in ``*.h`` files. + +- Use ``const`` whenever possible, e.g. in function arguments passed + via pointers. + +- Do not mix procedural and objective code together – if you write in + C++, use classes and objects. + +- Think about classes hierarchy, before starting to implement specific + methods. + + + +Bash-specific guidelines +------------------------ + + +- Avoid writing scripts in bash whenever possible. Use python instead. + Bash-scripts are Unix-specific and will not work under Windows VMs, + or in Windows admin domain, or Windows gui domain. + + diff --git a/developer/code/license.md b/developer/code/license.md deleted file mode 100644 index 564fbd0b84..0000000000 --- a/developer/code/license.md +++ /dev/null @@ -1,24 +0,0 @@ ---- -lang: en -layout: doc -permalink: /doc/license/ -redirect_from: -- /en/doc/license/ -- /doc/QubesLicensing/ -- /wiki/QubesLicensing/ -ref: 52 -title: Software license ---- - -Qubes OS is a compilation of software packages, each under its own license. The -compilation is made available under the GNU General Public License version 2 -(GPLv2). However, the license for this compilation does not supersede the -license of any package included in the compilation. - -The source code of Qubes OS is contained in repositories under the -[@QubesOS](https://github.com/QubesOS) account on GitHub. This source code is -made available under GPLv2, unless there is a `LICENSE` file in the root of the -containing repository that specifies a different license. - -The full text of the GPLv2 license can be found -[here](https://www.gnu.org/licenses/gpl-2.0.html). diff --git a/developer/code/license.rst b/developer/code/license.rst new file mode 100644 index 0000000000..c259213991 --- /dev/null +++ b/developer/code/license.rst @@ -0,0 +1,19 @@ +================ +Software license +================ + + +Qubes OS is a compilation of software packages, each under its own +license. The compilation is made available under the GNU General Public +License version 2 (GPLv2). However, the license for this compilation +does not supersede the license of any package included in the +compilation. + +The source code of Qubes OS is contained in repositories under the +`@QubesOS `__ account on GitHub. This source +code is made available under GPLv2, unless there is a ``LICENSE`` file +in the root of the containing repository that specifies a different +license. + +The full text of the GPLv2 license can be found +`here `__. diff --git a/developer/code/source-code.md b/developer/code/source-code.md deleted file mode 100644 index 85e5cf13a9..0000000000 --- a/developer/code/source-code.md +++ /dev/null @@ -1,82 +0,0 @@ ---- -lang: en -layout: doc -permalink: /doc/source-code/ -redirect_from: -- /en/doc/source-code/ -- /doc/SourceCode/ -- /wiki/SourceCode/ -ref: 54 -title: Source code ---- - -All the Qubes code is kept in Git repositories. We have divided the project into -several components, each of which has its own separate repository, for example: - -* `core-admin.git` -- The core Qubes infrastructure, responsible for VM - management, VM templates, fs sharing, etc. -* `gui-daemon.git` -- GUI virtualization, Dom0 side. -* `gui-agent-linux.git` -- GUI virtualization, Linux VM side. -* `linux-template-builder.git` -- Scripts and other files used to create Qubes - template images. - -All of our repositories are available under the [QubesOS GitHub account](https://github.com/QubesOS/). - -To clone a repository: - -~~~ -git clone https://github.com/QubesOS/qubes-.git -~~~ - -e.g.: - -~~~ -git clone https://github.com/QubesOS/qubes-core-admin.git core-admin -~~~ - -To build Qubes you do not need to download all these repositories. -If you use [qubes builder](/doc/QubesBuilder/) you can specify *what* you want to build, and download only the repositories needed to build that target. - -If you really do want to clone **all** of the repositories, you can use these commands: - -~~~ -curl "https://api.github.com/orgs/QubesOS/repos?page=1&per_page=100" | grep -e 'clone_url*' | cut -d \" -f 4 | xargs -L1 git clone -curl "https://api.github.com/orgs/QubesOS/repos?page=2&per_page=100" | grep -e 'clone_url*' | cut -d \" -f 4 | xargs -L1 git clone -~~~ - -To update (git fetch) **all** of these repositories : - -~~~ -find . -mindepth 1 -maxdepth 1 -type d -exec git -C {} fetch --tags --recurse-submodules=on-demand --all \; -~~~ - -(Alternatively, you can pull instead of just fetching.) - -How to Send Patches -------------------- - -If you want to [contribute code](/doc/contributing/#contributing-code) to the project, there are two ways. Whichever -method you choose, you must [sign your code](/doc/code-signing/) before it can be accepted. - -* **Preferred**: Use GitHub's [fork & pull requests](https://guides.github.com/activities/forking/). - - Opening a pull request on GitHub greatly eases the code review and tracking - process. In addition, especially for bigger changes, it's a good idea to send - a message to the [qubes-devel mailing list](/support/#qubes-devel) in order to notify people who - do not receive GitHub notifications. - -* Send a patch to the [qubes-devel mailing list](/support/#qubes-devel) (`git format-patch`). - - 1. Make all the changes in your working directory, i.e. edit files, move them - around (you can use 'git mv' for this), etc. - 2. Add the changes and commit them (`git add`, `git commit`). Never mix - different changes into one commit! Write a good description of the commit. - The first line should contain a short summary, and then, if you feel like - more explanations are needed, enter an empty new line, and then start the - long, detailed description (optional). - 3. Test your changes NOW: check if RPMs build fine, etc. - 4. Create the patch using `git format-patch`. This has an advantage over - `git diff`, because the former will also include your commit message, your - name and email, so that *your* name will be used as a commit's author. - 5. Send your patch to `qubes-devel`. Start the message subject with - `[PATCH]`. diff --git a/developer/code/source-code.rst b/developer/code/source-code.rst new file mode 100644 index 0000000000..aa6dd81a2e --- /dev/null +++ b/developer/code/source-code.rst @@ -0,0 +1,102 @@ +=========== +Source code +=========== + + +All the Qubes code is kept in Git repositories. We have divided the +project into several components, each of which has its own separate +repository, for example: + +- ``core-admin.git`` – The core Qubes infrastructure, responsible for + VM management, VM templates, fs sharing, etc. + +- ``gui-daemon.git`` – GUI virtualization, Dom0 side. + +- ``gui-agent-linux.git`` – GUI virtualization, Linux VM side. + +- ``linux-template-builder.git`` – Scripts and other files used to + create Qubes template images. + + + +All of our repositories are available under the `QubesOS GitHub account `__. + +To clone a repository: + +.. code:: bash + + git clone https://github.com/QubesOS/qubes-.git + + + +e.g.: + +.. code:: bash + + git clone https://github.com/QubesOS/qubes-core-admin.git core-admin + + + +To build Qubes you do not need to download all these repositories. If +you use :doc:`qubes builder ` you can specify *what* +you want to build, and download only the repositories needed to build +that target. + +If you really do want to clone **all** of the repositories, you can use +these commands: + +.. code:: bash + + curl "https://api.github.com/orgs/QubesOS/repos?page=1&per_page=100" | grep -e 'clone_url*' | cut -d \" -f 4 | xargs -L1 git clone + curl "https://api.github.com/orgs/QubesOS/repos?page=2&per_page=100" | grep -e 'clone_url*' | cut -d \" -f 4 | xargs -L1 git clone + + + +To update (git fetch) **all** of these repositories : + +.. code:: bash + + find . -mindepth 1 -maxdepth 1 -type d -exec git -C {} fetch --tags --recurse-submodules=on-demand --all \; + + + +(Alternatively, you can pull instead of just fetching.) + +How to Send Patches +------------------- + + +If you want to :ref:`contribute code ` to the project, there are +two ways. Whichever method you choose, you must :doc:`sign your code ` before it can be accepted. + +- **Preferred**: Use GitHub’s `fork & pull requests `__. + Opening a pull request on GitHub greatly eases the code review and + tracking process. In addition, especially for bigger changes, it’s a + good idea to send a message to the :ref:`qubes-devel mailing list ` in order to notify people who do not + receive GitHub notifications. + +- Send a patch to the :ref:`qubes-devel mailing list ` (``git format-patch``). + + 1. Make all the changes in your working directory, i.e. edit files, + move them around (you can use ‘git mv’ for this), etc. + + 2. Add the changes and commit them (``git add``, ``git commit``). + Never mix different changes into one commit! Write a good + description of the commit. The first line should contain a short + summary, and then, if you feel like more explanations are needed, + enter an empty new line, and then start the long, detailed + description (optional). + + 3. Test your changes NOW: check if RPMs build fine, etc. + + 4. Create the patch using ``git format-patch``. This has an advantage + over ``git diff``, because the former will also include your + commit message, your name and email, so that *your* name will be + used as a commit’s author. + + 5. Send your patch to ``qubes-devel``. Start the message subject with + ``[PATCH]``. + + + + diff --git a/developer/debugging/automated-tests.md b/developer/debugging/automated-tests.md deleted file mode 100644 index 38e4aca277..0000000000 --- a/developer/debugging/automated-tests.md +++ /dev/null @@ -1,279 +0,0 @@ ---- -lang: en -layout: doc -permalink: /doc/automated-tests/ -redirect_from: -- /en/doc/automated-tests/ -- /doc/AutomatedTests/ -ref: 45 -title: Automated tests ---- - -## Unit and Integration Tests - -Starting with Qubes R3 we use [python unittest](https://docs.python.org/3/library/unittest.html) to perform automatic tests of Qubes OS. -Despite the name, we use it for both [unit tests](https://en.wikipedia.org/wiki/Unit_tests) and [integration tests](https://en.wikipedia.org/wiki/Integration_tests). -The main purpose is, of course, to deliver much more stable releases. - -The integration tests must be run in dom0, but some unit tests can run inside a VM as well. - -### Integration & unit testing in dom0 - -Integration tests are written with the assumption that they will be executed on dedicated hardware and must be run in dom0. All other unit tests can also be run in dom0. - -**Do not run the tests on installations with important data, because you might lose it.** - -All the VMs with a name starting with `test-` on the installation are removed during the process, and all the tests are recklessly started from dom0, even when testing (& possibly breaking) VM components. - -First you need to build all packages that you want to test. Please do not mix branches as this will inevitably lead to failures. Then setup Qubes OS with these packages installed. - -For testing you'll have to stop the `qubesd` service as the tests will use its own custom variant of the service: -`sudo systemctl stop qubesd` - -Don't forget to start it after testing again. - -To start testing you can then use the standard python unittest runner: - -`sudo -E python3 -m unittest -v qubes.tests` - -Alternatively, use the custom Qubes OS test runner: - -`sudo -E python3 -m qubes.tests.run -v` - -Our test runner runs mostly the same as the standard one, but it has some nice additional features like colored output and not needing the "qubes.test" prefix. - -You can use `python3 -m qubes.tests.run -h` to get usage information: - -``` -[user@dom0 ~]$ python3 -m qubes.tests.run -h -usage: run.py [-h] [--verbose] [--quiet] [--list] [--failfast] [--no-failfast] - [--do-not-clean] [--do-clean] [--loglevel LEVEL] - [--logfile FILE] [--syslog] [--no-syslog] [--kmsg] [--no-kmsg] - [TESTNAME [TESTNAME ...]] - -positional arguments: - TESTNAME list of tests to run named like in description - (default: run all tests) - -optional arguments: - -h, --help show this help message and exit - --verbose, -v increase console verbosity level - --quiet, -q decrease console verbosity level - --list, -l list all available tests and exit - --failfast, -f stop on the first fail, error or unexpected success - --no-failfast disable --failfast - --loglevel LEVEL, -L LEVEL - logging level for file and syslog forwarding (one of: - NOTSET, DEBUG, INFO, WARN, WARNING, ERROR, CRITICAL; - default: DEBUG) - --logfile FILE, -o FILE - if set, test run will be also logged to file - --syslog reenable logging to syslog - --no-syslog disable logging to syslog - --kmsg, --very-brave-or-very-stupid - log most important things to kernel ring-buffer - --no-kmsg, --i-am-smarter-than-kay-sievers - do not abuse kernel ring-buffer - --allow-running-along-qubesd - allow running in parallel with qubesd; this is - DANGEROUS and WILL RESULT IN INCONSISTENT SYSTEM STATE - --break-to-repl break to REPL after tests - -When running only specific tests, write their names like in log, in format: -MODULE+"/"+CLASS+"/"+FUNCTION. MODULE should omit initial "qubes.tests.". -Example: basic/TC_00_Basic/test_000_create -``` - -For instance, to run only the tests for the fedora-21 template, you can use the `-l` option, then filter the list: - -``` -[user@dom0 ~]$ python3 -m qubes.tests.run -l | grep fedora-21 -network/VmNetworking_fedora-21/test_000_simple_networking -network/VmNetworking_fedora-21/test_010_simple_proxyvm -network/VmNetworking_fedora-21/test_020_simple_proxyvm_nm -network/VmNetworking_fedora-21/test_030_firewallvm_firewall -network/VmNetworking_fedora-21/test_040_inter_vm -vm_qrexec_gui/TC_00_AppVM_fedora-21/test_000_start_shutdown -vm_qrexec_gui/TC_00_AppVM_fedora-21/test_010_run_gui_app -vm_qrexec_gui/TC_00_AppVM_fedora-21/test_050_qrexec_simple_eof -vm_qrexec_gui/TC_00_AppVM_fedora-21/test_051_qrexec_simple_eof_reverse -vm_qrexec_gui/TC_00_AppVM_fedora-21/test_052_qrexec_vm_service_eof -vm_qrexec_gui/TC_00_AppVM_fedora-21/test_053_qrexec_vm_service_eof_reverse -vm_qrexec_gui/TC_00_AppVM_fedora-21/test_060_qrexec_exit_code_dom0 -vm_qrexec_gui/TC_00_AppVM_fedora-21/test_065_qrexec_exit_code_vm -vm_qrexec_gui/TC_00_AppVM_fedora-21/test_100_qrexec_filecopy -vm_qrexec_gui/TC_00_AppVM_fedora-21/test_110_qrexec_filecopy_deny -vm_qrexec_gui/TC_00_AppVM_fedora-21/test_120_qrexec_filecopy_self -vm_qrexec_gui/TC_20_DispVM_fedora-21/test_000_prepare_dvm -vm_qrexec_gui/TC_20_DispVM_fedora-21/test_010_simple_dvm_run -vm_qrexec_gui/TC_20_DispVM_fedora-21/test_020_gui_app -vm_qrexec_gui/TC_20_DispVM_fedora-21/test_030_edit_file -[user@dom0 ~]$ sudo -E python3 -m qubes.tests.run -v `python3 -m qubes.tests.run -l | grep fedora-21` -``` - -Example test run: - -![snapshot-tests2.png](/attachment/doc/snapshot-tests2.png) - -Tests are also compatible with nose2 test runner, so you can use this instead: - -```bash -sudo systemctl stop qubesd; sudo -E nose2 -v --plugin nose2.plugins.loader.loadtests qubes.tests; sudo systemctl start qubesd -``` - -This may be especially useful together with various nose2 plugins to store tests results (for example `nose2.plugins.junitxml`), to ease presenting results. This is what we use on [OpenQA](https://open.qa/). - -### Unit testing inside a VM - -Many unit tests will also work inside a VM. However all of the tests requiring a dedicated VM to be run (mostly the integration tests) will be skipped. - -Whereas integration tests are mostly stored in the [qubes-core-admin](https://github.com/QubesOS/qubes-core-admin) repository, unit tests can be found in each of the Qubes OS repositories. - -To for example run the `qubes-core-admin` unit tests, you currently have to clone at least [qubes-core-admin](https://github.com/QubesOS/qubes-core-admin) and -its dependency [qubes-core-qrexec](https://github.com/QubesOS/qubes-core-qrexec) repository in the branches that you want to test. - -The below example however will assume that you set up a build environment as described in the [Qubes Builder documentation](/doc/qubes-builder/). - -Assuming you cloned the `qubes-builder` repository to your home directory inside a fedora VM, you can use the following commands to run the unit tests: - -```{.bash} -cd ~ -sudo dnf install python3-pip lvm2 python35 python3-virtualenv -virtualenv -p /usr/bin/python35 python35 -source python35/bin/activate -python3 -V -cd ~/qubes-builder/qubes-src/core-admin -pip3 install -r ci/requirements.txt -export PYTHONPATH=../core-qrexec:test-packages -./run-tests -``` - -To run only the tests related to e.g. `lvm`, you may use: - -`./run-tests -v $(python3 -m qubes.tests.run -l | grep lvm)` - -You can later re-use the created virtual environment including all of the via `pip3` installed packages with `source ~/python35/bin/activate`. - -We recommend to run the unit tests with the Python version that the code is meant to be run with in dom0 (3.5 was just an example above). For instance, the `release4.0` (Qubes 4.0) branch is intended -to be run with Python 3.5 whereas the Qubes 4.1 branch (`master` as of 2020-07) is intended to be run with Python 3.7 or higher. You can always check your dom0 installation for the Python version of -the current stable branch. - -### Tests configuration - -Test runs can be altered using environment variables: - -- `DEFAULT_LVM_POOL` - LVM thin pool to use for tests, in `VolumeGroup/ThinPool` format -- `QUBES_TEST_PCIDEV` - PCI device to be used in PCI passthrough tests (for example sound card) -- `QUBES_TEST_TEMPLATES` - space separated list of templates to run tests on; if not set, all installed templates are tested -- `QUBES_TEST_LOAD_ALL` - load all tests (including tests for all templates) when relevant test modules are imported; this needs to be set for test runners not supporting [load_tests protocol](https://docs.python.org/3/library/unittest.html#load-tests-protocol) - -### Adding a new test to core-admin - -After adding a new unit test to [core-admin/qubes/tests](https://github.com/QubesOS/qubes-core-admin/tree/master/qubes/tests) you'll have to include it in [core-admin/qubes/tests/\_\_init\_\_.py](https://github.com/QubesOS/qubes-core-admin/tree/master/qubes/tests/__init__.py) - -#### Editing `__init__.py` - -You'll also need to add your test at the bottom of the `__init__.py` file, in the method `def load_tests`, in the for loop with `modname`. -Again, given the hypothetical `example.py` test: - -~~~python -for modname in ( - 'qubes.tests.basic', - 'qubes.tests.dom0_update', - 'qubes.tests.network', - 'qubes.tests.vm_qrexec_gui', - 'qubes.tests.backup', - 'qubes.tests.backupcompatibility', - 'qubes.tests.regressions', - 'qubes.tests.example', # This is our newly added test - ): -~~~ - -### Testing PyQt applications - -When testing (Py)QT applications, it's useful to create a separate QApplication object for each test. -But QT framework does not allow multiple QApplication objects in the same process at the same time. -This means it's critical to reliably cleanup the previous instance before creating a new one. -This turns out to be a non-trivial task, especially if _any_ test uses the event loop. -Failure to perform proper cleanup in many cases results in SEGV. -Below you can find steps for the proper cleanup: - -~~~python -import asyncio -import quamash -import unittest -import gc - -class SomeTestCase(unittest.TestCase): - def setUp(self): - [...] - - # force "cleanlooks" style, the default one on Xfce (GtkStyle) use - # static variable internally and caches pointers to later destroyed - # objects (result: SEGV) - self.qtapp = QtGui.QApplication(["test", "-style", "cleanlooks"]) - - # construct event loop even if this particular test doesn't use it, - # otherwise events with qtapp references will be queued there anyway and the - # first test that actually use event loop will try to dereference (already - # destroyed) objects, resulting in SEGV - self.loop = quamash.QEventLoop(self.qtapp) - - def tearDown(self): - [...] - # process any pending events before destroying the object - self.qtapp.processEvents() - - # queue destroying the QApplication object, do that for any other QT - # related objects here too - self.qtapp.deleteLater() - - # process any pending events (other than just queued destroy), just in case - self.qtapp.processEvents() - - # execute main loop, which will process all events, _including just queued destroy_ - self.loop.run_until_complete(asyncio.sleep(0)) - - # at this point it QT objects are destroyed, cleanup all remaining references; - # del other QT object here too - self.loop.close() - del self.qtapp - del self.loop - gc.collect() -~~~ - -## Automated tests with openQA - -**URL:** -**Tests:** - -Manually testing Qubes OS and its installation is a time-consuming process. -We use [OpenQA](https://open.qa/) to automate this process. -It works by installing Qubes in KVM and interacting with it as a user would, including simulating mouse clicks and keyboard presses. -Then, it checks the output to see whether various tests were passed, e.g. by comparing the virtual screen output to screenshots of a successful installation. - -Using openQA to automatically test the Qubes installation process works as of Qubes 4.0-rc4 on 2018-01-26, provided that the versions of KVM and QEMU are new enough and the hardware has VT-x and EPT. -KVM also supports nested virtualization, so HVM should theoretically work. -In practice, however, either Xen or QEMU crashes when this is attempted. -Nonetheless, PV works well, which is sufficient for automated installation testing. - -Thanks to present and past donors who have provided the infrastructure for Qubes' openQA system with hardware that meets these requirements. - -### Looking for patterns in tests - -In order to better visualize patterns in tests the [`openqa_investigator`](https://github.com/QubesOS/openqa-tests-qubesos/blob/master/utils/openqa_investigator.py) script can be used. -It feeds off of the openQA test data to make graph plots. Here is an example: - -![openqa-investigator-splitgpg-example.png](/attachment/doc/openqa-investigator-splitgpg-example.png) - -Some outputs: - - plot by tests - - plot by errors - - markdown - -Some filters: - - filter by error - - filter by test name - -Check out the script's help with `python3 openqa_investigator.py --help` -to see all available options. diff --git a/developer/debugging/automated-tests.rst b/developer/debugging/automated-tests.rst new file mode 100644 index 0000000000..e851ac1fa4 --- /dev/null +++ b/developer/debugging/automated-tests.rst @@ -0,0 +1,355 @@ +=============== +Automated tests +=============== + + +Unit and Integration Tests +-------------------------- + + +Starting with Qubes R3 we use `python unittest `__ to perform +automatic tests of Qubes OS. Despite the name, we use it for both `unit tests `__ and `integration tests `__. The main +purpose is, of course, to deliver much more stable releases. + +The integration tests must be run in dom0, but some unit tests can run +inside a VM as well. + +Integration & unit testing in dom0 +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + + +Integration tests are written with the assumption that they will be +executed on dedicated hardware and must be run in dom0. All other unit +tests can also be run in dom0. + +**Do not run the tests on installations with important data, because you might lose it.** + +All the VMs with a name starting with ``test-`` on the installation are +removed during the process, and all the tests are recklessly started +from dom0, even when testing (& possibly breaking) VM components. + +First you need to build all packages that you want to test. Please do +not mix branches as this will inevitably lead to failures. Then setup +Qubes OS with these packages installed. + +For testing you’ll have to stop the ``qubesd`` service as the tests will +use its own custom variant of the service: +``sudo systemctl stop qubesd`` + +Don’t forget to start it after testing again. + +To start testing you can then use the standard python unittest runner: + +``sudo -E python3 -m unittest -v qubes.tests`` + +Alternatively, use the custom Qubes OS test runner: + +``sudo -E python3 -m qubes.tests.run -v`` + +Our test runner runs mostly the same as the standard one, but it has +some nice additional features like colored output and not needing the +“qubes.test” prefix. + +You can use ``python3 -m qubes.tests.run -h`` to get usage information: + +.. code:: bash + + [user@dom0 ~]$ python3 -m qubes.tests.run -h + usage: run.py [-h] [--verbose] [--quiet] [--list] [--failfast] [--no-failfast] + [--do-not-clean] [--do-clean] [--loglevel LEVEL] + [--logfile FILE] [--syslog] [--no-syslog] [--kmsg] [--no-kmsg] + [TESTNAME [TESTNAME ...]] + + positional arguments: + TESTNAME list of tests to run named like in description + (default: run all tests) + + optional arguments: + -h, --help show this help message and exit + --verbose, -v increase console verbosity level + --quiet, -q decrease console verbosity level + --list, -l list all available tests and exit + --failfast, -f stop on the first fail, error or unexpected success + --no-failfast disable --failfast + --loglevel LEVEL, -L LEVEL + logging level for file and syslog forwarding (one of: + NOTSET, DEBUG, INFO, WARN, WARNING, ERROR, CRITICAL; + default: DEBUG) + --logfile FILE, -o FILE + if set, test run will be also logged to file + --syslog reenable logging to syslog + --no-syslog disable logging to syslog + --kmsg, --very-brave-or-very-stupid + log most important things to kernel ring-buffer + --no-kmsg, --i-am-smarter-than-kay-sievers + do not abuse kernel ring-buffer + --allow-running-along-qubesd + allow running in parallel with qubesd; this is + DANGEROUS and WILL RESULT IN INCONSISTENT SYSTEM STATE + --break-to-repl break to REPL after tests + + When running only specific tests, write their names like in log, in format: + MODULE+"/"+CLASS+"/"+FUNCTION. MODULE should omit initial "qubes.tests.". + Example: basic/TC_00_Basic/test_000_create + + +For instance, to run only the tests for the fedora-21 template, you can +use the ``-l`` option, then filter the list: + +.. code:: bash + + [user@dom0 ~]$ python3 -m qubes.tests.run -l | grep fedora-21 + network/VmNetworking_fedora-21/test_000_simple_networking + network/VmNetworking_fedora-21/test_010_simple_proxyvm + network/VmNetworking_fedora-21/test_020_simple_proxyvm_nm + network/VmNetworking_fedora-21/test_030_firewallvm_firewall + network/VmNetworking_fedora-21/test_040_inter_vm + vm_qrexec_gui/TC_00_AppVM_fedora-21/test_000_start_shutdown + vm_qrexec_gui/TC_00_AppVM_fedora-21/test_010_run_gui_app + vm_qrexec_gui/TC_00_AppVM_fedora-21/test_050_qrexec_simple_eof + vm_qrexec_gui/TC_00_AppVM_fedora-21/test_051_qrexec_simple_eof_reverse + vm_qrexec_gui/TC_00_AppVM_fedora-21/test_052_qrexec_vm_service_eof + vm_qrexec_gui/TC_00_AppVM_fedora-21/test_053_qrexec_vm_service_eof_reverse + vm_qrexec_gui/TC_00_AppVM_fedora-21/test_060_qrexec_exit_code_dom0 + vm_qrexec_gui/TC_00_AppVM_fedora-21/test_065_qrexec_exit_code_vm + vm_qrexec_gui/TC_00_AppVM_fedora-21/test_100_qrexec_filecopy + vm_qrexec_gui/TC_00_AppVM_fedora-21/test_110_qrexec_filecopy_deny + vm_qrexec_gui/TC_00_AppVM_fedora-21/test_120_qrexec_filecopy_self + vm_qrexec_gui/TC_20_DispVM_fedora-21/test_000_prepare_dvm + vm_qrexec_gui/TC_20_DispVM_fedora-21/test_010_simple_dvm_run + vm_qrexec_gui/TC_20_DispVM_fedora-21/test_020_gui_app + vm_qrexec_gui/TC_20_DispVM_fedora-21/test_030_edit_file + [user@dom0 ~]$ sudo -E python3 -m qubes.tests.run -v `python3 -m qubes.tests.run -l | grep fedora-21` + + +Example test run: + +.. figure:: /attachment/doc/snapshot-tests2.png + :alt: snapshot-tests2.png + + snapshot-tests2.png + +Tests are also compatible with nose2 test runner, so you can use this +instead: + +.. code:: bash + + sudo systemctl stop qubesd; sudo -E nose2 -v --plugin nose2.plugins.loader.loadtests qubes.tests; sudo systemctl start qubesd + + +This may be especially useful together with various nose2 plugins to +store tests results (for example ``nose2.plugins.junitxml``), to ease +presenting results. This is what we use on +`OpenQA `__. + +Unit testing inside a VM +^^^^^^^^^^^^^^^^^^^^^^^^ + + +Many unit tests will also work inside a VM. However all of the tests +requiring a dedicated VM to be run (mostly the integration tests) will +be skipped. + +Whereas integration tests are mostly stored in the +`qubes-core-admin `__ +repository, unit tests can be found in each of the Qubes OS +repositories. + +To for example run the ``qubes-core-admin`` unit tests, you currently +have to clone at least +`qubes-core-admin `__ and +its dependency +`qubes-core-qrexec `__ +repository in the branches that you want to test. + +The below example however will assume that you set up a build +environment as described in the :doc:`Qubes Builder documentation `. + +Assuming you cloned the ``qubes-builder`` repository to your home +directory inside a fedora VM, you can use the following commands to run +the unit tests: + +.. code:: bash + + cd ~ + sudo dnf install python3-pip lvm2 python35 python3-virtualenv + virtualenv -p /usr/bin/python35 python35 + source python35/bin/activate + python3 -V + cd ~/qubes-builder/qubes-src/core-admin + pip3 install -r ci/requirements.txt + export PYTHONPATH=../core-qrexec:test-packages + ./run-tests + + +To run only the tests related to e.g. ``lvm``, you may use: + +``./run-tests -v $(python3 -m qubes.tests.run -l | grep lvm)`` + +You can later re-use the created virtual environment including all of +the via ``pip3`` installed packages with +``source ~/python35/bin/activate``. + +We recommend to run the unit tests with the Python version that the code +is meant to be run with in dom0 (3.5 was just an example above). For +instance, the ``release4.0`` (Qubes 4.0) branch is intended to be run +with Python 3.5 whereas the Qubes 4.1 branch (``master`` as of 2020-07) +is intended to be run with Python 3.7 or higher. You can always check +your dom0 installation for the Python version of the current stable +branch. + +Tests configuration +^^^^^^^^^^^^^^^^^^^ + + +Test runs can be altered using environment variables: + +- ``DEFAULT_LVM_POOL`` - LVM thin pool to use for tests, in + ``VolumeGroup/ThinPool`` format + +- ``QUBES_TEST_PCIDEV`` - PCI device to be used in PCI passthrough + tests (for example sound card) + +- ``QUBES_TEST_TEMPLATES`` - space separated list of templates to run + tests on; if not set, all installed templates are tested + +- ``QUBES_TEST_LOAD_ALL`` - load all tests (including tests for all + templates) when relevant test modules are imported; this needs to be + set for test runners not supporting `load_tests protocol `__ + + + +Adding a new test to core-admin +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + + +After adding a new unit test to +`core-admin/qubes/tests `__ +you’ll have to include it in +`core-admin/qubes/tests/__init__.py `__ + +Editing ``__init__.py`` +^^^^^^^^^^^^^^^^^^^^^^^ + + +You’ll also need to add your test at the bottom of the ``__init__.py`` +file, in the method ``def load_tests``, in the for loop with +``modname``. Again, given the hypothetical ``example.py`` test: + +.. code:: python + + for modname in ( + 'qubes.tests.basic', + 'qubes.tests.dom0_update', + 'qubes.tests.network', + 'qubes.tests.vm_qrexec_gui', + 'qubes.tests.backup', + 'qubes.tests.backupcompatibility', + 'qubes.tests.regressions', + 'qubes.tests.example', # This is our newly added test + ): + + +Testing PyQt applications +^^^^^^^^^^^^^^^^^^^^^^^^^ + + +When testing (Py)QT applications, it’s useful to create a separate +QApplication object for each test. But QT framework does not allow +multiple QApplication objects in the same process at the same time. This +means it’s critical to reliably cleanup the previous instance before +creating a new one. This turns out to be a non-trivial task, especially +if *any* test uses the event loop. Failure to perform proper cleanup in +many cases results in SEGV. Below you can find steps for the proper +cleanup: + +.. code:: python + + import asyncio + import quamash + import unittest + import gc + + class SomeTestCase(unittest.TestCase): + def setUp(self): + [...] + + # force "cleanlooks" style, the default one on Xfce (GtkStyle) use + # static variable internally and caches pointers to later destroyed + # objects (result: SEGV) + self.qtapp = QtGui.QApplication(["test", "-style", "cleanlooks"]) + + # construct event loop even if this particular test doesn't use it, + # otherwise events with qtapp references will be queued there anyway and the + # first test that actually use event loop will try to dereference (already + # destroyed) objects, resulting in SEGV + self.loop = quamash.QEventLoop(self.qtapp) + + def tearDown(self): + [...] + # process any pending events before destroying the object + self.qtapp.processEvents() + + # queue destroying the QApplication object, do that for any other QT + # related objects here too + self.qtapp.deleteLater() + + # process any pending events (other than just queued destroy), just in case + self.qtapp.processEvents() + + # execute main loop, which will process all events, _including just queued destroy_ + self.loop.run_until_complete(asyncio.sleep(0)) + + # at this point it QT objects are destroyed, cleanup all remaining references; + # del other QT object here too + self.loop.close() + del self.qtapp + del self.loop + gc.collect() + + +Automated tests with openQA +--------------------------- + + +**URL:** https://openqa.qubes-os.org/ **Tests:** +https://github.com/marmarek/openqa-tests-qubesos + +Manually testing Qubes OS and its installation is a time-consuming +process. We use `OpenQA `__ to automate this process. +It works by installing Qubes in KVM and interacting with it as a user +would, including simulating mouse clicks and keyboard presses. Then, it +checks the output to see whether various tests were passed, e.g. by +comparing the virtual screen output to screenshots of a successful +installation. + +Using openQA to automatically test the Qubes installation process works +as of Qubes 4.0-rc4 on 2018-01-26, provided that the versions of KVM and +QEMU are new enough and the hardware has VT-x and EPT. KVM also supports +nested virtualization, so HVM should theoretically work. In practice, +however, either Xen or QEMU crashes when this is attempted. Nonetheless, +PV works well, which is sufficient for automated installation testing. + +Thanks to present and past donors who have provided the infrastructure +for Qubes’ openQA system with hardware that meets these requirements. + +Looking for patterns in tests +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + + +In order to better visualize patterns in tests the +`openqa_investigator `__ +script can be used. It feeds off of the openQA test data to make graph +plots. Here is an example: + +.. figure:: /attachment/doc/openqa-investigator-splitgpg-example.png + :alt: openqa-investigator-splitgpg-example.png + + openqa-investigator-splitgpg-example.png + +Some outputs: - plot by tests - plot by errors - markdown + +Some filters: - filter by error - filter by test name + +Check out the script’s help with +``python3 openqa_investigator.py --help`` to see all available options. diff --git a/developer/debugging/mount-lvm-image.md b/developer/debugging/mount-lvm-image.md deleted file mode 100644 index d5b7608078..0000000000 --- a/developer/debugging/mount-lvm-image.md +++ /dev/null @@ -1,54 +0,0 @@ ---- -lang: en -layout: doc -permalink: /doc/mount-lvm-image/ -ref: 46 -title: How to mount LVM images ---- - -You want to read your LVM image (e.g., there is a problem where you can't start any VMs except dom0). - -1: make the image available for qubesdb. -From dom0 terminal: - -```bash -# Example: /dev/qubes_dom0/vm-debian-9-tmp-root -[user@dom0]$ dev=$(basename $(readlink /dev/YOUR_LVM_VG/YOUR_LVM_IMAGE)) -[user@dom0]$ qubesdb-write /qubes-block-devices/$dev/desc "YOUR_LVM_IMAGE" -``` - -2: Create a new disposable VM - -```bash -[user@dom0]$ qvm-run -v --dispvm=YOUR_DVM_TEMPLATE --service qubes.StartApp+xterm & -``` - -3: Attach the device to your newly created disp VM - -From the GUI, or from the command line: - -```bash -[user@dom0]$ qvm-block attach NEWLY_CREATED_DISPVM dom0:$dev -``` - -4: Mount the partition you want to, and do what you want with it - -```bash -[user@dispXXXX]$ mount /dev/xvdiX /mnt/ -``` - -5: Umount and kill the VM - -```bash -[user@dispXXXX]$ umount /mnt/ -``` - -6: Remove the image from qubesdb - -```bash -[user@dom0]$ qubesdb-rm /qubes-block-devices/$dev/ -``` - -# References - -Please consult this issue's [comment](https://github.com/QubesOS/qubes-issues/issues/4687#issuecomment-451626625). diff --git a/developer/debugging/mount-lvm-image.rst b/developer/debugging/mount-lvm-image.rst new file mode 100644 index 0000000000..900d3cb42c --- /dev/null +++ b/developer/debugging/mount-lvm-image.rst @@ -0,0 +1,60 @@ +======================= +How to mount LVM images +======================= + + +You want to read your LVM image (e.g., there is a problem where you +can’t start any VMs except dom0). + +1: make the image available for qubesdb. From dom0 terminal: + +.. code:: bash + + # Example: /dev/qubes_dom0/vm-debian-9-tmp-root + [user@dom0]$ dev=$(basename $(readlink /dev/YOUR_LVM_VG/YOUR_LVM_IMAGE)) + [user@dom0]$ qubesdb-write /qubes-block-devices/$dev/desc "YOUR_LVM_IMAGE" + + +2: Create a new disposable VM + +.. code:: bash + + [user@dom0]$ qvm-run -v --dispvm=YOUR_DVM_TEMPLATE --service qubes.StartApp+xterm & + + +3: Attach the device to your newly created disp VM + +From the GUI, or from the command line: + +.. code:: bash + + [user@dom0]$ qvm-block attach NEWLY_CREATED_DISPVM dom0:$dev + + +4: Mount the partition you want to, and do what you want with it + +.. code:: bash + + [user@dispXXXX]$ mount /dev/xvdiX /mnt/ + + +5: Umount and kill the VM + +.. code:: bash + + [user@dispXXXX]$ umount /mnt/ + + +6: Remove the image from qubesdb + +.. code:: bash + + [user@dom0]$ qubesdb-rm /qubes-block-devices/$dev/ + + +References +---------- + + +Please consult this issue’s +`comment `__. diff --git a/developer/debugging/safe-remote-ttys.md b/developer/debugging/safe-remote-ttys.md deleted file mode 100644 index b618311603..0000000000 --- a/developer/debugging/safe-remote-ttys.md +++ /dev/null @@ -1,76 +0,0 @@ ---- -lang: en -layout: doc -permalink: /doc/safe-remote-ttys/ -redirect_from: -- /en/doc/safe-remote-ttys/ -ref: 49 -title: Safe remote dom0 terminals ---- - -If you do not have working graphics in Dom0, then using a terminal can be quite annoying! -This was the case for the author while trying to debug PCI-passthrough of a machine's primary (only) GPU. - -Your first thought might be to just allow network access to Dom0, enable ssh, and connect in remotely. -But, this gravely violates the Qubes security model. - -Instead, a better solution is to split the input and output paths of using a terminal. -Use your normal keyboard for input, but have the output go to a remote machine in a unidirectional manner. - -To do this, we make use of script(1), qvm-run, and optionally your network transport of choice. - -To a different VM ------------------ - -As an example of forwarding terminal output to another VM on the same machine: - -~~~ -$ mkfifo /tmp/foo -$ qvm-run -p some-vm 'xterm -e "cat 0<&5" 5<&0' /dev/null 2>&1 & -$ script -f /tmp/foo -~~~ - -To a different machine ----------------------- - -In this case over SSH (from a network-connected VM): - -~~~ -$ mkfifo /tmp/foo -$ qvm-run -p some-vm \ - 'ssh user@host sh -c "DISPLAY=:0 xterm -e \"cat 0<&5\" 5<&0"' \ - /dev/null 2>&1 & -$ script -f /tmp/foo -~~~ - -Note that no data received over SSH is ever treated as terminal input in Dom0. -The input path remains only from your trusted local keyboard. - -Multiple terminals ------------------- - -For multiple terminals, you may find it easier to just use tmux than to try to blindly switch to the correct window. - -Terminal size -------------- - -It is up to you to ensure the sizes of the local and remote terminal are the same, otherwise things may display incorrectly (especially in interactive programs). -Depending on your shell, the size of your local (blind) terminal is likely stored in the `$LINES` and `$COLUMNS` variables. - -~~~ -$ echo $COLUMNS $LINES -80 24 -~~~ - -A note on serial consoles -------------------------- - -If your machine has a serial console, you may with to use that, but note that a similar split-I/O model should be used to ensure Dom0 integrity. -If you use the serial console as normal (via e.g. getty on ttyX, and logging in as normal), then the machine at the end of the serial cable could compromise your machine! -Ideally, you would take input from your trusted keyboard, and only send the output over the serial cable via e.g. disabling getty and using: - -~~~ -script -f /dev/ttyS0 -~~~ - -You don't even need to connect the TX pin. diff --git a/developer/debugging/safe-remote-ttys.rst b/developer/debugging/safe-remote-ttys.rst new file mode 100644 index 0000000000..b3901f02c6 --- /dev/null +++ b/developer/debugging/safe-remote-ttys.rst @@ -0,0 +1,97 @@ +========================== +Safe remote dom0 terminals +========================== + + +If you do not have working graphics in Dom0, then using a terminal can +be quite annoying! This was the case for the author while trying to +debug PCI-passthrough of a machine’s primary (only) GPU. + +Your first thought might be to just allow network access to Dom0, enable +ssh, and connect in remotely. But, this gravely violates the Qubes +security model. + +Instead, a better solution is to split the input and output paths of +using a terminal. Use your normal keyboard for input, but have the +output go to a remote machine in a unidirectional manner. + +To do this, we make use of script(1), qvm-run, and optionally your +network transport of choice. + +To a different VM +----------------- + + +As an example of forwarding terminal output to another VM on the same +machine: + +.. code:: bash + + $ mkfifo /tmp/foo + $ qvm-run -p some-vm 'xterm -e "cat 0<&5" 5<&0' /dev/null 2>&1 & + $ script -f /tmp/foo + + + +To a different machine +---------------------- + + +In this case over SSH (from a network-connected VM): + +.. code:: bash + + $ mkfifo /tmp/foo + $ qvm-run -p some-vm \ + 'ssh user@host sh -c "DISPLAY=:0 xterm -e \"cat 0<&5\" 5<&0"' \ + /dev/null 2>&1 & + $ script -f /tmp/foo + + + +Note that no data received over SSH is ever treated as terminal input in +Dom0. The input path remains only from your trusted local keyboard. + +Multiple terminals +------------------ + + +For multiple terminals, you may find it easier to just use tmux than to +try to blindly switch to the correct window. + +Terminal size +------------- + + +It is up to you to ensure the sizes of the local and remote terminal are +the same, otherwise things may display incorrectly (especially in +interactive programs). Depending on your shell, the size of your local +(blind) terminal is likely stored in the ``$LINES`` and ``$COLUMNS`` +variables. + +.. code:: bash + + $ echo $COLUMNS $LINES + 80 24 + + + +A note on serial consoles +------------------------- + + +If your machine has a serial console, you may with to use that, but note +that a similar split-I/O model should be used to ensure Dom0 integrity. +If you use the serial console as normal (via e.g. getty on ttyX, and +logging in as normal), then the machine at the end of the serial cable +could compromise your machine! Ideally, you would take input from your +trusted keyboard, and only send the output over the serial cable via +e.g. disabling getty and using: + +.. code:: bash + + script -f /dev/ttyS0 + + + +You don’t even need to connect the TX pin. diff --git a/developer/debugging/test-bench.md b/developer/debugging/test-bench.md deleted file mode 100644 index ca48a5eab6..0000000000 --- a/developer/debugging/test-bench.md +++ /dev/null @@ -1,235 +0,0 @@ ---- -lang: en -layout: doc -permalink: /doc/test-bench/ -redirect_from: -- /en/doc/test-bench/ -- /doc/TestBench/ -- /wiki/TestBench/ -ref: 44 -title: How to set up a test bench ---- - -This guide shows how to set up simple test bench that automatically test your code you're about to push. It is written especially for `core3` branch of `core-admin.git` repo, but some ideas are universal. - -We will set up a spare machine (bare metal, not a virtual) that will be hosting our experimental Dom0. We will communicate with it via Ethernet and SSH. This tutorial assumes you are familiar with [QubesBuilder](/doc/qubes-builder/) and you have it set up and running flawlessly. - -> **Notice:** -> This setup intentionally weakens some security properties in the testing system. So make sure you understand the risks and use exclusively for testing. - -## Setting up the Machine - -### Install ISO -First, do a clean install from the `.iso` [you built](/doc/qubes-iso-building/) or grabbed elsewhere (for example [here](https://forum.qubes-os.org/t/qubesos-4-1-alpha-signed-weekly-builds/3601)). - -### Enabling Network Access in Dom0 - -Internet access is intentionally disabled by default in dom0. But to ease the deployment process we will give it access. The following steps should be done in `dom0`. - -> **Note:** the following assume you have only one network card. If you have two, pick one and leave the other attached to `sys-net`. - -1. Remove the network card (PCI device) from `sys-net` -2. Restart your computer (for the removal to take effect) -3. Install `dhcp-client` and `openssh-server` on your testbench's dom0. -4. Save the following script in `/home/user/bin/dom0_network.sh` and make it executable. It should enable your network card in dom0. *Be sure to adjust the script's variables to suit your needs.* - - ```bash - #!/bin/sh - - # adjust this for your NIC (run lspci) - BDF=0000:02:00.0 - - # adjust this for your network driver - DRIVER=e1000e - - prog=$(basename $0) - - pciunbind() { - local path - path=/sys/bus/pci/devices/${1}/driver/unbind - if ! [ -w ${path} ]; then - echo "${prog}: Device ${1} not bound" - return 1 - fi - echo -n ${1} >${path} - } - - pcibind() { - local path - path=/sys/bus/pci/drivers/${2}/bind - if ! [ -w ${path} ]; then - echo "${prog}: Driver ${2} not found" - return 1 - fi - echo ${1} >${path} - } - - pciunbind ${BDF} - pcibind ${BDF} ${DRIVER} - - sleep 1 - dhclient - ``` - -5. Configure your DHCP server so your testbench gets static IP and connect your machine to your local network. You should ensure that your testbench can reach the Internet. - -6. You'll need to run the above script on every startup. To automate this save the following systemd service `/etc/systemd/system/dom0-network-direct.service` - - ``` - [Unit] - Description=Connect network to dom0 - - [Service] - Type=oneshot - ExecStart=/home/user/bin/dom0_network.sh - - [Install] - WantedBy=multi-user.target - ``` - -7. Then, enable and start the SSH Server and the script on boot: - - ```bash - sudo systemctl enable sshd - sudo systemctl start sshd - - sudo systemctl enable dom0-network-direct - sudo systemctl start dom0-network-direct - ``` - -> **Note:** If you want to install additional software in dom0 and your only network card was assigned to dom0, then _instead_ of the usual `sudo qubes-dom0-update ` now you run `sudo dnf --setopt=reposdir=/etc/yum.repos.d install `. - -### Install Tests and Their Dependencies - -A regular Qubes installation isn't ready to run the full suite of tests. For example, in order to run the [Split GPG tests](https://github.com/QubesOS/qubes-app-linux-split-gpg/blob/4bc201bb70c011119eed19df25dc5b46120d04ed/tests/splitgpg/tests.py) you need to have the `qubes-gpg-split-tests` package installed in your app qubes. - -Because of the above reason, some additional configurations need to be done to your testing environment. This can be done in an automated manner with the help of the [Salt](/doc/salt) configuration that provisions the [automated testing environment](/doc/automated-tests/). - -The following commands should work for you, but do keep in mind that the provisioning scripts are designed for the [openQA environment](https://openqa.qubes-os.org/) and not your specific local testing system. Run the following in `dom0`: - - ```bash - # For future reference the following commands are an adaptation of - # https://github.com/marmarek/openqa-tests-qubesos/blob/master/tests/update.pm - - # Install git - sudo qubes-dom0-update git || sudo dnf --setopt=reposdir=/etc/yum.repos.d install git - - # Download the openQA automated testing environment Salt configuration - git clone https://github.com/marmarek/openqa-tests-qubesos/ - cd openqa-tests-qubesos/extra-files - sudo cp -a system-tests/ /srv/salt/ - sudo qubesctl top.enable system-tests - - # Install the same configuration as the one in openQA - QUBES_VERSION=4.1 - PILLAR_DIR=/srv/pillar/base/update - sudo mkdir -p $PILLAR_DIR - printf 'update:\n qubes_ver: '$QUBES_VERSION'\n' | sudo tee $PILLAR_DIR/init.sls - printf "base:\n '*':\n - update\n" | sudo tee $PILLAR_DIR/init.top - sudo qubesctl top.enable update pillar=True - - # Apply states to dom0 and VMs - # NOTE: These commands can take several minutes (if not more) without showing output - sudo qubesctl --show-output state.highstate - sudo qubesctl --max-concurrency=2 --skip-dom0 --templates --show-output state.highstate - ``` - -## Development VM - -### SSH - -Arrange firewall so you can reach the testbench from your `qubes-dev` VM. Generate SSH key in `qubes-dev`: - -~~~ -ssh-keygen -t ecdsa -b 521 -~~~ - -Add the following section in `.ssh/config` in `qubes-dev`: - -~~~ -Host testbench - # substitute username in testbench - User user - # substitute address of your testbench - HostName 192.168.123.45 -~~~ - -#### Passwordless SSH Login - -To log to your testbench without entering password every time, copy your newly generated public key (`id_ecdsa.pub`) to `~/.ssh/authorized_keys` on your testbench. You can do this easily by running this command on `qubes-dev`: `ssh-copy-id -i ~/.ssh/id_ecdsa.pub user@192.168.123.45` (substituting with the actual username address of your testbench). - -### Scripting - -This step is optional, but very helpful. Put these scripts somewhere in your `${PATH}`, like `/usr/local/bin`. - -`qtb-runtests`: - -```bash -#!/bin/sh - -ssh testbench python -m qubes.tests.run -``` - -`qtb-install`: - -```bash -#!/bin/sh - -TMPDIR=/tmp/qtb-rpms - -if [ $# -eq 0 ]; then - echo "usage: $(basename $0) ..." - exit 2 -fi - -set -e - -ssh testbench mkdir -p "${TMPDIR}" -scp "${@}" testbench:"${TMPDIR}" || echo "check if you have 'scp' installed on your testbench" - -while [ $# -gt 0 ]; do - ssh testbench sudo rpm -i --replacepkgs --replacefiles "${TMPDIR}/$(basename ${1})" - shift -done -``` - -`qtb-iterate`: - -```bash -#!/bin/sh - -set -e - -# substitute path to your builder installation -pushd ${HOME}/builder >/dev/null - -# the following are needed only if you have sources outside builder -#rm -rf qubes-src/core-admin -#make COMPONENTS=core-admin get-sources - -make core-admin -qtb-install qubes-src/core-admin/rpm/x86_64/qubes-core-dom0-*.rpm -qtb-runtests -``` - -### Hooking git - -I (woju) have those two git hooks. They ensure tests are passing (or are marked as expected failure) when committing and pushing. For committing it is only possible to run tests that may be executed from git repo (even if the rest were available, I probably wouldn't want to do that). For pushing, I also install RPM and run tests on testbench. - -`core-admin/.git/hooks/pre-commit`: (you may retain also the default hook, here omitted for readability) - -```bash -#!/bin/sh - -set -e - -python -c "import sys, qubes.tests.run; sys.exit(not qubes.tests.run.main())" -``` - -`core-admin/.git/hooks/pre-push`: - -```bash -#!/bin/sh - -exec qtb-iterate -``` diff --git a/developer/debugging/test-bench.rst b/developer/debugging/test-bench.rst new file mode 100644 index 0000000000..c9356dd6e5 --- /dev/null +++ b/developer/debugging/test-bench.rst @@ -0,0 +1,306 @@ +========================== +How to set up a test bench +========================== + + +This guide shows how to set up simple test bench that automatically test +your code you’re about to push. It is written especially for ``core3`` +branch of ``core-admin.git`` repo, but some ideas are universal. + +We will set up a spare machine (bare metal, not a virtual) that will be +hosting our experimental Dom0. We will communicate with it via Ethernet +and SSH. This tutorial assumes you are familiar with +:doc:`QubesBuilder ` and you have it set up and +running flawlessly. + + **Notice:** This setup intentionally weakens some security properties + in the testing system. So make sure you understand the risks and use + exclusively for testing. + +Setting up the Machine +---------------------- + + +Install ISO +^^^^^^^^^^^ + + +First, do a clean install from the ``.iso`` :doc:`you built ` or grabbed elsewhere (for example +`here `__). + +Enabling Network Access in Dom0 +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + + +Internet access is intentionally disabled by default in dom0. But to +ease the deployment process we will give it access. The following steps +should be done in ``dom0``. + + **Note:** the following assume you have only one network card. If you + have two, pick one and leave the other attached to ``sys-net``. + +1. Remove the network card (PCI device) from ``sys-net`` + +2. Restart your computer (for the removal to take effect) + +3. Install ``dhcp-client`` and ``openssh-server`` on your testbench’s + dom0. + +4. Save the following script in ``/home/user/bin/dom0_network.sh`` and + make it executable. It should enable your network card in dom0. *Be sure to adjust the script’s variables to suit your needs.* + + .. code:: bash + + #!/bin/sh + + # adjust this for your NIC (run lspci) + BDF=0000:02:00.0 + + # adjust this for your network driver + DRIVER=e1000e + + prog=$(basename $0) + + pciunbind() { + local path + path=/sys/bus/pci/devices/${1}/driver/unbind + if ! [ -w ${path} ]; then + echo "${prog}: Device ${1} not bound" + return 1 + fi + echo -n ${1} >${path} + } + + pcibind() { + local path + path=/sys/bus/pci/drivers/${2}/bind + if ! [ -w ${path} ]; then + echo "${prog}: Driver ${2} not found" + return 1 + fi + echo ${1} >${path} + } + + pciunbind ${BDF} + pcibind ${BDF} ${DRIVER} + + sleep 1 + dhclient + + +5. Configure your DHCP server so your testbench gets static IP and + connect your machine to your local network. You should ensure that + your testbench can reach the Internet. + +6. You’ll need to run the above script on every startup. To automate + this save the following systemd service + ``/etc/systemd/system/dom0-network-direct.service`` + + .. code:: bash + + [Unit] + Description=Connect network to dom0 + + [Service] + Type=oneshot + ExecStart=/home/user/bin/dom0_network.sh + + [Install] + WantedBy=multi-user.target + + + +7. Then, enable and start the SSH Server and the script on boot: + + .. code:: bash + + sudo systemctl enable sshd + sudo systemctl start sshd + + sudo systemctl enable dom0-network-direct + sudo systemctl start dom0-network-direct + + + + + **Note:** If you want to install additional software in dom0 and your + only network card was assigned to dom0, then *instead* of the usual + ``sudo qubes-dom0-update `` now you run + ``sudo dnf --setopt=reposdir=/etc/yum.repos.d install ``. + +Install Tests and Their Dependencies +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + + +A regular Qubes installation isn’t ready to run the full suite of tests. +For example, in order to run the `Split GPG tests `__ +you need to have the ``qubes-gpg-split-tests`` package installed in your +app qubes. + +Because of the above reason, some additional configurations need to be +done to your testing environment. This can be done in an automated +manner with the help of the :doc:`Salt ` configuration that +provisions the :doc:`automated testing environment `. + +The following commands should work for you, but do keep in mind that the +provisioning scripts are designed for the `openQA environment `__ and not your specific +local testing system. Run the following in ``dom0``: + +.. code:: bash + + # For future reference the following commands are an adaptation of + # https://github.com/marmarek/openqa-tests-qubesos/blob/master/tests/update.pm + + # Install git + sudo qubes-dom0-update git || sudo dnf --setopt=reposdir=/etc/yum.repos.d install git + + # Download the openQA automated testing environment Salt configuration + git clone https://github.com/marmarek/openqa-tests-qubesos/ + cd openqa-tests-qubesos/extra-files + sudo cp -a system-tests/ /srv/salt/ + sudo qubesctl top.enable system-tests + + # Install the same configuration as the one in openQA + QUBES_VERSION=4.1 + PILLAR_DIR=/srv/pillar/base/update + sudo mkdir -p $PILLAR_DIR + printf 'update:\n qubes_ver: '$QUBES_VERSION'\n' | sudo tee $PILLAR_DIR/init.sls + printf "base:\n '*':\n - update\n" | sudo tee $PILLAR_DIR/init.top + sudo qubesctl top.enable update pillar=True + + # Apply states to dom0 and VMs + # NOTE: These commands can take several minutes (if not more) without showing output + sudo qubesctl --show-output state.highstate + sudo qubesctl --max-concurrency=2 --skip-dom0 --templates --show-output state.highstate + + +Development VM +-------------- + + +SSH +^^^ + + +Arrange firewall so you can reach the testbench from your ``qubes-dev`` +VM. Generate SSH key in ``qubes-dev``: + +.. code:: bash + + ssh-keygen -t ecdsa -b 521 + + + +Add the following section in ``.ssh/config`` in ``qubes-dev``: + +.. code:: bash + + Host testbench + # substitute username in testbench + User user + # substitute address of your testbench + HostName 192.168.123.45 + + + +Passwordless SSH Login +^^^^^^^^^^^^^^^^^^^^^^ + + +To log to your testbench without entering password every time, copy your +newly generated public key (``id_ecdsa.pub``) to +``~/.ssh/authorized_keys`` on your testbench. You can do this easily by +running this command on ``qubes-dev``: +``ssh-copy-id -i ~/.ssh/id_ecdsa.pub user@192.168.123.45`` (substituting +with the actual username address of your testbench). + +Scripting +^^^^^^^^^ + + +This step is optional, but very helpful. Put these scripts somewhere in +your ``${PATH}``, like ``/usr/local/bin``. + +``qtb-runtests``: + +.. code:: bash + + #!/bin/sh + + ssh testbench python -m qubes.tests.run + + +``qtb-install``: + +.. code:: bash + + #!/bin/sh + + TMPDIR=/tmp/qtb-rpms + + if [ $# -eq 0 ]; then + echo "usage: $(basename $0) ..." + exit 2 + fi + + set -e + + ssh testbench mkdir -p "${TMPDIR}" + scp "${@}" testbench:"${TMPDIR}" || echo "check if you have 'scp' installed on your testbench" + + while [ $# -gt 0 ]; do + ssh testbench sudo rpm -i --replacepkgs --replacefiles "${TMPDIR}/$(basename ${1})" + shift + done + + +``qtb-iterate``: + +.. code:: bash + + #!/bin/sh + + set -e + + # substitute path to your builder installation + pushd ${HOME}/builder >/dev/null + + # the following are needed only if you have sources outside builder + #rm -rf qubes-src/core-admin + #make COMPONENTS=core-admin get-sources + + make core-admin + qtb-install qubes-src/core-admin/rpm/x86_64/qubes-core-dom0-*.rpm + qtb-runtests + + +Hooking git +^^^^^^^^^^^ + + +I (woju) have those two git hooks. They ensure tests are passing (or are +marked as expected failure) when committing and pushing. For committing +it is only possible to run tests that may be executed from git repo +(even if the rest were available, I probably wouldn’t want to do that). +For pushing, I also install RPM and run tests on testbench. + +``core-admin/.git/hooks/pre-commit``: (you may retain also the default +hook, here omitted for readability) + +.. code:: bash + + #!/bin/sh + + set -e + + python -c "import sys, qubes.tests.run; sys.exit(not qubes.tests.run.main())" + + +``core-admin/.git/hooks/pre-push``: + +.. code:: bash + + #!/bin/sh + + exec qtb-iterate + diff --git a/developer/debugging/vm-interface.md b/developer/debugging/vm-interface.md deleted file mode 100644 index 730c76e720..0000000000 --- a/developer/debugging/vm-interface.md +++ /dev/null @@ -1,225 +0,0 @@ ---- -lang: en -layout: doc -permalink: /doc/vm-interface/ -redirect_from: -- /en/doc/vm-interface/ -- /doc/VMInterface/ -- /doc/SystemDoc/VMInterface/ -- /wiki/SystemDoc/VMInterface/ -ref: 47 -title: Qube configuration interface ---- - -Qubes VM have some settings set by dom0 based on VM settings. There are multiple configuration channels, which includes: - -- QubesDB -- XenStore (in Qubes 2, data the same as in QubesDB, keys without leading `/`) -- Qubes RPC (called at VM startup, or when configuration changed) -- GUI protocol - -## QubesDB - -### Keys exposed by dom0 to VM - -- `/qubes-vm-type` - VM type, the same as `type` field in `qvm-prefs`. One of `AppVM`, `ProxyVM`, `NetVM`, `TemplateVM`, `HVM`, `TemplateHVM` -- `/qubes-vm-updatable` - flag whether VM is updatable (whether changes in root.img will survive VM restart). One of `True`, `False` -- `/qubes-vm-persistence` - what data do persist between VM restarts: - - `full` - all disks - - `rw-only` - only `/rw` disk - - `none` - none -- `/qubes-timezone` - name of timezone based on dom0 timezone. For example `Europe/Warsaw` -- `/qubes-keyboard` (deprecated in R4.1) - keyboard layout based on dom0 layout. Its syntax is suitable for `xkbcomp` command (after expanding escape sequences like `\n` or `\t`). This is meant only as some default value, VM can ignore this option and choose its own keyboard layout (this is what keyboard setting from Qubes Manager does). This entry is created as part of gui-daemon initialization (so not available when gui-daemon disabled, or not started yet). -- `/keyboard-layout` - keyboard layout based on GuiVM layout. Its syntax can be `layout+variant+options`, `layout+variant`, `layout++options` or simply `layout`. For example, `fr+oss`, `pl++compose:caps` or `fr`. This is meant only as some default value, VM can ignore this option and choose its own keyboard layout (this is what keyboard setting from Qubes Manager does). -- `/qubes-debug-mode` - flag whether VM has debug mode enabled (qvm-prefs setting). One of `1`, `0` -- `/qubes-service/SERVICE_NAME` - subtree for VM services controlled from dom0 (using the `qvm-service` command or Qubes Manager). One of `1`, `0`. Note that not every service will be listed here, if entry is missing, it means "use VM default". A list of currently supported services is in the `qvm-service` man page. -- `/qubes-netmask` - network mask (only when VM has netvm set); currently hardcoded "255.255.255.0" -- `/qubes-ip` - IP address for this VM (only when VM has netvm set) -- `/qubes-gateway` - default gateway IP (only when VM has netvm set); VM should add host route to this address directly via eth0 (or whatever default interface name is) -- `/qubes-primary-dns` - primary DNS address (only when VM has netvm set) -- `/qubes-secondary-dns` - secondary DNS address (only when VM has netvm set) -- `/qubes-netvm-gateway` - same as `qubes-gateway` in connected VMs (only when VM serves as network backend - ProxyVM and NetVM) -- `/qubes-netvm-netmask` - same as `qubes-netmask` in connected VMs (only when VM serves as network backend - ProxyVM and NetVM) -- `/qubes-netvm-network` - network address (only when VM serves as network backend - ProxyVM and NetVM); can be also calculated from qubes-netvm-gateway and qubes-netvm-netmask -- `/qubes-netvm-primary-dns` - same as `qubes-primary-dns` in connected VMs (only when VM serves as network backend - ProxyVM and NetVM); traffic sent to this IP on port 53 should be redirected to primary DNS server -- `/qubes-netvm-secondary-dns` - same as `qubes-secondary-dns` in connected VMs (only when VM serves as network backend - ProxyVM and NetVM); traffic sent to this IP on port 53 should be redirected to secondary DNS server -- `/guivm-windows-prefix` - title prefix for any window not originating from another qube. This means windows of applications running in GuiVM itself - -#### Firewall rules in 3.x - -QubesDB is also used to configure firewall in ProxyVMs. Rules are stored in -separate key for each target VM. Entries: - -- `/qubes-iptables` - control entry - dom0 writing `reload` here signals `qubes-firewall` service to reload rules -- `/qubes-iptables-header` - rules not related to any particular VM, should be applied before domains rules -- `/qubes-iptables-domainrules/NNN` - rules for domain `NNN` (arbitrary number) -in `iptables-save` format. Rules are self-contained - fill `FORWARD` iptables -chain and contains all required matches (source IP address etc), as well as -final default action (`DROP`/`ACCEPT`) - -VM after applying rules may signal some error, writing a message to -`/qubes-iptables-error` key. This does not exclude any other way of -communicating problems - like a popup. - -#### Firewall rules in 4.x - -QubesDB is also used to configure firewall in ProxyVMs. Each rule is stored as -a separate entry, grouped on target VM: - -- `/qubes-firewall/SOURCE_IP` - base tree under which rules are placed. All -rules there should be applied to filter traffic coming from `SOURCE_IP`. This -can be either IPv4 or IPv6 address. -Dom0 will do an empty write to this top level entry after finishing rules -update, so VM can setup a watch here to trigger rules reload. -- `/qubes-firewall/SOURCE_IP/policy` - default action if no rule matches: -`drop` or `accept`. -- `/qubes-firewall/SOURCE_IP/NNNN` - rule number `NNNN` - decimal number, - padded with zeros. Se below for rule format. All the rules should be - applied in order of rules implied by those numbers. Note that QubesDB - itself does not impose any ordering (you need to sort the rules after - retrieving them). The first rule has number `0000`. - -Each rule is a single QubesDB entry, consisting of pairs `key=value` separated -by space. QubesDB enforces limit on a single entry length - 3072 bytes. -Possible options for a single rule: - -- `action`, values: `accept`, `drop`; this is present in every rule -- `dst4`, value: destination IPv4 address with a mask; for example: `192.168.0.0/24` -- `dst6`, value: destination IPv6 address with a mask; for example: `2000::/3` -- `dsthost`, value: DNS hostname of destination host -- `proto`, values: `tcp`, `udp`, `icmp` -- `specialtarget`, value: One of predefined target, currently defined values: - - `dns` - such option should match DNS traffic to default DNS server (but - not any DNS server), on both TCP and UDP -- `dstports`, value: destination ports range separated with `-`, valid only - together with `proto=tcp` or `proto=udp`; for example `1-1024`, `80-80` -- `icmptype`, value: numeric (decimal) icmp message type, for example `8` for - echo request, valid only together with `proto=icmp` -- `dpi`, value: Deep Packet Inspection protocol (like: HTTP, SSL, SMB, SSH, SMTP) or the default 'NO' as no DPI, only packet filtering - -Options must appear in the rule in the order listed above. Duplicated options -are forbidden. - -A rule matches only when all predicates match. Only one of `dst4`, `dst6` or -`dsthost` can be used in a single rule. - -If tool applying firewall encounters any parse error (unknown option, invalid -value, duplicated option, etc), it should drop all the traffic coming from that `SOURCE_IP`, -regardless of properly parsed rules. - -Example valid rules: - -- `action=accept dst4=8.8.8.8 proto=udp dstports=53-53` -- `action=drop dst6=2a00:1450:4000::/37 proto=tcp` -- `action=accept specialtarget=dns` -- `action=drop proto=tcp specialtarget=dns` - drop DNS queries sent using TCP -- `action=drop` - -### Keys set by VM for passing info to dom0 - -- `memory/meminfo` (**xenstore**) - used memory (updated by qubes-meminfo-writer), input information for qmemman; - - Qubes 3.x format: 6 lines (EOL encoded as `\n`), each in format "FIELD: VALUE kB"; fields: `MemTotal`, `MemFree`, `Buffers`, `Cached`, `SwapTotal`, `SwapFree`; meaning the same as in `/proc/meminfo` in Linux. - - Qubes 4.0+ format: used memory size in the VM, in kbytes -- `/qubes-block-devices` - list of block devices exposed by this VM, each device (subdirectory) should be named in a way that VM can attach the device based on it. Each should contain these entries: - - `desc` - device description (ASCII text) - - `size` - device size in bytes - - `mode` - default connection mode; `r` for read-only, `w` for read-write -- `/qubes-usb-devices` - list of USB devices exposed by this VM, each device (subdirectory) should contain: - - `desc` - device description (ASCII text) - - `usb-ver` - USB version (1, 2 or 3) - -## Qubes RPC - -Services called by dom0 to provide some VM configuration: - -- `qubes.SetMonitorLayout` - provide list of monitors, one per line. Each line contains four numbers: `width height X Y width_mm height_mm` (physical dimensions - `width_mm` and `height_mm` - are optional) -- `qubes.WaitForSession` - called to wait for full VM startup -- `qubes.GetAppmenus` - receive appmenus from given VM (template); TODO: describe format here -- `qubes.GetImageRGBA` - receive image/application icon. Protocol: - - 1. Caller sends name of requested icon. This can be one of: - * `xdgicon:NAME` - search for NAME in standard icons theme - * `-` - get icon data from stdin (the caller), can be prefixed with format name, for example `png:-` - * file name - 2. The service responds with image dimensions: width and height as - decimal numbers, separated with space and with EOL marker at the and; then - image data in RGBA format (32 bits per pixel) -- `qubes.SetDateTime` - set VM time, called periodically by dom0 (can be - triggered manually from dom0 by calling `qvm-sync-clock`). The service - receives one line at stdin - time in format of `date -u -Iseconds`, for - example `2015-07-31T16:10:43+0000`. -- `qubes.SetGuiMode` - called in HVM to switch between fullscreen and seamless - GUI mode. The service receives a single word on stdin - either `FULLSCREEN` - or `SEAMLESS` -- `qubes.ResizeDisk` - called to inform that underlying disk was resized. - Name of disk image is passed on standard input (`root`, `private`, `volatile`, - or other). This is used starting with Qubes 4.0. - -Other Qrexec services installed by default: - -- `qubes.Backup` - store Qubes backup. The service receives location chosen by - the user (one line, terminated by `\n`), the backup archive ([description of - backup format](/doc/BackupEmergencyRestoreV2/)) -- `qubes.DetachPciDevice` - service called in reaction to `qvm-pci -d` call on - running VM. The service receives one word - BDF of device to detach. When the - service call ends, the device will be detached -- `qubes.Filecopy` - receive some files from other VM. Files sent in [qfile format](/doc/qfilecopy/) -- `qubes.OpenInVM` - open a file in called VM. Service receives a single file on stdin (in - [qfile format](/doc/qfilecopy/). After a file viewer/editor is terminated, if - the file was modified, can be sent back (just raw content, without any - headers); otherwise service should just terminate without sending anything. - This service is used by both `qvm-open-in-vm` and `qvm-open-in-dvm` tools. When - called in DispVM, service termination will trigger DispVM cleanup. -- `qubes.Restore` - retrieve Qubes backup. The service receives backup location - entered by the user (one line, terminated by `\n`), then should output backup - archive in [qfile format](/doc/qfilecopy/) (core-agent-linux component contains - `tar2qfile` utility to do the conversion) -- `qubes.SelectDirectory`, `qubes.SelectFile` - services which should show - file/directory selection dialog and return (to stdout) a single line - containing selected path, or nothing in the case of cancellation -- `qubes.SuspendPre` - service called in every VM with PCI device attached just - before system suspend -- `qubes.SuspendPost` - service called in every VM with PCI device attached just - after system resume -- `qubes.SyncNtpClock` - service called to trigger network time synchronization. - Service should synchronize local VM time and terminate when done. -- `qubes.WindowIconUpdater` - service called by VM to send icons of individual - windows. The protocol there is simple one direction stream: VM sends window ID - followed by icon in `qubes.GetImageRGBA` format, then next window ID etc. VM - can send icon for the same window multiple times to replace previous one (for - example for animated icons) -- `qubes.VMShell` - call any command in the VM; the command(s) is passed one per line - - `qubes.VMShell+WaitForSession` waits for full VM startup first -- `qubes.VMExec` - call any command in the VM, without using shell, the command - needs to be passed as argument and encoded as follows: - - the executable name and arguments are separated by `+` - - everything except alphanumeric characters, `.` and `_` needs to be - escaped - - bytes are escaped as `-HH` (where `HH` is hex code, capital letters only) - - `-` itself can be escaped as `--` - - example: to run `ls -a /home/user`, use - `qubes.VMExec+ls+--a+-2Fhome-2Fuser` -- `qubes.VMExecGUI` - a variant of `qubes.VMExec` that waits for full VM - startup first - -Services called in GuiVM: - -- `policy.Ask`, `policy.Notify` - confirmation prompt and notifications for -Qubes RPC calls, see [qrexec-policy implementation](/doc/qrexec-internals/#qrexec-policy-implementation) -for a detailed description. - -Currently Qubes still calls few tools in VM directly, not using service -abstraction. This will change in the future. Those tools are: - -- `/usr/lib/qubes/qubes-download-dom0-updates.sh` - script to download updates (or new packages to be installed) for dom0 (`qubes-dom0-update` tool) -- `date -u -Iseconds` - called directly to retrieve time after calling `qubes.SyncNtpClock` service (`qvm-sync-clock` tool) -- `nm-online -x` - called before `qubes.SyncNtpClock` service call by `qvm-sync-clock` tool -- `resize2fs` - called to resize filesystem on /rw partition by `qvm-grow-private` tool -- `gpk-update-viewer` - called by Qubes Manager to display available updates in a TemplateVM -- `systemctl start qubes-update-check.timer` (and similarly stop) - called when enabling/disabling updates checking in given VM (`qubes-update-check` [qvm-service](/doc/qubes-service/)) - -Additionally, automatic tests extensively run various commands directly in VMs. We do not plan to change that. - -## GUI protocol - -GUI initialization includes passing the whole screen dimensions from dom0 to VM. This will most likely be overwritten by qubes.SetMonitorLayout Qubes RPC call. diff --git a/developer/debugging/vm-interface.rst b/developer/debugging/vm-interface.rst new file mode 100644 index 0000000000..9833a6208f --- /dev/null +++ b/developer/debugging/vm-interface.rst @@ -0,0 +1,441 @@ +============================ +Qube configuration interface +============================ + + +Qubes VM have some settings set by dom0 based on VM settings. There are +multiple configuration channels, which includes: + +- QubesDB + +- XenStore (in Qubes 2, data the same as in QubesDB, keys without + leading ``/``) + +- Qubes RPC (called at VM startup, or when configuration changed) + +- GUI protocol + + + +QubesDB +------- + + +Keys exposed by dom0 to VM +^^^^^^^^^^^^^^^^^^^^^^^^^^ + + +- ``/qubes-vm-type`` - VM type, the same as ``type`` field in + ``qvm-prefs``. One of ``AppVM``, ``ProxyVM``, ``NetVM``, + ``TemplateVM``, ``HVM``, ``TemplateHVM`` + +- ``/qubes-vm-updatable`` - flag whether VM is updatable (whether + changes in root.img will survive VM restart). One of ``True``, + ``False`` + +- ``/qubes-vm-persistence`` - what data do persist between VM restarts: + + - ``full`` - all disks + + - ``rw-only`` - only ``/rw`` disk + + - ``none`` - none + + + +- ``/qubes-timezone`` - name of timezone based on dom0 timezone. For + example ``Europe/Warsaw`` + +- ``/qubes-keyboard`` (deprecated in R4.1) - keyboard layout based on + dom0 layout. Its syntax is suitable for ``xkbcomp`` command (after + expanding escape sequences like ``\n`` or ``\t``). This is meant only + as some default value, VM can ignore this option and choose its own + keyboard layout (this is what keyboard setting from Qubes Manager + does). This entry is created as part of gui-daemon initialization (so + not available when gui-daemon disabled, or not started yet). + +- ``/keyboard-layout`` - keyboard layout based on GuiVM layout. Its + syntax can be ``layout+variant+options``, ``layout+variant``, + ``layout++options`` or simply ``layout``. For example, ``fr+oss``, + ``pl++compose:caps`` or ``fr``. This is meant only as some default + value, VM can ignore this option and choose its own keyboard layout + (this is what keyboard setting from Qubes Manager does). + +- ``/qubes-debug-mode`` - flag whether VM has debug mode enabled + (qvm-prefs setting). One of ``1``, ``0`` + +- ``/qubes-service/SERVICE_NAME`` - subtree for VM services controlled + from dom0 (using the ``qvm-service`` command or Qubes Manager). One + of ``1``, ``0``. Note that not every service will be listed here, if + entry is missing, it means “use VM default”. A list of currently + supported services is in the ``qvm-service`` man page. + +- ``/qubes-netmask`` - network mask (only when VM has netvm set); + currently hardcoded “255.255.255.0” + +- ``/qubes-ip`` - IP address for this VM (only when VM has netvm set) + +- ``/qubes-gateway`` - default gateway IP (only when VM has netvm set); + VM should add host route to this address directly via eth0 (or + whatever default interface name is) + +- ``/qubes-primary-dns`` - primary DNS address (only when VM has netvm + set) + +- ``/qubes-secondary-dns`` - secondary DNS address (only when VM has + netvm set) + +- ``/qubes-netvm-gateway`` - same as ``qubes-gateway`` in connected VMs + (only when VM serves as network backend - ProxyVM and NetVM) + +- ``/qubes-netvm-netmask`` - same as ``qubes-netmask`` in connected VMs + (only when VM serves as network backend - ProxyVM and NetVM) + +- ``/qubes-netvm-network`` - network address (only when VM serves as + network backend - ProxyVM and NetVM); can be also calculated from + qubes-netvm-gateway and qubes-netvm-netmask + +- ``/qubes-netvm-primary-dns`` - same as ``qubes-primary-dns`` in + connected VMs (only when VM serves as network backend - ProxyVM and + NetVM); traffic sent to this IP on port 53 should be redirected to + primary DNS server + +- ``/qubes-netvm-secondary-dns`` - same as ``qubes-secondary-dns`` in + connected VMs (only when VM serves as network backend - ProxyVM and + NetVM); traffic sent to this IP on port 53 should be redirected to + secondary DNS server + +- ``/guivm-windows-prefix`` - title prefix for any window not + originating from another qube. This means windows of applications + running in GuiVM itself + + + +Firewall rules in 3.x +^^^^^^^^^^^^^^^^^^^^^ + + +QubesDB is also used to configure firewall in ProxyVMs. Rules are stored +in separate key for each target VM. Entries: + +- ``/qubes-iptables`` - control entry - dom0 writing ``reload`` here + signals ``qubes-firewall`` service to reload rules + +- ``/qubes-iptables-header`` - rules not related to any particular VM, + should be applied before domains rules + +- ``/qubes-iptables-domainrules/NNN`` - rules for domain ``NNN`` + (arbitrary number) in ``iptables-save`` format. Rules are + self-contained - fill ``FORWARD`` iptables chain and contains all + required matches (source IP address etc), as well as final default + action (``DROP``/``ACCEPT``) + + + +VM after applying rules may signal some error, writing a message to +``/qubes-iptables-error`` key. This does not exclude any other way of +communicating problems - like a popup. + +Firewall rules in 4.x +^^^^^^^^^^^^^^^^^^^^^ + + +QubesDB is also used to configure firewall in ProxyVMs. Each rule is +stored as a separate entry, grouped on target VM: + +- ``/qubes-firewall/SOURCE_IP`` - base tree under which rules are + placed. All rules there should be applied to filter traffic coming + from ``SOURCE_IP``. This can be either IPv4 or IPv6 address. Dom0 + will do an empty write to this top level entry after finishing rules + update, so VM can setup a watch here to trigger rules reload. + +- ``/qubes-firewall/SOURCE_IP/policy`` - default action if no rule + matches: ``drop`` or ``accept``. + +- ``/qubes-firewall/SOURCE_IP/NNNN`` - rule number ``NNNN`` - decimal + number, padded with zeros. Se below for rule format. All the rules + should be applied in order of rules implied by those numbers. Note + that QubesDB itself does not impose any ordering (you need to sort + the rules after retrieving them). The first rule has number ``0000``. + + + +Each rule is a single QubesDB entry, consisting of pairs ``key=value`` +separated by space. QubesDB enforces limit on a single entry length - +3072 bytes. Possible options for a single rule: + +- ``action``, values: ``accept``, ``drop``; this is present in every + rule + +- ``dst4``, value: destination IPv4 address with a mask; for example: + ``192.168.0.0/24`` + +- ``dst6``, value: destination IPv6 address with a mask; for example: + ``2000::/3`` + +- ``dsthost``, value: DNS hostname of destination host + +- ``proto``, values: ``tcp``, ``udp``, ``icmp`` + +- ``specialtarget``, value: One of predefined target, currently defined + values: + + - ``dns`` - such option should match DNS traffic to default DNS + server (but not any DNS server), on both TCP and UDP + + + +- ``dstports``, value: destination ports range separated with ``-``, + valid only together with ``proto=tcp`` or ``proto=udp``; for example + ``1-1024``, ``80-80`` + +- ``icmptype``, value: numeric (decimal) icmp message type, for example + ``8`` for echo request, valid only together with ``proto=icmp`` + +- ``dpi``, value: Deep Packet Inspection protocol (like: HTTP, SSL, + SMB, SSH, SMTP) or the default ‘NO’ as no DPI, only packet filtering + + + +Options must appear in the rule in the order listed above. Duplicated +options are forbidden. + +A rule matches only when all predicates match. Only one of ``dst4``, +``dst6`` or ``dsthost`` can be used in a single rule. + +If tool applying firewall encounters any parse error (unknown option, +invalid value, duplicated option, etc), it should drop all the traffic +coming from that ``SOURCE_IP``, regardless of properly parsed rules. + +Example valid rules: + +- ``action=accept dst4=8.8.8.8 proto=udp dstports=53-53`` + +- ``action=drop dst6=2a00:1450:4000::/37 proto=tcp`` + +- ``action=accept specialtarget=dns`` + +- ``action=drop proto=tcp specialtarget=dns`` - drop DNS queries sent + using TCP + +- ``action=drop`` + + + +Keys set by VM for passing info to dom0 +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + + +- ``memory/meminfo`` (**xenstore**) - used memory (updated by + qubes-meminfo-writer), input information for qmemman; + + - Qubes 3.x format: 6 lines (EOL encoded as ``\n``), each in format + “FIELD: VALUE kB”; fields: ``MemTotal``, ``MemFree``, ``Buffers``, + ``Cached``, ``SwapTotal``, ``SwapFree``; meaning the same as in + ``/proc/meminfo`` in Linux. + + - Qubes 4.0+ format: used memory size in the VM, in kbytes + + + +- ``/qubes-block-devices`` - list of block devices exposed by this VM, + each device (subdirectory) should be named in a way that VM can + attach the device based on it. Each should contain these entries: + + - ``desc`` - device description (ASCII text) + + - ``size`` - device size in bytes + + - ``mode`` - default connection mode; ``r`` for read-only, ``w`` for + read-write + + + +- ``/qubes-usb-devices`` - list of USB devices exposed by this VM, each + device (subdirectory) should contain: + + - ``desc`` - device description (ASCII text) + + - ``usb-ver`` - USB version (1, 2 or 3) + + + + + +Qubes RPC +--------- + + +Services called by dom0 to provide some VM configuration: + +- ``qubes.SetMonitorLayout`` - provide list of monitors, one per line. + Each line contains four numbers: + ``width height X Y width_mm height_mm`` (physical dimensions - + ``width_mm`` and ``height_mm`` - are optional) + +- ``qubes.WaitForSession`` - called to wait for full VM startup + +- ``qubes.GetAppmenus`` - receive appmenus from given VM (template); + TODO: describe format here + +- ``qubes.GetImageRGBA`` - receive image/application icon. Protocol: + + 1. Caller sends name of requested icon. This can be one of: + + + + - ``xdgicon:NAME`` - search for NAME in standard icons theme + + - ``-`` - get icon data from stdin (the caller), can be prefixed + with format name, for example ``png:-`` + + - file name + + + + 2. The service responds with image dimensions: width and height as + decimal numbers, separated with space and with EOL marker at the + and; then image data in RGBA format (32 bits per pixel) + + + + + +- ``qubes.SetDateTime`` - set VM time, called periodically by dom0 (can + be triggered manually from dom0 by calling ``qvm-sync-clock``). The + service receives one line at stdin - time in format of + ``date -u -Iseconds``, for example ``2015-07-31T16:10:43+0000``. + +- ``qubes.SetGuiMode`` - called in HVM to switch between fullscreen and + seamless GUI mode. The service receives a single word on stdin - + either ``FULLSCREEN`` or ``SEAMLESS`` + +- ``qubes.ResizeDisk`` - called to inform that underlying disk was + resized. Name of disk image is passed on standard input (``root``, + ``private``, ``volatile``, or other). This is used starting with + Qubes 4.0. + + + +Other Qrexec services installed by default: + +- ``qubes.Backup`` - store Qubes backup. The service receives location + chosen by the user (one line, terminated by ``\n``), the backup + archive (:doc:`description of backup format `) + +- ``qubes.DetachPciDevice`` - service called in reaction to + ``qvm-pci -d`` call on running VM. The service receives one word - + BDF of device to detach. When the service call ends, the device will + be detached + +- ``qubes.Filecopy`` - receive some files from other VM. Files sent in + :doc:`qfile format ` + +- ``qubes.OpenInVM`` - open a file in called VM. Service receives a + single file on stdin (in :doc:`qfile format `. After a + file viewer/editor is terminated, if the file was modified, can be + sent back (just raw content, without any headers); otherwise service + should just terminate without sending anything. This service is used + by both ``qvm-open-in-vm`` and ``qvm-open-in-dvm`` tools. When called + in DispVM, service termination will trigger DispVM cleanup. + +- ``qubes.Restore`` - retrieve Qubes backup. The service receives + backup location entered by the user (one line, terminated by ``\n``), + then should output backup archive in :doc:`qfile format ` (core-agent-linux component contains + ``tar2qfile`` utility to do the conversion) + +- ``qubes.SelectDirectory``, ``qubes.SelectFile`` - services which + should show file/directory selection dialog and return (to stdout) a + single line containing selected path, or nothing in the case of + cancellation + +- ``qubes.SuspendPre`` - service called in every VM with PCI device + attached just before system suspend + +- ``qubes.SuspendPost`` - service called in every VM with PCI device + attached just after system resume + +- ``qubes.SyncNtpClock`` - service called to trigger network time + synchronization. Service should synchronize local VM time and + terminate when done. + +- ``qubes.WindowIconUpdater`` - service called by VM to send icons of + individual windows. The protocol there is simple one direction + stream: VM sends window ID followed by icon in ``qubes.GetImageRGBA`` + format, then next window ID etc. VM can send icon for the same window + multiple times to replace previous one (for example for animated + icons) + +- ``qubes.VMShell`` - call any command in the VM; the command(s) is + passed one per line + + - ``qubes.VMShell+WaitForSession`` waits for full VM startup first + + + +- ``qubes.VMExec`` - call any command in the VM, without using shell, + the command needs to be passed as argument and encoded as follows: + + - the executable name and arguments are separated by ``+`` + + - everything except alphanumeric characters, ``.`` and ``_`` needs + to be escaped + + - bytes are escaped as ``-HH`` (where ``HH`` is hex code, capital + letters only) + + - ``-`` itself can be escaped as ``--`` + + - example: to run ``ls -a /home/user``, use + ``qubes.VMExec+ls+--a+-2Fhome-2Fuser`` + + + +- ``qubes.VMExecGUI`` - a variant of ``qubes.VMExec`` that waits for + full VM startup first + + + +Services called in GuiVM: + +- ``policy.Ask``, ``policy.Notify`` - confirmation prompt and + notifications for Qubes RPC calls, see :ref:`qrexec-policy implementation ` + for a detailed description. + + + +Currently Qubes still calls few tools in VM directly, not using service +abstraction. This will change in the future. Those tools are: + +- ``/usr/lib/qubes/qubes-download-dom0-updates.sh`` - script to + download updates (or new packages to be installed) for dom0 + (``qubes-dom0-update`` tool) + +- ``date -u -Iseconds`` - called directly to retrieve time after + calling ``qubes.SyncNtpClock`` service (``qvm-sync-clock`` tool) + +- ``nm-online -x`` - called before ``qubes.SyncNtpClock`` service call + by ``qvm-sync-clock`` tool + +- ``resize2fs`` - called to resize filesystem on /rw partition by + ``qvm-grow-private`` tool + +- ``gpk-update-viewer`` - called by Qubes Manager to display available + updates in a TemplateVM + +- ``systemctl start qubes-update-check.timer`` (and similarly stop) - + called when enabling/disabling updates checking in given VM + (``qubes-update-check`` :doc:`qvm-service `) + + + +Additionally, automatic tests extensively run various commands directly +in VMs. We do not plan to change that. + +GUI protocol +------------ + + +GUI initialization includes passing the whole screen dimensions from +dom0 to VM. This will most likely be overwritten by +qubes.SetMonitorLayout Qubes RPC call. diff --git a/developer/debugging/windows-debugging.md b/developer/debugging/windows-debugging.md deleted file mode 100644 index 089dfecd61..0000000000 --- a/developer/debugging/windows-debugging.md +++ /dev/null @@ -1,89 +0,0 @@ ---- -lang: en -layout: doc -permalink: /doc/windows-debugging/ -redirect_from: -- /en/doc/windows-debugging/ -- /doc/WindowsDebugging/ -- /wiki/WindowsDebugging/ -ref: 50 -title: Windows debugging ---- - -Debugging Windows code can be tricky in a virtualized environment. The guide below assumes Qubes 4.2 and Windows 7 or later VMs. - -User-mode debugging is usually straightforward if it can be done on one machine. Just duplicate your normal debugging environment in the VM. - -Things get complicated if you need to perform kernel debugging or troubleshoot problems that only manifest on system boot, user logoff or similar. For that you need two Windows VMs: the *host* and the *target*. The *host* will contain the debugger, your source code and private symbols. The *target* will run the code being debugged. We will use kernel debugging over network which is supported from Windows 7 onwards. The main caveat is that Windows kernel supports only specific network adapters for this, and the default one in Qubes won't work. - -## Important note - -- Do not install Xen network PV drivers in the target VM. Network kernel debugging needs a specific type of NIC or it won't work, the network PV drivers interfere with that. - -- If you have kernel debugging active when the Xen PV drivers are being installed, make sure to disable it before rebooting (`bcdedit /set debug off`). You can re-enable debugging after the reboot. The OS won't boot otherwise. I'm not sure what's the exact cause. I know that busparams for the debugging NIC change when PV drivers are installed (see later), but even changing that accordingly in the debug settings doesn't help -- so it's best to disable debug for this one reboot. - -## Modifying the NIC of the target VM - -You will need to create a custom libvirt config for the target VM. See [the documentation](https://dev.qubes-os.org/projects/core-admin/en/latest/libvirt.html) for overview of how libvirt templates work in Qubes. The following assumes the target VM is named `target-vm`. - -- Edit `/usr/share/qubes/templates/libvirt/xen.xml` to prepare our custom config to override just the NIC part of the global template: - - add `{% raw %}{% block network %}{% endraw %}` before `{% raw %}{% if vm.netvm %}{% endraw %}` - - add `{% raw %}{% endblock %}{% endraw %}` after the matching `{% raw %}{% endif %}{% endraw %}` -- Copy `/usr/share/qubes/templates/libvirt/devices/net.xml` to `/etc/qubes/templates/libvirt/xen/by-name/target-vm.xml`. -- Add `` to the `` section. -- Enclose everything within `{% raw %}{% block network %}{% endraw %}` + `{% raw %}{% endblock %}{% endraw %}`. -- Add `{% raw %}{% extends 'libvirt/xen.xml' %}{% endraw %}` at the start. -- The final `target-vm.xml` should look something like this: - -~~~ -{% raw %} -{% extends 'libvirt/xen.xml' %} -{% block network %} - - - - -