-
Notifications
You must be signed in to change notification settings - Fork 45
/
README
248 lines (167 loc) · 8.33 KB
/
README
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
KERNEL SOURCE:
==============
We currently use the Ubuntu kernel sources, available from our mirror:
https://git.proxmox.com/?p=mirror_ubuntu-kernels.git;a=summary
Ubuntu will maintain those kernels till:
https://wiki.ubuntu.com/Kernel/Dev/ExtendedStable
or
https://pve.proxmox.com/pve-docs/chapter-pve-faq.html#faq-support-table
whatever happens to be earlier.
Additional/Updated Modules:
---------------------------
- include native OpenZFS filesystem kernel modules for Linux
* https://github.com/zfsonlinux/
For licensing questions, see: http://open-zfs.org/wiki/Talk:FAQ
BUILD
=====
As this is packaging for the Linux kernel with some extra integrations, like
ZFS, this repo cannot be handled like a plain Linux kernel git repository.
The actual Linux kernel source lives in a git submodule.
For a build you should init the submodules and then handle it like most our
Debian packaging builds. If unsure you can follow this:
Installing Build-Dependencies
-----------------------------
You can either just check the package metadata template `debian/control.in`
and install the packages listed in the `Build-Depends` section manually
(replace `debhelper-compat` with just `debhelper`) or use a more automated way
described below:
# install base build-dependencies and helpers
apt update
apt install devscripts
# create build-directory so that we got final packaging control files from the
# .in templates generated
make build-dir-fresh
# install build-dependencies (replace BUILD-DIR with actual one)
mk-build-deps -ir BUILD-DIR/debian/control
Package Build
-------------
# start the actual build
make deb
For simple KConfig modifications you can adapt the list in `debian/rules` file.
For quick code changes to the actual kernel code you can do them directly in
the submodule/ubuntu-kernels directory, then re-create the build-directory, e.g.:
make clean
# now build again, explicitly creating the build-dir isn't required anymore
# after one has the build-dependencies already installed.
make deb
Modify-Build-Test Cycles
------------------------
Ideally you avoid the need for doing a full package build and just directly
build linux from the ubuntu-kernels or the mainline (stable) repo with copying
over a build-config of a proxmox-kernel to that as .config and then using the
`make olddefconfig` target.
If you need full package builds you can try to make changes inside the
BUILD-DIR directly and then continue build from there, e.g., using
`dpkg-buildpackage -b -uc -us --no-pre-clean`. Depending on what stage you want
to continue build you might need to touch, or remove some *.prepared files.
Just check `debian/rules` for how kernel build progress is tracked by make.
SUBMODULE
=========
We track the current upstream repository as submodule. Besides obvious
advantages over tracking binary tar archives this also has some implications.
For building the submodule directory gets copied into build/ and a few patches
get applied with the `patch` tool. From a git point-of-view, the copied
directory remains clean even with extra patches applied since it does not
contain a .git directory, but a reference to the (still pristine) submodule:
$ cat build/ubuntu-kernel/.git
If you mistakenly cloned the upstream repo as "normal" clone (not via the
submodule mechanics) this means that you have a real .git directory with its
independent objects and tracking info when copying for building, thus git
operates on the copied directory - and "sees" that it was dirtied by `patch`,
and thus the kernel buildsystem sees this too and will add a '+' to the version
as a result. This changes the output directories for modules and other build
artefacts and let's then the build fail on packaging.
So always ensure that you really checked it out as submodule, not as full
"normal" clone. You can also explicitly set the LOCALVERSION variable to
undefined with: `export LOCALVERSION= but that should only be done for test
builds.
RELATED PACKAGES:
=================
proxmox-ve
----------
top level meta package, depends on current default kernel series meta package.
git clone git://git.proxmox.com/git/proxmox-ve.git
proxmox-default-kernel
----------------------
Depends on default kernel and header meta package, e.g., proxmox-kernel-6.2 /
proxmox-headers-6.2.
git clone git://git.proxmox.com/git/pve-kernel-meta.git
proxmox-kernel-X.Y
------------------
Depends on the latest kernel (or header, in case of proxmox-headers-X.Y)
package within a certain series.
e.g., proxmox-kernel-6.2 depends on proxmox-kernel-6.2.16-6-pve
NOTE: Since Proxmox VE 8, based on Debian 12 Bookworm, the kernel ABI is bumped
with every version bump due to module signing. Since then the meta package was
pulled into the kernel repo, before that it lived in pve-kernel-meta.git.
pve-firmware
------------
Contains the firmware for all released PVE kernels.
git clone git://git.proxmox.com/git/pve-firmware.git
NOTES:
======
ABI versions, package versions and package name:
------------------------------------------------
We follow debian's versioning w.r.t ABI changes:
https://kernel-team.pages.debian.net/kernel-handbook/ch-versions.html
https://wiki.debian.org/DebianKernelABIChanges
The debian/rules file has a target comparing the build kernel's ABI against the
version stored in the repository and indicates when an ABI bump is necessary.
An ABI bump within one upstream version consists of incrementing the KREL
variable in the Makefile, rebuilding the packages and running 'make abiupdate'
(the 'abiupdate' target in 'Makefile' contains the steps for consistently
updating the repository).
Watchdog blacklist
------------------
By default, all watchdog modules are black-listed because it is totally undefined
which device is actually used for /dev/watchdog.
We ship this list in /lib/modprobe.d/blacklist_proxmox-kernel-<VERSION>.conf
The user typically edit /etc/modules to enable a specific watchdog device.
Debug kernel and modules
------------------------
In order to build a -dbgsym package containing an unstripped copy of the kernel
image and modules, enable the 'pkg.proxmox-kernel.debug' build profile (e.g. by
exporting DEB_BUILD_PROFILES='pkg.proxmox-kernel.debug'). The resulting package can
be used together with 'crash'/'kdump-tools' to debug kernel crashes.
Note: the -dbgsym package is only valid for the proxmox-kernel packages produced by
the same build. A kernel/module from a different build will likely not match,
even if both builds are of the same kernel and package version.
Additional information
----------------------
We use the default configuration provided by Ubuntu, and apply
the following modifications:
NOTE: For the exact and current list see debian/rules (PVE_CONFIG_OPTS)
- enable INTEL_MEI_WDT=m (to allow disabling via patch)
- disable CONFIG_SND_PCM_OSS (enabled by default in Ubuntu, not needed)
- switch CONFIG_TRANSPARENT_HUGEPAGE to MADVISE from ALWAYS
- enable CONFIG_CEPH_FS=m (request from user)
- enable common CONFIG_BLK_DEV_XXX to avoid hardware detection
problems (udev, update-initramfs have serious problems without that)
CONFIG_BLK_DEV_SD=y
CONFIG_BLK_DEV_SR=y
CONFIG_BLK_DEV_DM=y
- compile NBD and RBD modules
CONFIG_BLK_DEV_NBD=m
CONFIG_BLK_DEV_RBD=m
- enable IBM JFS file system as module
requested by users (bug #64)
- enable apple HFS and HFSPLUS as module
requested by users
- enable CONFIG_BCACHE=m (requested by user)
- enable CONFIG_BRIDGE=y
to avoid warnings on boot, e.g. that net.bridge.bridge-nf-call-iptables is an unknown key
- enable CONFIG_DEFAULT_SECURITY_APPARMOR
We need this for lxc
- set CONFIG_CPU_FREQ_DEFAULT_GOV_PERFORMANCE=y
because if not set, it can give some dynamic memory or cpu frequencies
change, and vms can crash (mainly windows guest).
see http://forum.proxmox.com/threads/18238-Windows-7-x64-VMs-crashing-randomly-during-process-termination?p=93273#post93273
- use 'deadline' as default scheduler
This is the suggested setting for KVM. We also measure bad fsync performance with ext4 and cfq.
- disable CONFIG_INPUT_EVBUG
Module evbug is not blacklisted on debian, so we simply disable it to avoid
key-event logs (which is a big security problem)
- enable CONFIG_MODVERSIONS (needed for ABI tracking)
- switch default UNWINDER to FRAME_POINTER
the recently introduced ORC_UNWINDER is not 100% stable yet, especially in combination with ZFS
- enable CONFIG_PAGE_TABLE_ISOLATION (Meltdown mitigation)