-
-
Notifications
You must be signed in to change notification settings - Fork 604
Modularization
By design, OSv has always been a "fat" unikernel and by default has provided a large subset of glibc functionality and has included full standard C++ library (libstdc++), the ZFS implementation, drivers for many devices, and has supported many hypervisors. On one hand, it makes running arbitrary applications on any hypervisor very easy using a single universal kernel. On another hand, such universality comes with the price of the bloated kernel with many symbols and drivers and possibly ZFS unused, thus causing inefficient memory usage, longer boot time, and potential security vulnerabilities. In addition, the C++ applications linked against a version of libstdc++ different than the version the kernel was linked against, may simply not work.
To address these issues, release 0.57 introduced a new experimental build mode to hide the non-glibc symbols and libstdc++ and extracted ZFS code out of the kernel in form of a dynamically linked library. It also provided another build option to tailor the kernel to a set of specific device drivers - 'driver profiles', and finally allowed building a version of the kernel with a subset of glibc symbols needed to support a specific application. The following paragraphs explain these changes and features in more detail.
As issue 97 explains, OSv by default exports all symbols including full standard C++ library. This has many negative implications - overly large kernel ELF, thus slower boot time, and potential for a collision of the kernel symbols with the application ones. On top of this, the C++ applications may not work if linked with the version of libstdc++ different than the OSv kernel.
To address these, release 0.57 added a new build option conf_hide_symbols that would hide most non-glibc and the libstdc++ symbols if enabled. In essence, most source files except the ones under the musl/ and libc/ directories would be compiled with the special compiler flags - -fvisibility=hidden and -fvisibility-inlines-hidden (C++ only) if conf_hide_symbols enabled. On another hand, the symbols to be exposed as public (like the glibc ones), would be annotated with OSV_***_API macros that translate to __attribute__ ((visibility ("default"))). Finally, in order to remove all unneeded code - "garbage", all files would be compiled with the flags -ffunction-sections -fdata-sections and linked with the flag --gc-sections. For more details about symbol visibility please read this article.
./scripts/build image=native-example fs=rofs conf_hide_symbols=1 -j$(nproc)
The kernel ELF file, built with most symbols hidden as above is roughly 3.6M in size, compared to 6M - a reduction of ~ 40%. This great reduction stems from the fact that the libstdc++ is no longer linked with --whole-archive, the symbol table is much smaller and unsued code is garbage collected. Please note the resulting kernel is still universal as it exports all glibc symbols and includes all device drivers.
The conf_hide_symbols is disabled by default because there are still some OSv modules like httpserver-api and more importantly some unit tests that depend on some internal C++ ABI of OSv. For more details look at this issue.
The list of the public symbols exported by the kernel is actually enforced during the build process based on the symbol list files for each advertised library like libc.so.6 located and maintained under the directory /exported_symbols (for more details look at this readme). These files are concatenated using scripts/generate_version_script.sh to the $(out)/version_script and fed to the linker as an argument to --version-script flag.
For more insight into the technical details, please look at this commit and that one.
ZFS is one of the three filesystems that OSv implements. The original code comes from FreeBSD circa 2014 and has been since integrated and adapted to work within OSv. ZFS is one of the best file systems for persistent mutable data and what one needs to run mysql or elasticsearch on OSv for example. Although a lot of effort has gone into integrating ZFS into OSv, its sheer code base volume made OSv kernel bigger than it needs to be and there are a plethora of stateless use cases like redis, memcached, microservices, etc where ZFS is not needed and instead we can use ROFS now.
To allow to optionally include ZFS, we extracted the ZFS code out of the kernel in form of a dynamically shared library (aka module) as part of the 0.57 release. The ZFS-less kernel is now roughly 800K smaller, thus making OSv images smaller, using less memory, and booting slightly faster. The ZFS library - libsolaris.so - can be loaded from the ROFS filesystem and the ZFS filesystem mounted on the same or different disk as explained in this wiki and illustrated by this example:
# include zfs module that puts /usr/lib/fs/libsolaris.so on RoFS filesystem
./scripts/build image=native-example,zfs fs=rofs_with_zfs
Please look at this commit to better understand how ZFS code got extracted from the kernel and this one to get insight into the logic behind loading the libsolaris.so and mounting the ZFS filesystem.
The release 0.57 introduced a new build mechanism that allows the creation of a custom kernel with a specific list of device drivers intended to target a given hypervisor. Such kernel benefits from the smaller size and better security as all unneeded code is removed.
In essence, we introduce a new build script and makefile parameter: drivers_profile. This new parameter is intended to specify a drivers profile which is simply a list of device drivers to be linked into the kernel with some extra functionality like PCI or ACPI these drivers
depend on. Each profile is specified in a tiny make include file (*.mk) under the conf/profiles/$(arch) directory and included by the main makefile as requested by the drivers_profile parameter. The main makefile has a number of ifeq expressions that add conditionally given driver object file to the linked objects list depending on the value (0 or 1) of the given conf_drivers_* variable specified in the relevant profile file.
The benefits of using drivers are most profound when building a kernel with most symbols hidden. Below you can see examples of some build commands along with the kernel size produced:
./scripts/build fs=rofs conf_hide_symbols=1 image=native-example #all drivers
3632K build/release/kernel-stripped.elf
./scripts/build fs=rofs conf_hide_symbols=1 image=native-example drivers_profile=virtio-pci
3380K build/release/kernel-stripped.elf
./scripts/build fs=rofs conf_hide_symbols=1 image=native-example drivers_profile=vmware
3308K build/release/kernel-stripped.elf
./scripts/build fs=rofs conf_hide_symbols=1 image=native-example drivers_profile=virtio-mmio
3120K build/release/kernel-stripped.elf
./scripts/build fs=rofs conf_hide_symbols=1 image=native-example drivers_profile=base #most drivers out
3036K build/release/kernel-stripped.elf
It is also possible to enable or disable individual drivers on
top of what given profiles defines like so:
./scripts/build fs=rofs conf_hide_symbols=1 image=native-example drivers_profile=base \
conf_drivers_acpi=1 conf_drivers_virtio_fs=1 conf_drivers_virtio_net=1 conf_drivers_pvpanic=1
To get a better understanding of how the driver profiles are implemented please look at this commit.
See https://github.com/cloudius-systems/osv/commit/d19ccb1cde100ab4a5c8e6db9a0d69560cabbd04
This patch introduces another new build mechanism that allows creating custom kernel exporting only symbols required by specific application. Such kernel benefits from smaller size and better security as all unneeded code is removed. This patch addresses remaining part of the modularization/librarization functionality as explained by the issue https://github.com/cloudius-systems/osv/issues/1110 and this part of the roadmap - https://github.com/cloudius-systems/osv/wiki/Roadmap#modularizationlibrarization. This idea was also mentioned in the P99 OSv presentation - see slide 12.