x86/defconfig: Add kata containers defconfig #5

devimc · 2017-11-23T15:00:04Z

kata_containers_common_defconfig is based on Clear Containers config
https://github.com/clearcontainers/packaging/blob/master/kernel/kernel-config-4.9.x
and was adapted to the 4.14.13 kernel by just accepting the defaults configurations

For example, to compile a x86 kernel with KVM enabled:

cat arch/x86/configs/kata_containers_common_defconfig arch/x86/configs/kata_containers_kvm_defconfig > arch/x86/configs/kata_containers_defconfig
make ARCH=x86 kata_containers_defconfig
make -j8

and to compile a x86 kernel with XEN enabled:

cat arch/x86/configs/kata_containers_common_defconfig arch/x86/configs/kata_containers_xen_defconfig > arch/x86/configs/kata_containers_defconfig
make ARCH=x86 kata_containers_defconfig
make -j8

fixes #3

Signed-off-by: Julio Montes [email protected]

grahamwhaley · 2017-11-23T15:12:40Z

Hi @devimc - as 2000+ lines of kernel config is quite tricky and takes some time to review ;-), can you give us a little background:

is this derived from the Clear Containers clearcontainers@9a69950 for instance?
did we adapt to the 4.13.3 kernel by just accepting the defaults for all the new CONFIG entries?

Also, I think you may have submitted this PR against master - should it be against/towards linux-kata-containers-4.13.3 instead?

@gnawux - if you have a kernel config expert, please point /cc them in here - we'd welcome input to share kernel config knowledge (but I also understand it can be quite a task to do a config compare or review, so maybe we will have to schedule that as a task for later).

Generally though, presuming we have tested this kernel in some form, and we get this PR aimed at the correct branch:
lgtm

devimc · 2017-11-23T15:17:23Z

Hi @grahamwhaley
oops! changing to linux-kata-containers-4.13.3

bergwolf

The list is quite long to review. I might have missed something but I already see following differences between cc and hyper kernels:

initramfs support
loadable module support
XEN PV guset
veth support

We need to get consensus on them before this can be merged.

bergwolf · 2017-11-27T10:26:17Z

arch/x86/configs/kata_containers_defconfig

+CONFIG_CGROUP_PERF=y
+# CONFIG_CGROUP_DEBUG is not set
+CONFIG_SOCK_CGROUP_DATA=y
+CONFIG_CHECKPOINT_RESTORE=y


why do we need checkpoint/restore?

@devimc, did we need this as part of the investigation of implementing docker checkpoint using CRIU?
https://docs.docker.com/engine/reference/commandline/checkpoint/
https://github.com/clearcontainers/runtime/blob/master/docs/limitations.md#checkpointandrestore
https://criu.org/Main_Page

I actually don't think we need it.

@grahamwhaley we have to options to support live migration:

using libcontainer to migrate only the process

using qemu micro checkpoint to migrate all the VM

probably both options won't work because of 9p gaps

anyway, I'll disable it

bergwolf · 2017-11-27T10:26:55Z

arch/x86/configs/kata_containers_defconfig

+CONFIG_NET_NS=y
+CONFIG_SCHED_AUTOGROUP=y
+# CONFIG_SYSFS_DEPRECATED is not set
+CONFIG_RELAY=y


and relay? Do you want debugfs?

I suspect we don't need this - anybody know for sure?

We don explicitly need it, but it may come as a dependency from something else.

I'll try to disable it

bergwolf · 2017-11-27T10:34:56Z

arch/x86/configs/kata_containers_defconfig

+CONFIG_SCHED_AUTOGROUP=y
+# CONFIG_SYSFS_DEPRECATED is not set
+CONFIG_RELAY=y
+# CONFIG_BLK_DEV_INITRD is not set


Can you please enable CONFIG_BLK_DEV_INITRD support? We want to use initramfs.

Yes, @devimc please enable INITRD.

bergwolf · 2017-11-27T10:47:36Z

arch/x86/configs/kata_containers_defconfig

+CONFIG_SLABINFO=y
+CONFIG_RT_MUTEXES=y
+CONFIG_BASE_SMALL=0
+# CONFIG_MODULES is not set


Can you please enable module support? We'd like to use loadable modules to keep the kernel small.

Ah, here we can have an interesting discussion :-)
We have deliberately not enabled modules before for a couple of reasons:

size, speed, probing etc.

where will you store the modules (in the container images or on a volume??), and how will you ensure the modules that live on the host match the kernel that you are running inside the VM?

We have debated this a few times. Quite often a new network or filesystem or feature will be needed by a specific client, and having modules available would be one way to fix that.

If we turn on modules, then we should:

assess any size/speed impact

I believe turn on kernel API checking so we can try to avoid loading a module/kernel that have a mismatch in API

discuss signing of modules as well?

@grahamwhaley All fair points, that should be handled by osbuilder or the kata containers user tools for building its image. Provided that kernel modules support does not impact boot time for kernels that do not load any module at all, I think we should enable it.

Currently we store modules in sandbox rootfs and bind mount it to container rootfs images. Modules are built together with the guest kernel and compressed in the initramfs.

Optional features are built as modules. A sample list includes crypto, nfs, vsock, ipv6, l2tp etc. They are not essential to run a pod and are not loaded if unneeded.

And providing loadable modules opens up the opportunity to provide guest modules if the default kernel config misses something important to a user. In such case, we only need to provide additional kernel modules rather than rebuilding entire kernel.

bergwolf · 2017-11-27T10:50:50Z

arch/x86/configs/kata_containers_defconfig

+CONFIG_PARAVIRT=y
+# CONFIG_PARAVIRT_DEBUG is not set
+# CONFIG_PARAVIRT_SPINLOCKS is not set
+# CONFIG_XEN is not set


We'd like to have XEN PV guest support.

bergwolf · 2017-11-27T11:02:20Z

arch/x86/configs/kata_containers_defconfig

+# CONFIG_NET_POLL_CONTROLLER is not set
+CONFIG_TUN=y
+# CONFIG_TUN_VNET_CROSS_LE is not set
+# CONFIG_VETH is not set


we need CONFIG_VETH.

@bergwolf You need veth in the guest?

@sameo not for the common case. But we got quite a few docker in hyperd and hyperd in hyperd requests, which is why we enabled KVM/cgroups support in guest kernel as well.

@bergwolf Fair enough, thanks.

laijs · 2017-11-27T12:15:10Z

Since we have https://github.com/kata-containers/osbuilder, IMO, it would be better to maintain the kernel config in it.

grahamwhaley · 2017-11-27T13:22:47Z

@laijs that's a good question. My only concern would be separating the config away from any patches we are carrying. Having said that, if we end up supporting multiple configs for different performance characteristic/feature sets for instance, then maybe holding those in osbuilder makes sense.
/cc @jcvenegas

sameo · 2017-11-27T13:57:43Z

@laijs I believe we should keep a reference config file in the kernel and maintain variants in osbuilder, as you suggest.
It should not be mandatory to use osbuilder to build a kata containers kernel, imho.

laijs · 2017-11-28T01:57:14Z

@grahamwhaley @sameo Thanks, I got it.

teawater · 2017-11-28T07:22:04Z

Hi @devimc ,

We need CONFIG_VMAP_STACK.

Thanks,
Hui

teawater · 2017-11-28T07:31:44Z

Hi @devimc ,

I think the kata config should not be included in the kernel repo because I think the kernel repo should just keep the patch that we want to upstream it.

Thanks,
Hui

sameo · 2017-11-28T12:14:53Z

@teawater

We need CONFIG_VMAP_STACK.

Have you benchmarked this? Having stacked vmalloc'ed can potentially introduce TLB misses from stack accesses and could hurt performance.

cc @mcastelino

devimc · 2017-11-28T14:31:07Z

after apply the changes suggested by @bergwolf:

CONFIG_CHECKPOINT_RESTORE is not set
CONFIG_RELAY is not set
CONFIG_KPROBES is not set
CONFIG_BLK_DEV_INITRD=y
CONFIG_RD_GZIP=y
CONFIG_RD_BZIP2=y
CONFIG_RD_LZMA=y
CONFIG_RD_XZ=y
CONFIG_RD_LZO=y
CONFIG_RD_LZ4=y
CONFIG_STRICT_MODULE_RWX=y
CONFIG_MODULES=y
CONFIG_XEN=y
CONFIG_XEN_PV=y
CONFIG_XEN_PV_SMP=y
CONFIG_XEN_DOM0=y
CONFIG_XEN_PVHVM=y
CONFIG_XEN_PVHVM_SMP=y
CONFIG_XEN_512GB=y
CONFIG_XEN_SAVE_RESTORE=y
CONFIG_VETH=y
CONFIG_HIBERNATE_CALLBACKS=y
CONFIG_PM_SLEEP=y
CONFIG_PM_SLEEP_SMP=y
CONFIG_PM=y
CONFIG_PM_CLK=y
CONFIG_ACPI_TABLE_UPGRADE=y
CONFIG_PCIE_PME=y
CONFIG_XEN_PCIDEV_FRONTEND=y
CONFIG_SYS_HYPERVISOR=y
CONFIG_XEN_BLKDEV_FRONTEND=y
CONFIG_XEN_NETDEV_FRONTEND=y
CONFIG_INPUT_XEN_KBDDEV_FRONTEND=y
CONFIG_HVC_IRQ=y
CONFIG_HVC_XEN=y
CONFIG_HVC_XEN_FRONTEND=y
CONFIG_VT_CONSOLE_SLEEP=y
CONFIG_ZLIB_INFLATE=y
CONFIG_LZO_DECOMPRESS=y
CONFIG_LZ4_DECOMPRESS=y
CONFIG_DECOMPRESS_GZIP=y
CONFIG_DECOMPRESS_BZIP2=y
CONFIG_DECOMPRESS_LZMA=y
CONFIG_DECOMPRESS_XZ=y
CONFIG_DECOMPRESS_LZO=y
CONFIG_DECOMPRESS_LZ4=y
#
# Xen driver support
#
CONFIG_XEN_BALLOON=y
CONFIG_XEN_SCRUB_PAGES=y
CONFIG_XEN_DEV_EVTCHN=y
CONFIG_XEN_BACKEND=y
CONFIG_XENFS=y
CONFIG_XEN_COMPAT_XENFS=y
CONFIG_XEN_SYS_HYPERVISOR=y
CONFIG_XEN_XENBUS_FRONTEND=y
CONFIG_XEN_GNTDEV=m
CONFIG_XEN_GRANT_DEV_ALLOC=m
CONFIG_SWIOTLB_XEN=y
CONFIG_XEN_PCIDEV_BACKEND=m
CONFIG_XEN_PRIVCMD=y
CONFIG_XEN_ACPI_PROCESSOR=m
CONFIG_XEN_HAVE_PVMMU=y
CONFIG_XEN_AUTO_XLATE=y
CONFIG_XEN_ACPI=y
CONFIG_XEN_SYMS=y
CONFIG_XEN_HAVE_VPMU=y

teawater · 2017-11-29T14:14:22Z

@sameo ,
After open vmap stack, the execution time of test with sysbench memory was decreased about 7% in my part.
And we have long history to use it anti long time execution fragmentation in low memory environment.

Thanks,
Hui

mcastelino · 2017-11-29T16:43:28Z

@teawater with vmap stack the performance degrades then. Did you check the kvm stats to see if you are seeing much higher ept violations?

teawater · 2017-11-30T00:56:08Z

Sorry to make a mistake in language. I want to say performance increased but input the execution time increased.

This the clear test result of sysbench memory inside a qemu:
Without vmap stack:
32.6117
31.169
29.1881
30.4291
31.5273

With vmap stack:
29.4660
29.2134
28.0476
28.9640
28.2300

teawater · 2017-12-05T15:05:13Z

Following part is what we need:
CONFIG_VMAP_STACK
CONFIG_SERIAL_EARLYCON=y
CONFIG_SERIAL_8250=y
CONFIG_SERIAL_8250_CONSOLE=y
CONFIG_SERIAL_CORE=y
CONFIG_SERIAL_CORE_CONSOLE=y
CONFIG_SCSI=y
CONFIG_SCSI_DMA=y
CONFIG_SG_POOL=y
CONFIG_SCSI_LOWLEVEL=y
CONFIG_SCSI_VIRTIO=y
CONFIG_BLK_DEV_SD=y
CONFIG_CHR_DEV_SG=y
CONFIG_NFS_FS=m
CONFIG_NFS_V2=m
CONFIG_NFS_V3=m
CONFIG_NFS_V4=m

Thanks,
Hui

devimc · 2017-12-06T15:40:43Z

@sameo @mcastelino @bergwolf are you agree with @teawater 's changes ?

devimc · 2017-12-06T21:25:33Z

Update

CONFIG_NR_CPUS=255

sameo · 2017-12-10T06:26:20Z

@teawater Why do you need all the serial options? Same question for NFS.

mcastelino · 2017-12-11T16:58:04Z

The SCSI changes make sense for sure. We plan to move to SCSI for all block devices. The VMAP change also makes sense as the data from @teawater seems to indicate that this does not degrade performance. It will give us some density benifits. However we still need to run I/O benchmarks.

@grahamwhaley @GabyCT we should run PnP with VMAP enabled for storage and network I/O. We really care about I/O performance more than anything else.

teawater · 2017-12-12T07:40:59Z

@sameo We use these kernel options with hyperstart.

Thanks,
Hui

sameo · 2017-12-12T11:12:56Z

@teawater I don't think this will be needed with the Kata agent. Could you please double check?

bergwolf · 2017-12-12T11:38:04Z

@sameo Yes, we are sure we need them. hyperstart supports NFS volumes. We'd expect the kata agent to support NFS volumes as well. And we need serial ports in case vsock is unavailable, right?

sameo · 2017-12-12T12:50:53Z

@bergwolf Yes, I figured you'd need NFS volumes, and that is fine with me, especially since it's going to be a module.
But for serial, the only thing we need is virtio console and tty.

devimc · 2018-02-19T18:26:06Z

@sameo @bergwolf changes applied

sameo · 2018-02-20T09:59:10Z

@egernst Yes, that's the idea.

sameo · 2018-02-20T10:31:48Z

@devimc Looks good. Maybe we should have a script to safely generate the final Kata config?
@bergwolf @egernst WDYT?

jodh-intel · 2018-02-20T10:47:28Z

@sameo - I think that's a very good idea. I've raised kata-containers/packaging#8 (and kata-containers/packaging#7 for qemu config).

/cc @grahamwhaley.

devimc · 2018-02-26T15:26:29Z

@sameo a script in packaging or here?

jodh-intel · 2018-03-01T16:09:25Z

Personally, I think such scripts should live in the packaging repo as that's where they will be used. It also allows repos like this (and qemu) to remain as "vanilla" as possible.

sameo · 2018-03-13T13:31:36Z

@devimc @jodh-intel In packaging, yes. Also, see:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/scripts/kconfig/merge_config.sh

devimc · 2018-04-11T14:26:09Z

@kata-containers/linux can we merge this PR ?

grahamwhaley · 2018-04-11T14:30:11Z

Looks like this is mergeable @devimc , althought the discussion about having a config generating script in the packaging did not seem to conclude.
This PR is also very old - is this config file upto date - I suspect it could do with a re-generation before a merge maybe??

jcvenegas · 2018-04-18T15:05:32Z

Any update on this?

jcvenegas · 2018-04-18T15:10:45Z

btw I would be good to have the steps described in this PR documented somewhere.

cat arch/x86/configs/kata_containers_common_defconfig arch/x86/configs/kata_containers_kvm_defconfig > arch/x86/configs/kata_containers_defconfig
make ARCH=x86 kata_containers_defconfig
make -j8

jodh-intel · 2018-04-18T15:22:48Z

tbh, I think we could start by simply having the kernel config in the packaging area, as we did for Clear Containers:

https://github.com/clearcontainers/packaging/blob/master/kernel/kernel-config-4.14.x

WeiZhang555 · 2018-04-19T06:32:34Z

I am thinking if it's better to put the kernel config in kernel repo, and qemu config in qemu repo. this can allow us compile our own kernel/qemu without packaging repo.
Seperating kernel repo with kernel config can bring confusion to developers.

jodh-intel · 2018-04-19T07:01:44Z

I can understand that viewpoint too. But I think as long as we clearly document the process, either would work. It's worth pointing out that we already have precedent for the hypervisor though:

-https://github.com/kata-containers/packaging/blob/master/scripts/configure-hypervisor.sh

gnawux · 2018-04-19T07:08:30Z

@WeiZhang555

I am thinking if it's better to put the kernel config in kernel repo, and qemu config in qemu repo. this can allow us compile our own kernel/qemu without packaging repo.

That's reasonable, however, as linux and qemu are forked from upstream, we'd better keep the stuff not tend to be upstream outside the repo. A packaging script works too as mentioned by @jodh-intel .

WeiZhang555 · 2018-04-19T08:25:41Z

@gnawux @jodh-intel That's fine. I believe we can eliminate the confusion by providing some good documentations 😄

kata_containers_common_defconfig is based on Clear Containers config https://github.com/clearcontainers/packaging/blob/master/kernel/kernel-config-4.9.x and was adapted to the 4.14.13 kernel by just accepting the defaults configurations For example, to compile a x86 kernel with KVM enabled: ``` cat arch/x86/configs/kata_containers_common_defconfig arch/x86/configs/kata_containers_kvm_defconfig > arch/x86/configs/kata_containers_defconfig make ARCH=x86 kata_containers_defconfig make -j8 ``` and to compile a x86 kernel with XEN enabled: ``` cat arch/x86/configs/kata_containers_common_defconfig arch/x86/configs/kata_containers_xen_defconfig > arch/x86/configs/kata_containers_defconfig make ARCH=x86 kata_containers_defconfig make -j8 ``` fixes kata-containers#3 Signed-off-by: Julio Montes <[email protected]>

jcvenegas · 2018-04-19T21:58:44Z

@grahamwhaley @jodh-intel @gnawux @laijs what about the proposed PR by @devimc in kata-containers/packaging#18

jodh-intel · 2018-04-20T07:25:07Z

Thanks for the pointer @jcvenegas.

@devimc - do we need to add the do-not-merge label to this PR then?

devimc changed the base branch from master to linux-kata-containers-4.13.3 November 23, 2017 15:15

devimc force-pushed the config/kata_defconfig branch 3 times, most recently from 6d45e5d to 4feec8b Compare November 25, 2017 16:51

bergwolf reviewed Nov 27, 2017

View reviewed changes

devimc force-pushed the config/kata_defconfig branch from 4feec8b to 41e0e79 Compare November 28, 2017 14:30

devimc force-pushed the config/kata_defconfig branch from 41e0e79 to 6647ef5 Compare December 6, 2017 21:25

jcvenegas mentioned this pull request Dec 6, 2017

Add kernel builder script kata-containers/osbuilder#6

Closed

devimc changed the base branch from linux-kata-containers-4.13.3 to linux-kata-containers-4.14.13 February 19, 2018 18:24

jodh-intel mentioned this pull request Feb 20, 2018

Create Linux kernel configuration script kata-containers/packaging#8

Closed

jodh-intel mentioned this pull request Mar 28, 2018

CI: Move from qemu 2.7 to qemu 2.11 kata-containers/tests#182

Merged

jcvenegas mentioned this pull request Apr 4, 2018

Add Release Process doc kata-containers/documentation#32

Closed

devimc added the do-not-merge label Apr 9, 2018

devimc mentioned this pull request Apr 9, 2018

network: specify netlink protocol kata-containers/agent#210

Merged

devimc force-pushed the config/kata_defconfig branch from f95bdc7 to f4e94c0 Compare April 11, 2018 14:24

devimc removed the do-not-merge label Apr 11, 2018

jcvenegas mentioned this pull request Apr 18, 2018

docs: Add Release process docs kata-containers/documentation#56

Merged

devimc force-pushed the config/kata_defconfig branch from f4e94c0 to 1355353 Compare April 19, 2018 20:26

devimc closed this Apr 20, 2018

x86/defconfig: Add kata containers defconfig #5

x86/defconfig: Add kata containers defconfig #5

Conversation

devimc commented Nov 23, 2017 • edited Loading

grahamwhaley commented Nov 23, 2017

devimc commented Nov 23, 2017

bergwolf left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

laijs commented Nov 27, 2017

grahamwhaley commented Nov 27, 2017

sameo commented Nov 27, 2017

laijs commented Nov 28, 2017

teawater commented Nov 28, 2017

teawater commented Nov 28, 2017

sameo commented Nov 28, 2017

devimc commented Nov 28, 2017

teawater commented Nov 29, 2017 • edited Loading

mcastelino commented Nov 29, 2017

teawater commented Nov 30, 2017

teawater commented Dec 5, 2017

devimc commented Dec 6, 2017

devimc commented Dec 6, 2017

sameo commented Dec 10, 2017

mcastelino commented Dec 11, 2017

teawater commented Dec 12, 2017 • edited Loading

sameo commented Dec 12, 2017

bergwolf commented Dec 12, 2017

sameo commented Dec 12, 2017

devimc commented Feb 19, 2018

sameo commented Feb 20, 2018

sameo commented Feb 20, 2018

jodh-intel commented Feb 20, 2018

devimc commented Feb 26, 2018

jodh-intel commented Mar 1, 2018

sameo commented Mar 13, 2018

devimc commented Apr 11, 2018

grahamwhaley commented Apr 11, 2018

jcvenegas commented Apr 18, 2018

jcvenegas commented Apr 18, 2018

jodh-intel commented Apr 18, 2018

WeiZhang555 commented Apr 19, 2018

jodh-intel commented Apr 19, 2018

gnawux commented Apr 19, 2018

WeiZhang555 commented Apr 19, 2018

jcvenegas commented Apr 19, 2018

jodh-intel commented Apr 20, 2018

devimc commented Nov 23, 2017 •

edited

Loading

bergwolf left a comment •

edited

Loading

teawater commented Nov 29, 2017 •

edited

Loading

teawater commented Dec 12, 2017 •

edited

Loading