Run monai-label in kubernetes

[ism93]

Hello,
is it possible to install monai-label on kubernetes?
I'm trying to make it work via a classic deployment but I'm encountering the following problems in my pods:

[k8sgpu-01|monai-dev] imarson_adm@pc12:~/monai$ k logs monai-label-6cd85976c5-6rjtl

=============
== PyTorch ==

NVIDIA Release 24.03 (build 85286408)
PyTorch Version 2.3.0a0+40ec155e58
Container image Copyright (c) 2024, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
Copyright (c) 2014-2024 Facebook Inc.
Copyright (c) 2011-2014 Idiap Research Institute (Ronan Collobert)
Copyright (c) 2012-2014 Deepmind Technologies (Koray Kavukcuoglu)
Copyright (c) 2011-2012 NEC Laboratories America (Koray Kavukcuoglu)
Copyright (c) 2011-2013 NYU (Clement Farabet)
Copyright (c) 2006-2010 NEC Laboratories America (Ronan Collobert, Leon Bottou, Iain Melvin, Jason Weston)
Copyright (c) 2006 Idiap Research Institute (Samy Bengio)
Copyright (c) 2001-2004 Idiap Research Institute (Ronan Collobert, Samy Bengio, Johnny Mariethoz)
Copyright (c) 2015 Google Inc.
Copyright (c) 2015 Yangqing Jia
Copyright (c) 2013-2016 The Caffe contributors
All rights reserved.

Various files include modifications (c) NVIDIA CORPORATION & AFFILIATES. All rights reserved.

This container image and its contents are governed by the NVIDIA Deep Learning Container License.
By pulling and using the container, you accept the terms and conditions of this license:
https://developer.nvidia.com/ngc/nvidia-deep-learning-container-license

NOTE: CUDA Forward Compatibility mode ENABLED.
Using CUDA 12.4 driver version 550.54.14 with kernel driver version 535.129.03.
See https://docs.nvidia.com/deploy/cuda-compatibility/ for details.

NOTE: The SHMEM allocation limit is set to the default of 64MB. This may be
insufficient for PyTorch. NVIDIA recommends the use of the following flags:
docker run --gpus all --ipc=host --ulimit memlock=-1 --ulimit stack=67108864 ...

=> And my pod is in crashLoopBackOff
Do You have some advice for using it?

Thank you

[diazandr3s]

Hi @ism93,

I'd recommend first making sure the MONAI Label Docker container works on this instance. Have you tried that?
What's the use case here? How are you planning to orchestrate the different pods including MONAI Label?

Please let us know,

[ism93]

Hi, thank you for your answer.
I can't run on docker because my GPU cluster is running on kubernetes. Yes I'm tryin to run monai-label image.

Thank you

[diazandr3s]

Hi @ism93,

As far as I know, Docker is used to create container images and run individual containers and Kubernetes uses these Docker containers as the basic unit of deployment and manages them across multiple hosts.

I'd suggest you first confirm that the MONAI Label Docker container works on your end before orchestrating more containers.