Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[feature] Can't increase/attach shared memory to pipeline task in kfp sdk v2 #11136

Open
hankug1234 opened this issue Aug 26, 2024 · 1 comment

Comments

@hankug1234
Copy link

Feature Area

apiVersion: "kubeflow.org/v1alpha1"
kind: PodDefault
metadata:
name: shm
namespace: kubeflow-user-example-com
spec:
selector:
matchLabels:
pipelines.kubeflow.org/v2_component: "true"
desc: "shm volume"
volumeMounts:

  • name: shmdir
    mountPath: /dev/shm
    volumes:
  • name: shmdir
    emptyDir:
    medium: Memory
    sizeLimit: "1G"

What feature would you like to see?

i faced problem that share memory out ouf bound error when use kfp v2 with pytorch dataLoder
and i found "[bug] Can't increase/attach shared memory to pipeline task in kfp sdk v2, e.g. PyTorch training fails. " bug report
but there wasn't instruction about proper solution but i found other way to solve this problem so i suggest this way

What is the use case or pain point?

workaround

i used PodDefault kubeflow crd and allocate new shm volume on each kubeflow pod
my yaml file instruction was written above

i hope this solution help to other people that faced same issue and have a nice day


Love this idea? Give it a 👍.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants
@hankug1234 and others