Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Python functions fail to build with mono repo structure / nested folders #4114

Open
cmanou opened this issue Sep 17, 2024 · 4 comments
Open

Python functions fail to build with mono repo structure / nested folders #4114

cmanou opened this issue Sep 17, 2024 · 4 comments
Assignees
@thdxr

Comments

@cmanou
Copy link
Contributor

cmanou commented Sep 17, 2024

It seems as though when you have a python function defined as follows

    new sst.aws.Function("MyPythonFunction", {
      handler: "projects/function-a/src/logic.handler",
      runtime: "python3.11",
      url: true,
    });

You'll hit the following error during the build

|  Error: Failed to build function "projects/function-a/src/logic.handler": Error: ENOENT: no such file or directory, copyfile '/Users/chris/test-project/projects/function-a/pyproject.toml' -> '/Users/chris/test-project/.sst/artifacts/MyPythonFunction-src/Users/chris/test-project/projects/function-a/pyproject.toml'
|      at file:///Users/chris/test-project/.sst/platform/src/components/aws/function.ts:1378:23 {
|    promise: Promise { <rejected> [Circular *1] }
|  }

Guessing this needs to trim the target path to be of the same parent as the project file
https://github.com/sst/ion/blob/5b183e67ed7b9134cd05577b2d417fc7d482534b/platform/src/runtime/python.ts#L73C1-L76C7
@cmanou cmanou mentioned this issue Sep 18, 2024
Closed
@walln walln mentioned this issue Sep 19, 2024
Merged
@thdxr thdxr self-assigned this Sep 23, 2024
@cmanou cmanou mentioned this issue Sep 25, 2024
Closed
@cosmic-ascendant
Copy link

I am having a related issue, except the errors appear to differ. I have the following structure:

├── package.json
├── package-lock.json
├── packages
│   └── functions
│       ├── function_a
│       │   ├── main.py
│       │   └── pyproject.toml
│       └── function_b
│           ├── main.py
│           └── pyproject.toml
├── README.md
├── sst.config.ts
└── tsconfig.json

Local (sst dev) actually works fine, but its only after deployment I get an error in lambda log (with and without container)

[ERROR] Runtime.ImportModuleError: Unable to import module 'packages/functions/function_a/main': No module named 'packages.functions.function_a.main'

The suggested PR fix had no material change to this.

On the other hand, if I have the following structure, all is well.

├── function_a
│   ├── main.py
│   └── pyproject.toml
├── function_b
│   ├── main.py
│   └── pyproject.toml
├── package.json
├── package-lock.json
├── README.md
├── sst.config.ts
└── tsconfig.json

I have not tested it, but it appears @walln 's example (https://github.com/walln/sst-modal/tree/main/packages) has a nested layout of which I assume is working?

@fwang
Copy link
Contributor

fwang commented Oct 16, 2024

@cmanou i'm getting the same error as @cosmic-ascendant w/ the PR:

[ERROR] Runtime.ImportModuleError: Unable to import module 'packages/function-a/src/python': No module named 'packages.function-a.src.python'

Both the container and non-container functions fail with the same error.

@cosmic-ascendant
Copy link

cosmic-ascendant commented Oct 16, 2024 •
edited
Loading

To add some color, given a monorepo setup like so:

├── package.json
├── packages
│   └── functions
│       ├── function_a
│       │   ├── lambda.py
│       │   └── pyproject.toml
│       └── function_b
│           ├── lambda.py
│           └── pyproject.toml
├── README.md
├── sst.config.ts
└── tsconfig.json

The lambda bundle looks like this:

FunctionA-src
├── aws_python_container-0.1.0.dist-info
│   ├── direct_url.json
│   ├── INSTALLER
│   ├── METADATA
│   ├── RECORD
│   ├── REQUESTED
│   ├── top_level.txt
│   └── WHEEL
├── __editable___aws_python_container_0_1_0_finder.py
├── __editable__.aws_python_container-0.1.0.pth
├── function_a
│   ├── lambda.py
│   └── sst.pyi
├── function_b
│   ├── lambda.py
│   └── sst.pyi
├── numpy
| ...
├── numpy-2.1.2.dist-info
| ...
├── numpy.libs
| ...
├── packages
│   └── functions
│       └── function_a
│           ├── pyproject.toml
│           ├── resources.json
│           └── uv.lock
├── _virtualenv.pth
└── _virtualenv.py

Hence we can see how the lambda handlers get moved up from their expected directories to the root. A fix to this would likely also take care of #4003 #4062

@cosmic-ascendant
Copy link

cosmic-ascendant commented Oct 16, 2024 •
edited
Loading

For zip deployments, one could do something ugly like this temporarily:

  async run() {
    const functionA = new sst.aws.Function("FunctionA", {
      python: {
        container: false,
      },
      handler: "packages/functions/function_a/lambda.handler",
      runtime: "python3.11",
      url: true,
      copyFiles: [{ from: "packages/functions/function_a/lambda.py", to: "packages/functions/function_a/lambda.py" }]
    });

Which does allow allow the handler to run with its dependencies. But would break resource linking.

@thdxr thdxr transferred this issue from sst/ion Oct 21, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@thdxr thdxr

Labels
None yet
Projects
None yet
Milestone
No milestone
4 participants
@fwang @cmanou @thdxr @cosmic-ascendant