Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Slow s3 exception raising due to usage pattern of jinja2 #7142

Closed
jogo opened this issue Dec 19, 2023 · 0 comments · Fixed by #7143
Closed

Slow s3 exception raising due to usage pattern of jinja2 #7142

jogo opened this issue Dec 19, 2023 · 0 comments · Fixed by #7143

Comments

@jogo
Copy link
Contributor

jogo commented Dec 19, 2023

import time

from moto.s3.exceptions import MissingBucket

start = time.time()
for _ in range(1000):
    try:
        raise MissingBucket(bucket="foo")
    except MissingBucket:
        pass

print(round(time.time() - start, 2))

with moto 4.2.12: 1.58

cProfile data shows it's related to jinja2 usage. Some refactoring should be able to significantly speed this up.

Tue Dec 19 00:09:04 2023    pstats

         4532388 function calls (4276064 primitive calls) in 2.932 seconds

   Ordered by: cumulative time
   List reduced from 4319 to 30 due to restriction <30>

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
   2658/1    0.010    0.000    2.975    2.975 {built-in method builtins.exec}
        1    0.003    0.003    2.975    2.975 foo.py:1(<module>)
     1000    0.002    0.000    2.501    0.003 /usr/local/lib/python3.8/dist-packages/moto/s3/exceptions.py:103(__init__)
     1000    0.002    0.000    2.499    0.002 /usr/local/lib/python3.8/dist-packages/moto/s3/exceptions.py:76(__init__)
     1000    0.002    0.000    2.496    0.002 /usr/local/lib/python3.8/dist-packages/moto/s3/exceptions.py:44(__init__)
     1000    0.009    0.000    2.494    0.002 /usr/local/lib/python3.8/dist-packages/moto/core/exceptions.py:58(__init__)
     2000    0.002    0.000    2.418    0.001 /usr/local/lib/python3.8/dist-packages/jinja2/environment.py:964(get_template)
     2000    0.007    0.000    2.415    0.001 /usr/local/lib/python3.8/dist-packages/jinja2/environment.py:939(_load_template)
     2000    0.007    0.000    2.393    0.001 /usr/local/lib/python3.8/dist-packages/jinja2/loaders.py:106(load)
     2000    0.008    0.000    2.359    0.001 /usr/local/lib/python3.8/dist-packages/jinja2/environment.py:717(compile)
     1000    0.003    0.000    1.433    0.001 /usr/local/lib/python3.8/dist-packages/jinja2/environment.py:1256(render)
26226/16158    0.014    0.000    1.411    0.000 {method 'join' of 'str' objects}

This means in code that has lots of calls to Moto for buckets that aren't set, there is a noticeable performance impact.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants