Draft: create CAtomMeta to avoid getattr when creating Atoms #244

frmdstryr · 2025-02-06T21:53:47Z

Currently CAtom_new relies on the metaclass to set __atom_members__ on the type and must use getattr on the type read the size in order to set the slot count.

This adds a CAtomMeta that subclasses the builtin PyType_Type and updates the existing AtomMeta to use that.

This means it can simply do a TypeCheck and then access the slot_count on the type. I see a repeatable speedup of several percent when creating atom objects.

With current master

IPython 8.31.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: from atom.api import *

In [2]: class Test(Atom):
   ...:     first_name = Str("First")
   ...:     last_name = Str("Last")
   ...:     age = Range(low=0)
   ...:     debug = Bool(False)
   ...:     items = List(default=[1, 2, 3])
   ...: 

In [3]: %timeit Test()
118 ns ± 9.84 ns per loop (mean ± std. dev. of 7 runs, 10,000,000 loops each)

In [4]: %timeit Test()
129 ns ± 10.5 ns per loop (mean ± std. dev. of 7 runs, 10,000,000 loops each)

In [5]: %timeit Test()
122 ns ± 7.26 ns per loop (mean ± std. dev. of 7 runs, 10,000,000 loops each)

In [6]: %timeit Test()
127 ns ± 8.6 ns per loop (mean ± std. dev. of 7 runs, 10,000,000 loops each)

With this branch (and logic to avoid the second write in #243)

IPython 8.31.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: from atom.api import *

In [2]: class Test(Atom):
   ...:     first_name = Str("First")
   ...:     last_name = Str("Last")
   ...:     age = Range(low=0)
   ...:     debug = Bool(False)
   ...:     items = List(default=[1, 2, 3])
   ...: 

In [3]: %timeit Test()
95.6 ns ± 5.43 ns per loop (mean ± std. dev. of 7 runs, 10,000,000 loops each)

In [4]: %timeit Test()
104 ns ± 6.38 ns per loop (mean ± std. dev. of 7 runs, 10,000,000 loops each)

In [5]: %timeit Test()
105 ns ± 7.8 ns per loop (mean ± std. dev. of 7 runs, 10,000,000 loops each)

In [6]: %timeit Test()
106 ns ± 7.71 ns per loop (mean ± std. dev. of 7 runs, 10,000,000 loops each)

However unfortunately I actually see a slight slowdown in my real application.. which has me confused. Is enaml creating subclasses at runtime?

sccolbert · 2025-02-06T23:02:35Z

Do you have a case where the current code is causing a bottleneck? How many Atom subclasses are you defining? Even at 10,000 classes, that still only a total runtime of 1ms.

Edit: I think I'm missing what you are trying to do here. Let me look a bit closer.

sccolbert · 2025-02-06T23:06:25Z

But yes, there are times when Enaml will create subclasses at runtime, but it shouldn't be too often unless you've hit a weird edge/pathological case.

frmdstryr · 2025-02-07T00:53:06Z

I'm trying to optimize since people are complaining my app is slow... since everything is built on atom/enaml even small improvements get multiplied a lot and typically across the whole app. The app makes like 25k nodes (enaml-web) per page. enaml-web defines about 100 subclases but all are done at import time, which is why I'm surprised this PR is slower.

This change is probably not worth the extra code given it needs millions of atoms to make any real timing difference (and for some reason for my app it makes things worse) but if you compare to atom to a builtin class with slots atom is already like ~20% slower (that's with this PR without its even more in the main branch).

sccolbert · 2025-02-07T01:08:22Z

20% slower makes sense. Atom meta has to do a lot more to set up the class than a normal Python class. The whole point is to do as much of that work as possible at class creation time since that should only happen a few thousands times per app, as opposed to maybe millions of object instantiations. Have you run a profile on your app to identify the hot spots? Maybe you would benefit more from some app/logic changes to be more efficient, instead of chasing micro-optimizations in Enaml?

…

On Thu, Feb 6, 2025 at 6:53 PM frmdstryr ***@***.***> wrote: I'm trying to optimize since people are complaining my app is slow... since everything is built on atom/enaml even small improvements get multiplied a lot and typically across the whole app. The app makes like 25k nodes (enaml-web) per page. enaml-web defines about 100 subclases but all are done at import time, which is why I'm surprised this PR is slower. This change is probably not worth the extra code given it needs millions of atoms to make any real timing difference (and for some reason for my app it makes things worse) but if you compare to atom to a builtin class with slots atom is already like ~20% slower (that's with this PR without its even more in the main branch). — Reply to this email directly, view it on GitHub <#244 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AABBQSP3D3JPOYVVWFXKHIL2OP7YPAVCNFSM6AAAAABWUR7BYWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMNBRGU2DCNJZG4> . You are receiving this because you commented.Message ID: ***@***.***>

frmdstryr · 2025-02-07T01:22:55Z

I suppose I can start rewriting more enaml-web stuff in C... but there's not really much to work with

         6293037 function calls (6045518 primitive calls) in 17.274 seconds

   Ordered by: cumulative time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
       45    0.000    0.000   15.835    0.352 /home/user/micromamba/envs/app/lib/python3.12/site-packages/tornado/web.py:1746(_execute)
       29    0.000    0.000   15.714    0.542 /home/user/app/handlers.py:123(get)
    150/1    0.001    0.000   15.490   15.490 /home/user/projects/enaml-web/web/components/html.py:319(render)
    150/1    0.001    0.000   15.471   15.471 /home/user/projects/enaml-web/web/components/html.py:305(prepare)
  22514/1    0.263    0.000   12.522   12.522 /home/user/projects/enaml/enaml/widgets/toolkit_object.py:201(activate_proxy)
22514/19475    0.060    0.000   12.012    0.001 /home/user/projects/enaml/enaml/widgets/toolkit_object.py:218(activate_top_down)
22514/19475    0.090    0.000   11.960    0.001 /home/user/projects/enaml-web/web/impl/lxml_toolkit_object.py:128(activate_top_down)
22514/19475    1.261    0.000   10.954    0.001 /home/user/projects/enaml-web/web/impl/lxml_toolkit_object.py:73(init_widget)
449396/385311    1.607    0.000    8.408    0.000 /home/user/projects/enaml/enaml/core/declarative_meta.py:27(__call__)
400730/354879    1.133    0.000    6.384    0.000 /home/user/projects/enaml/enaml/core/expression_engine.py:157(read)
48272/45052    0.292    0.000    4.770    0.000 /home/user/projects/enaml/enaml/core/standard_handlers.py:91(__call__)
69044/54742    0.201    0.000    4.285    0.000 {built-in method enaml.core.funchelper.call_func}
22515/150    0.129    0.000    3.029    0.020 /home/user/projects/enaml/enaml/widgets/toolkit_object.py:147(initialize)
24811/150    0.174    0.000    3.025    0.020 /home/user/projects/enaml/enaml/core/declarative.py:120(initialize)
     10/4    0.000    0.000    2.943    0.736 /home/user/projects/enaml-web/web/core/block.py:37(initialize)
  2286/61    0.033    0.000    2.925    0.048 /home/user/projects/enaml/enaml/core/pattern.py:33(initialize)
100510/99651    0.290    0.000    2.594    0.000 {built-in method builtins.getattr}
     1823    0.221    0.000    2.264    0.001 /home/user/projects/enaml/enaml/core/looper.py:138(refresh_items)
       60    0.005    0.000    1.698    0.028 /home/user/micromamba/envs/app/lib/python3.12/site-packages/atomdb/sql.py:1420(all)
   419612    1.047    0.000    1.583    0.000 /home/user/projects/enaml-web/web/components/html.py:157(_update_proxy)
   237443    0.654    0.000    1.548    0.000 /home/user/projects/enaml-web/web/components/html.py:839(_update_proxy)
      134    0.001    0.000    1.259    0.009 /home/user/app/views/grid.enaml:294(format_popover)
2023/1449    0.023    0.000    1.160    0.001 /home/user/micromamba/envs/app/lib/python3.12/site-packages/atomdb/sql.py:1915(restore)
2443/1396    0.007    0.000    1.094    0.001 /home/user/micromamba/envs/app/lib/python3.12/site-packages/atomdb/base.py:725(__restorestate__)
    22364    0.254    0.000    1.054    0.000 /home/user/projects/enaml-web/web/impl/lxml_toolkit_object.py:59(create_widget)
    69288    0.355    0.000    0.883    0.000 /home/user/projects/enaml/enaml/core/standard_tracer.py:95(load_attr)
24512/14223    0.135    0.000    0.842    0.000 /home/user/projects/enaml/enaml/core/compiler_nodes.py:154(__call__)
   567968    0.790    0.000    0.790    0.000 {method 'get' of 'atom.catom.sortedmap.sortedmap' objects}
   386533    0.584    0.000    0.735    0.000 {method 'do_default_value' of 'atom.catom.Member' objects}
26127/14221    0.142    0.000    0.721    0.000 /home/user/projects/enaml/enaml/core/compiler_nodes.py:177(populate)
    48301    0.124    0.000    0.702    0.000 /home/user/projects/enaml/enaml/core/standard_tracer.py:117(return_value)
       58    0.001    0.000    0.643    0.011 /home/user/micromamba/envs/app/lib/python3.12/site-packages/aiomysql/sa/connection.py:135(_execute)
    98453    0.385    0.000    0.642    0.000 /home/user/projects/enaml/enaml/core/standard_tracer.py:36(trace_atom)
   491778    0.625    0.000    0.625    0.000 /home/user/projects/enaml/enaml/core/object.py:36(getter)

sccolbert · 2025-02-07T01:37:18Z

What i mean is can you refactor the logic of your application so it simply doesnt make as many calls? I dont know your application, but if you are creating 25k enaml objects to represent 25k nodes in a DOM tree, that is not going to be efficient. I dont know of any QT application that has 25k widgets on the screen at one time, and Enaml was designed for desktop GUIs. You may need to write some javascript to encapsulate a web “widget” thats composed of several or dozens of separate DOM nodes, and then create *that* widget from Enaml.

…

On Thu, Feb 6, 2025 at 19:23 frmdstryr ***@***.***> wrote: I suppose I can start rewriting more enaml-web stuff in C... but there's not really much to work with 6293037 function calls (6045518 primitive calls) in 17.274 seconds Ordered by: cumulative time ncalls tottime percall cumtime percall filename:lineno(function) 45 0.000 0.000 15.835 0.352 /home/user/micromamba/envs/app/lib/python3.12/site-packages/tornado/web.py:1746(_execute) 29 0.000 0.000 15.714 0.542 /home/user/app/handlers.py:123(get) 150/1 0.001 0.000 15.490 15.490 /home/user/projects/enaml-web/web/components/html.py:319(render) 150/1 0.001 0.000 15.471 15.471 /home/user/projects/enaml-web/web/components/html.py:305(prepare) 22514/1 0.263 0.000 12.522 12.522 /home/user/projects/enaml/enaml/widgets/toolkit_object.py:201(activate_proxy) 22514/19475 0.060 0.000 12.012 0.001 /home/user/projects/enaml/enaml/widgets/toolkit_object.py:218(activate_top_down) 22514/19475 0.090 0.000 11.960 0.001 /home/user/projects/enaml-web/web/impl/lxml_toolkit_object.py:128(activate_top_down) 22514/19475 1.261 0.000 10.954 0.001 /home/user/projects/enaml-web/web/impl/lxml_toolkit_object.py:73(init_widget) 449396/385311 1.607 0.000 8.408 0.000 /home/user/projects/enaml/enaml/core/declarative_meta.py:27(__call__) 400730/354879 1.133 0.000 6.384 0.000 /home/user/projects/enaml/enaml/core/expression_engine.py:157(read) 48272/45052 0.292 0.000 4.770 0.000 /home/user/projects/enaml/enaml/core/standard_handlers.py:91(__call__) 69044/54742 0.201 0.000 4.285 0.000 {built-in method enaml.core.funchelper.call_func} 22515/150 0.129 0.000 3.029 0.020 /home/user/projects/enaml/enaml/widgets/toolkit_object.py:147(initialize) 24811/150 0.174 0.000 3.025 0.020 /home/user/projects/enaml/enaml/core/declarative.py:120(initialize) 10/4 0.000 0.000 2.943 0.736 /home/user/projects/enaml-web/web/core/block.py:37(initialize) 2286/61 0.033 0.000 2.925 0.048 /home/user/projects/enaml/enaml/core/pattern.py:33(initialize) 100510/99651 0.290 0.000 2.594 0.000 {built-in method builtins.getattr} 1823 0.221 0.000 2.264 0.001 /home/user/projects/enaml/enaml/core/looper.py:138(refresh_items) 60 0.005 0.000 1.698 0.028 /home/user/micromamba/envs/app/lib/python3.12/site-packages/atomdb/sql.py:1420(all) 419612 1.047 0.000 1.583 0.000 /home/user/projects/enaml-web/web/components/html.py:157(_update_proxy) 237443 0.654 0.000 1.548 0.000 /home/user/projects/enaml-web/web/components/html.py:839(_update_proxy) 134 0.001 0.000 1.259 0.009 /home/user/app/views/grid.enaml:294(format_popover) 2023/1449 0.023 0.000 1.160 0.001 /home/user/micromamba/envs/app/lib/python3.12/site-packages/atomdb/sql.py:1915(restore) 2443/1396 0.007 0.000 1.094 0.001 /home/user/micromamba/envs/app/lib/python3.12/site-packages/atomdb/base.py:725(__restorestate__) 22364 0.254 0.000 1.054 0.000 /home/user/projects/enaml-web/web/impl/lxml_toolkit_object.py:59(create_widget) 69288 0.355 0.000 0.883 0.000 /home/user/projects/enaml/enaml/core/standard_tracer.py:95(load_attr) 24512/14223 0.135 0.000 0.842 0.000 /home/user/projects/enaml/enaml/core/compiler_nodes.py:154(__call__) 567968 0.790 0.000 0.790 0.000 {method 'get' of 'atom.catom.sortedmap.sortedmap' objects} 386533 0.584 0.000 0.735 0.000 {method 'do_default_value' of 'atom.catom.Member' objects} 26127/14221 0.142 0.000 0.721 0.000 /home/user/projects/enaml/enaml/core/compiler_nodes.py:177(populate) 48301 0.124 0.000 0.702 0.000 /home/user/projects/enaml/enaml/core/standard_tracer.py:117(return_value) 58 0.001 0.000 0.643 0.011 /home/user/micromamba/envs/app/lib/python3.12/site-packages/aiomysql/sa/connection.py:135(_execute) 98453 0.385 0.000 0.642 0.000 /home/user/projects/enaml/enaml/core/standard_tracer.py:36(trace_atom) 491778 0.625 0.000 0.625 0.000 /home/user/projects/enaml/enaml/core/object.py:36(getter) — Reply to this email directly, view it on GitHub <#244 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AABBQSODHWKQMLLVTONPXXT2OQDINAVCNFSM6AAAAABWUR7BYWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMNBRGYYTINBTG4> . You are receiving this because you commented.Message ID: ***@***.***>

frmdstryr · 2025-02-11T15:47:30Z

Good point, I'll rethink my approach.

Create CAtomMeta to avoid getattr when creating Atoms

cb7829e

frmdstryr closed this Feb 11, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

Draft: create CAtomMeta to avoid getattr when creating Atoms #244

Draft: create CAtomMeta to avoid getattr when creating Atoms #244

Uh oh!

frmdstryr commented Feb 6, 2025

Uh oh!

sccolbert commented Feb 6, 2025 •

edited

Loading

Uh oh!

sccolbert commented Feb 6, 2025 •

edited

Loading

Uh oh!

frmdstryr commented Feb 7, 2025

Uh oh!

sccolbert commented Feb 7, 2025 via email

Uh oh!

frmdstryr commented Feb 7, 2025

Uh oh!

sccolbert commented Feb 7, 2025 via email

Uh oh!

frmdstryr commented Feb 11, 2025

Uh oh!

Uh oh!

Uh oh!

Draft: create CAtomMeta to avoid getattr when creating Atoms #244

Draft: create CAtomMeta to avoid getattr when creating Atoms #244

Uh oh!

Conversation

frmdstryr commented Feb 6, 2025

Uh oh!

sccolbert commented Feb 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sccolbert commented Feb 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

frmdstryr commented Feb 7, 2025

Uh oh!

sccolbert commented Feb 7, 2025 via email

Uh oh!

frmdstryr commented Feb 7, 2025

Uh oh!

sccolbert commented Feb 7, 2025 via email

Uh oh!

frmdstryr commented Feb 11, 2025

Uh oh!

Uh oh!

sccolbert commented Feb 6, 2025 •

edited

Loading

sccolbert commented Feb 6, 2025 •

edited

Loading