Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

eval: performance improvements #392

Merged
merged 2 commits into from
Sep 17, 2024
Merged

eval: performance improvements #392

merged 2 commits into from
Sep 17, 2024

Conversation

pgavlin
Copy link
Member

@pgavlin pgavlin commented Sep 17, 2024

These changes improve evaluation performance by memoizing exported values and merged object keys and avoiding copies when building object schemas.

This gives a tremendous improvement in execution time for evaluation-dominated scenarios, but results in little to no change for open- or load-dominated scenarios. However, the memoization and avoidance of copies cuts memory usage in half in all cases.

Strictly speaking, this is a breaking change for consumers of the schema package, but I think that it is worth taking given the improvements.

Local benchmark results:

goos: darwin
goarch: arm64
pkg: github.com/pulumi/esc/eval
cpu: Apple M1 Max
BenchmarkEval-10           	    453	  2566938 ns/op	2204129 B/op	  18883 allocs/op
BenchmarkEval-10           	    468	  2586703 ns/op	2204146 B/op	  18884 allocs/op
BenchmarkEval-10           	    462	  2588916 ns/op	2204202 B/op	  18883 allocs/op
BenchmarkEval-10           	    464	  2586365 ns/op	2204228 B/op	  18884 allocs/op
BenchmarkEval-10           	    463	  2577383 ns/op	2204362 B/op	  18884 allocs/op
BenchmarkEval-10           	    477	  2537640 ns/op	2204389 B/op	  18884 allocs/op
BenchmarkEval-10           	    468	  2582930 ns/op	2204594 B/op	  18884 allocs/op
BenchmarkEval-10           	    463	  2582915 ns/op	2204246 B/op	  18884 allocs/op
BenchmarkEval-10           	    469	  2608014 ns/op	2204382 B/op	  18884 allocs/op
BenchmarkEval-10           	    465	  2554270 ns/op	2204313 B/op	  18884 allocs/op
BenchmarkEvalOpen-10       	      9	119163125 ns/op	2208651 B/op	  18926 allocs/op
BenchmarkEvalOpen-10       	      9	118168319 ns/op	2209928 B/op	  18928 allocs/op
BenchmarkEvalOpen-10       	      9	118805454 ns/op	2208294 B/op	  18924 allocs/op
BenchmarkEvalOpen-10       	      9	118506347 ns/op	2208712 B/op	  18922 allocs/op
BenchmarkEvalOpen-10       	      9	118898060 ns/op	2210256 B/op	  18926 allocs/op
BenchmarkEvalOpen-10       	      9	118450250 ns/op	2208210 B/op	  18924 allocs/op
BenchmarkEvalOpen-10       	      9	117723833 ns/op	2207528 B/op	  18922 allocs/op
BenchmarkEvalOpen-10       	      9	117134787 ns/op	2209227 B/op	  18925 allocs/op
BenchmarkEvalOpen-10       	      9	116210269 ns/op	2208843 B/op	  18926 allocs/op
BenchmarkEvalOpen-10       	      9	116987444 ns/op	2208736 B/op	  18925 allocs/op
BenchmarkEvalEnvLoad-10    	      4	298021334 ns/op	2216058 B/op	  18951 allocs/op
BenchmarkEvalEnvLoad-10    	      4	302557979 ns/op	2213974 B/op	  18944 allocs/op
BenchmarkEvalEnvLoad-10    	      4	293050229 ns/op	2212098 B/op	  18945 allocs/op
BenchmarkEvalEnvLoad-10    	      4	304410510 ns/op	2211322 B/op	  18946 allocs/op
BenchmarkEvalEnvLoad-10    	      4	301698562 ns/op	2212554 B/op	  18947 allocs/op
BenchmarkEvalEnvLoad-10    	      4	299588854 ns/op	2214102 B/op	  18946 allocs/op
BenchmarkEvalEnvLoad-10    	      4	295087740 ns/op	2211650 B/op	  18944 allocs/op
BenchmarkEvalEnvLoad-10    	      4	295875531 ns/op	2212638 B/op	  18950 allocs/op
BenchmarkEvalEnvLoad-10    	      4	294871781 ns/op	2212038 B/op	  18945 allocs/op
BenchmarkEvalEnvLoad-10    	      4	294592875 ns/op	2211682 B/op	  18945 allocs/op
BenchmarkEvalAll-10        	      3	405058722 ns/op	2215330 B/op	  18976 allocs/op
BenchmarkEvalAll-10        	      3	407002764 ns/op	2215688 B/op	  18978 allocs/op
BenchmarkEvalAll-10        	      3	409757153 ns/op	2214973 B/op	  18976 allocs/op
BenchmarkEvalAll-10        	      3	404553611 ns/op	2215261 B/op	  18977 allocs/op
BenchmarkEvalAll-10        	      3	402620945 ns/op	2216994 B/op	  18980 allocs/op
BenchmarkEvalAll-10        	      3	405302139 ns/op	2213112 B/op	  18973 allocs/op
BenchmarkEvalAll-10        	      3	404533556 ns/op	2215848 B/op	  18978 allocs/op
BenchmarkEvalAll-10        	      3	403431236 ns/op	2215896 B/op	  18979 allocs/op
BenchmarkEvalAll-10        	      3	402586597 ns/op	2217245 B/op	  18983 allocs/op
BenchmarkEvalAll-10        	      3	404775236 ns/op	2217122 B/op	  18980 allocs/op

@pgavlin pgavlin requested a review from a team September 17, 2024 18:32
These changes improve evaluation performance by memoizing exported
values and merged object keys and avoiding copies when building object
schemas.

This gives a tremendous improvement in execution time for
evaluation-dominated scenarios, but results in little to no change for
open- or load-dominated scenarios.

Local benchmark results:

goos: darwin
goarch: arm64
pkg: github.com/pulumi/esc/eval
cpu: Apple M1 Max
BenchmarkEval-10           	    453	  2566938 ns/op	2204129 B/op	  18883 allocs/op
BenchmarkEval-10           	    468	  2586703 ns/op	2204146 B/op	  18884 allocs/op
BenchmarkEval-10           	    462	  2588916 ns/op	2204202 B/op	  18883 allocs/op
BenchmarkEval-10           	    464	  2586365 ns/op	2204228 B/op	  18884 allocs/op
BenchmarkEval-10           	    463	  2577383 ns/op	2204362 B/op	  18884 allocs/op
BenchmarkEval-10           	    477	  2537640 ns/op	2204389 B/op	  18884 allocs/op
BenchmarkEval-10           	    468	  2582930 ns/op	2204594 B/op	  18884 allocs/op
BenchmarkEval-10           	    463	  2582915 ns/op	2204246 B/op	  18884 allocs/op
BenchmarkEval-10           	    469	  2608014 ns/op	2204382 B/op	  18884 allocs/op
BenchmarkEval-10           	    465	  2554270 ns/op	2204313 B/op	  18884 allocs/op
BenchmarkEvalOpen-10       	      9	119163125 ns/op	2208651 B/op	  18926 allocs/op
BenchmarkEvalOpen-10       	      9	118168319 ns/op	2209928 B/op	  18928 allocs/op
BenchmarkEvalOpen-10       	      9	118805454 ns/op	2208294 B/op	  18924 allocs/op
BenchmarkEvalOpen-10       	      9	118506347 ns/op	2208712 B/op	  18922 allocs/op
BenchmarkEvalOpen-10       	      9	118898060 ns/op	2210256 B/op	  18926 allocs/op
BenchmarkEvalOpen-10       	      9	118450250 ns/op	2208210 B/op	  18924 allocs/op
BenchmarkEvalOpen-10       	      9	117723833 ns/op	2207528 B/op	  18922 allocs/op
BenchmarkEvalOpen-10       	      9	117134787 ns/op	2209227 B/op	  18925 allocs/op
BenchmarkEvalOpen-10       	      9	116210269 ns/op	2208843 B/op	  18926 allocs/op
BenchmarkEvalOpen-10       	      9	116987444 ns/op	2208736 B/op	  18925 allocs/op
BenchmarkEvalEnvLoad-10    	      4	298021334 ns/op	2216058 B/op	  18951 allocs/op
BenchmarkEvalEnvLoad-10    	      4	302557979 ns/op	2213974 B/op	  18944 allocs/op
BenchmarkEvalEnvLoad-10    	      4	293050229 ns/op	2212098 B/op	  18945 allocs/op
BenchmarkEvalEnvLoad-10    	      4	304410510 ns/op	2211322 B/op	  18946 allocs/op
BenchmarkEvalEnvLoad-10    	      4	301698562 ns/op	2212554 B/op	  18947 allocs/op
BenchmarkEvalEnvLoad-10    	      4	299588854 ns/op	2214102 B/op	  18946 allocs/op
BenchmarkEvalEnvLoad-10    	      4	295087740 ns/op	2211650 B/op	  18944 allocs/op
BenchmarkEvalEnvLoad-10    	      4	295875531 ns/op	2212638 B/op	  18950 allocs/op
BenchmarkEvalEnvLoad-10    	      4	294871781 ns/op	2212038 B/op	  18945 allocs/op
BenchmarkEvalEnvLoad-10    	      4	294592875 ns/op	2211682 B/op	  18945 allocs/op
BenchmarkEvalAll-10        	      3	405058722 ns/op	2215330 B/op	  18976 allocs/op
BenchmarkEvalAll-10        	      3	407002764 ns/op	2215688 B/op	  18978 allocs/op
BenchmarkEvalAll-10        	      3	409757153 ns/op	2214973 B/op	  18976 allocs/op
BenchmarkEvalAll-10        	      3	404553611 ns/op	2215261 B/op	  18977 allocs/op
BenchmarkEvalAll-10        	      3	402620945 ns/op	2216994 B/op	  18980 allocs/op
BenchmarkEvalAll-10        	      3	405302139 ns/op	2213112 B/op	  18973 allocs/op
BenchmarkEvalAll-10        	      3	404533556 ns/op	2215848 B/op	  18978 allocs/op
BenchmarkEvalAll-10        	      3	403431236 ns/op	2215896 B/op	  18979 allocs/op
BenchmarkEvalAll-10        	      3	402586597 ns/op	2217245 B/op	  18983 allocs/op
BenchmarkEvalAll-10        	      3	404775236 ns/op	2217122 B/op	  18980 allocs/op
@pgavlin pgavlin enabled auto-merge (squash) September 17, 2024 21:15
@pgavlin pgavlin merged commit a1203bf into main Sep 17, 2024
4 checks passed
@pgavlin pgavlin deleted the pgavlin/eval-perf branch September 17, 2024 21:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants