Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

eval: add benchmarks #387

Merged
merged 1 commit into from
Sep 12, 2024
Merged

eval: add benchmarks #387

merged 1 commit into from
Sep 12, 2024

Conversation

pgavlin
Copy link
Member

@pgavlin pgavlin commented Sep 11, 2024

These changes add four benchmarks for the evaluator: one for the evalutator itself, one focused on fn::open, one focused on environment loading, and one for the entire stack. Each benchmark uses the same set of environments with different parameters for the time taken by an fn::open call or an environment load.

Local results are below.

goos: darwin
goarch: arm64
pkg: github.com/pulumi/esc/eval
cpu: Apple M1 Max
BenchmarkEval-10           	    196	  6029274 ns/op	4626576 B/op	  23578 allocs/op
BenchmarkEval-10           	    195	  6051825 ns/op	4625788 B/op	  23576 allocs/op
BenchmarkEval-10           	    195	  6041508 ns/op	4625803 B/op	  23576 allocs/op
BenchmarkEval-10           	    195	  6070728 ns/op	4625819 B/op	  23577 allocs/op
BenchmarkEval-10           	    194	  6065964 ns/op	4625696 B/op	  23577 allocs/op
BenchmarkEval-10           	    196	  6075660 ns/op	4626653 B/op	  23578 allocs/op
BenchmarkEval-10           	    195	  6161176 ns/op	4626057 B/op	  23578 allocs/op
BenchmarkEval-10           	    194	  6123274 ns/op	4625977 B/op	  23577 allocs/op
BenchmarkEval-10           	    194	  6098131 ns/op	4626670 B/op	  23578 allocs/op
BenchmarkEval-10           	    196	  6064363 ns/op	4626654 B/op	  23579 allocs/op
BenchmarkEvalOpen-10       	      9	124478532 ns/op	4629023 B/op	  23614 allocs/op
BenchmarkEvalOpen-10       	      9	124931037 ns/op	4630025 B/op	  23609 allocs/op
BenchmarkEvalOpen-10       	      9	123642606 ns/op	4634537 B/op	  23618 allocs/op
BenchmarkEvalOpen-10       	      9	124292352 ns/op	4631007 B/op	  23611 allocs/op
BenchmarkEvalOpen-10       	      9	122027546 ns/op	4633108 B/op	  23617 allocs/op
BenchmarkEvalOpen-10       	      9	123727792 ns/op	4630472 B/op	  23613 allocs/op
BenchmarkEvalOpen-10       	      9	121888051 ns/op	4630566 B/op	  23617 allocs/op
BenchmarkEvalOpen-10       	      9	123239685 ns/op	4630572 B/op	  23619 allocs/op
BenchmarkEvalOpen-10       	      9	124360343 ns/op	4631780 B/op	  23616 allocs/op
BenchmarkEvalOpen-10       	      9	122856963 ns/op	4631718 B/op	  23613 allocs/op
BenchmarkEvalEnvLoad-10    	      4	305906031 ns/op	4634742 B/op	  23643 allocs/op
BenchmarkEvalEnvLoad-10    	      4	304883292 ns/op	4636056 B/op	  23644 allocs/op
BenchmarkEvalEnvLoad-10    	      4	309884938 ns/op	4637584 B/op	  23643 allocs/op
BenchmarkEvalEnvLoad-10    	      4	301879375 ns/op	4634852 B/op	  23632 allocs/op
BenchmarkEvalEnvLoad-10    	      4	301303094 ns/op	4632002 B/op	  23639 allocs/op
BenchmarkEvalEnvLoad-10    	      4	303479062 ns/op	4638610 B/op	  23647 allocs/op
BenchmarkEvalEnvLoad-10    	      4	303368458 ns/op	4634654 B/op	  23635 allocs/op
BenchmarkEvalEnvLoad-10    	      4	304764969 ns/op	4637982 B/op	  23655 allocs/op
BenchmarkEvalEnvLoad-10    	      4	308650562 ns/op	4636046 B/op	  23640 allocs/op
BenchmarkEvalEnvLoad-10    	      4	305385281 ns/op	4629074 B/op	  23628 allocs/op
BenchmarkEvalAll-10        	      3	421889528 ns/op	4647314 B/op	  23686 allocs/op
BenchmarkEvalAll-10        	      3	412783556 ns/op	4639096 B/op	  23683 allocs/op
BenchmarkEvalAll-10        	      3	418679208 ns/op	4642520 B/op	  23682 allocs/op
BenchmarkEvalAll-10        	      3	413973958 ns/op	4632093 B/op	  23650 allocs/op
BenchmarkEvalAll-10        	      3	415022083 ns/op	4645818 B/op	  23679 allocs/op
BenchmarkEvalAll-10        	      3	421364694 ns/op	4633874 B/op	  23662 allocs/op
BenchmarkEvalAll-10        	      3	413825070 ns/op	4634840 B/op	  23661 allocs/op
BenchmarkEvalAll-10        	      3	424170639 ns/op	4630058 B/op	  23666 allocs/op
BenchmarkEvalAll-10        	      3	421530069 ns/op	4638128 B/op	  23684 allocs/op
BenchmarkEvalAll-10        	      3	416247500 ns/op	4638744 B/op	  23670 allocs/op

@pgavlin pgavlin added the impact/no-changelog-required This issue doesn't require a CHANGELOG update label Sep 11, 2024
@pgavlin pgavlin requested a review from komalali September 11, 2024 22:15
These changes add four benchmarks for the evaluator: one for the
evalutator itself, one focused on `fn::open`, one focused on
environment loading, and one for the entire stack. Each
benchmark uses the same set of environments with different
parameters for the time taken by an `fn::open` call or an
environment load.

Local results are below.

goos: darwin
goarch: arm64
pkg: github.com/pulumi/esc/eval
cpu: Apple M1 Max
BenchmarkEval-10           	    196	  6029274 ns/op	4626576 B/op	  23578 allocs/op
BenchmarkEval-10           	    195	  6051825 ns/op	4625788 B/op	  23576 allocs/op
BenchmarkEval-10           	    195	  6041508 ns/op	4625803 B/op	  23576 allocs/op
BenchmarkEval-10           	    195	  6070728 ns/op	4625819 B/op	  23577 allocs/op
BenchmarkEval-10           	    194	  6065964 ns/op	4625696 B/op	  23577 allocs/op
BenchmarkEval-10           	    196	  6075660 ns/op	4626653 B/op	  23578 allocs/op
BenchmarkEval-10           	    195	  6161176 ns/op	4626057 B/op	  23578 allocs/op
BenchmarkEval-10           	    194	  6123274 ns/op	4625977 B/op	  23577 allocs/op
BenchmarkEval-10           	    194	  6098131 ns/op	4626670 B/op	  23578 allocs/op
BenchmarkEval-10           	    196	  6064363 ns/op	4626654 B/op	  23579 allocs/op
BenchmarkEvalOpen-10       	      9	124478532 ns/op	4629023 B/op	  23614 allocs/op
BenchmarkEvalOpen-10       	      9	124931037 ns/op	4630025 B/op	  23609 allocs/op
BenchmarkEvalOpen-10       	      9	123642606 ns/op	4634537 B/op	  23618 allocs/op
BenchmarkEvalOpen-10       	      9	124292352 ns/op	4631007 B/op	  23611 allocs/op
BenchmarkEvalOpen-10       	      9	122027546 ns/op	4633108 B/op	  23617 allocs/op
BenchmarkEvalOpen-10       	      9	123727792 ns/op	4630472 B/op	  23613 allocs/op
BenchmarkEvalOpen-10       	      9	121888051 ns/op	4630566 B/op	  23617 allocs/op
BenchmarkEvalOpen-10       	      9	123239685 ns/op	4630572 B/op	  23619 allocs/op
BenchmarkEvalOpen-10       	      9	124360343 ns/op	4631780 B/op	  23616 allocs/op
BenchmarkEvalOpen-10       	      9	122856963 ns/op	4631718 B/op	  23613 allocs/op
BenchmarkEvalEnvLoad-10    	      4	305906031 ns/op	4634742 B/op	  23643 allocs/op
BenchmarkEvalEnvLoad-10    	      4	304883292 ns/op	4636056 B/op	  23644 allocs/op
BenchmarkEvalEnvLoad-10    	      4	309884938 ns/op	4637584 B/op	  23643 allocs/op
BenchmarkEvalEnvLoad-10    	      4	301879375 ns/op	4634852 B/op	  23632 allocs/op
BenchmarkEvalEnvLoad-10    	      4	301303094 ns/op	4632002 B/op	  23639 allocs/op
BenchmarkEvalEnvLoad-10    	      4	303479062 ns/op	4638610 B/op	  23647 allocs/op
BenchmarkEvalEnvLoad-10    	      4	303368458 ns/op	4634654 B/op	  23635 allocs/op
BenchmarkEvalEnvLoad-10    	      4	304764969 ns/op	4637982 B/op	  23655 allocs/op
BenchmarkEvalEnvLoad-10    	      4	308650562 ns/op	4636046 B/op	  23640 allocs/op
BenchmarkEvalEnvLoad-10    	      4	305385281 ns/op	4629074 B/op	  23628 allocs/op
BenchmarkEvalAll-10        	      3	421889528 ns/op	4647314 B/op	  23686 allocs/op
BenchmarkEvalAll-10        	      3	412783556 ns/op	4639096 B/op	  23683 allocs/op
BenchmarkEvalAll-10        	      3	418679208 ns/op	4642520 B/op	  23682 allocs/op
BenchmarkEvalAll-10        	      3	413973958 ns/op	4632093 B/op	  23650 allocs/op
BenchmarkEvalAll-10        	      3	415022083 ns/op	4645818 B/op	  23679 allocs/op
BenchmarkEvalAll-10        	      3	421364694 ns/op	4633874 B/op	  23662 allocs/op
BenchmarkEvalAll-10        	      3	413825070 ns/op	4634840 B/op	  23661 allocs/op
BenchmarkEvalAll-10        	      3	424170639 ns/op	4630058 B/op	  23666 allocs/op
BenchmarkEvalAll-10        	      3	421530069 ns/op	4638128 B/op	  23684 allocs/op
BenchmarkEvalAll-10        	      3	416247500 ns/op	4638744 B/op	  23670 allocs/op
@pgavlin pgavlin enabled auto-merge (squash) September 12, 2024 17:31
@pgavlin pgavlin merged commit 840f7c0 into main Sep 12, 2024
4 checks passed
@pgavlin pgavlin deleted the pgavlin/eval-bench branch September 12, 2024 17:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
impact/no-changelog-required This issue doesn't require a CHANGELOG update
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants