Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Version comparison, but normalized to (best of each "era"? or maybe even separate points for v0.4.0/0.5.x vs. 1.x.x?) #60

Open
jowens opened this issue May 8, 2020 · 26 comments
Assignees

Comments

@jowens
Copy link
Contributor

jowens commented May 8, 2020

I want to clearly be able to tell where v1.x.x is worse (or better) than older versions. Maybe taking a single GPU result (like K40c/80 --- because that's what we have used in past the most), and comparison the results for all datasets and all primitives.

I am considering this as two versions:

  • Everything 1.0 and beyond = 1.x.x
  • Everything before 1.0 = 0.x.x (0.4.0 and 0.5.x)

Because 0.5.x was a feature release.

EXCLUDING:

  • datasets that don't have results for both versions
  • 0.3.0 results, I almost know nothing about the code of this version and I don't know how relevant it is now.
@jowens
Copy link
Contributor Author

jowens commented May 24, 2020

@neoblizz : would you recommend doing this on ... K40? V100? Both? We have enough data for both? I think we probably do.

@neoblizz
Copy link
Member

Please do it with both. We have enough data for V100 and K40c, and maybe even some Titans (Xp, V).

@jowens
Copy link
Contributor Author

jowens commented May 26, 2020

OK. And just run this over ... the entire io repo?

Basically this graph, yes? gunrock/gunrock#642 (comment)

@jowens
Copy link
Contributor Author

jowens commented May 26, 2020

(Not quite that graph. But something not too different from that graph. Would be helpful to be able to pick a subset of gunrock/io to search through, if you knew where to look for pre-1.0 stuff.)

@jowens
Copy link
Contributor Author

jowens commented May 26, 2020

Ah, that's helpful, thank you.

@jowens jowens self-assigned this May 26, 2020
@jowens
Copy link
Contributor Author

jowens commented May 28, 2020

OK, I ran these over the entire database. These 8 plots are consistent with gunrock/gunrock#642 , mostly. There are no K40 comparisons. Right now there is no differentiation between different options (undirected, mark_pred, etc.); it just takes the fastest one. Anything "above" the line at 1 is where a pre-1.0 version is faster than our fastest 1.0+ version.

There are no PageRank comparisons on K40 AFAICT.

If I'm missing something, let me know. Tell me what I need to do better, @neoblizz ?

gunrock_version_compare_sssp_Tesla K40_80
gunrock_version_compare_sssp_Tesla V100
gunrock_version_compare_pr_Tesla K40_80
gunrock_version_compare_pr_Tesla V100
gunrock_version_compare_bfs_Tesla K40_80
gunrock_version_compare_bfs_Tesla V100
gunrock_version_compare_bc_Tesla K40_80
gunrock_version_compare_bc_Tesla V100

@jowens
Copy link
Contributor Author

jowens commented May 28, 2020

(These look really scant, but our data is pretty scant. We didn't run over very many datasets until we got to 1.0.)

@jowens
Copy link
Contributor Author

jowens commented May 28, 2020

(Easy to leave out primitives that don't have comparisons.)

@neoblizz
Copy link
Member

That's ok, I like the list of datasets we have. You can leave that. Also, hard for me to believe there's no PR run on K40, I'll look for that. These look perfect, they have a 1000 times more value to me when I can hover over it and actually see the exact config for that version/commit.

@neoblizz
Copy link
Member

Off-topic: How difficult is it to set-up your script on say daisy? If we can create a GitHub action for this kind of graph, where whenever a new result is pushed onto a certain directory, it automatically updates these graphs and makes them live -- that will be really neat.

@jowens
Copy link
Contributor Author

jowens commented May 28, 2020

It is certainly helpful if you point to a run and say "why isn't that included in the comparison".

We could set it up on daisy. We'd have to install a bunch of stuff, but no biggie. (Altair etc. uses a lot of nodejs stuff, Selenium for headless PDF rendering, etc.).

@jowens
Copy link
Contributor Author

jowens commented May 28, 2020

(There are no results at all for PR/K40 on any Gunrock version, so clearly an error on my part, I'll look into that.)

@neoblizz
Copy link
Member

(There are no results at all for PR/K40 on any Gunrock version, so clearly an error on my part, I'll look into that.)

https://github.com/gunrock/io/blob/master/gunrock-output/topc/PageRank_hollywood-2009_Thu%20Nov%2017%20230545%202016.json#L45 here's an example.

@jowens
Copy link
Contributor Author

jowens commented May 28, 2020

On it.

@jowens
Copy link
Contributor Author

jowens commented May 28, 2020

Ah. OK. After much debugging, it turns out the reason we don't have any PR speedups to show on K40 is that we don't have any 1.0+ PR results at all on K40. Sooooo that makes sense.

https://github.com/gunrock/io/tree/master/gunrock-output/v1-0-0/pr

@jowens
Copy link
Contributor Author

jowens commented May 28, 2020

And now this is more what I think we want. For each primitive, I have an "all" plot (one row, fastest for each primitive/GPU combo). Then I have a more detailed plot that separates out the options (which are different for each primitive) (multiple rows, one for each primitive/GPU/options combo). Gonna paste in one comment per primitive now. @neoblizz lemme know what you think.

@jowens
Copy link
Contributor Author

jowens commented May 28, 2020

gunrock_version_compare_bc_Tesla K40_80_all
gunrock_version_compare_bc_Tesla K40_80_undirected
gunrock_version_compare_bc_Tesla V100_all
gunrock_version_compare_bc_Tesla V100_undirected

@jowens
Copy link
Contributor Author

jowens commented May 28, 2020

gunrock_version_compare_bfs_Tesla K40_80_all
gunrock_version_compare_bfs_Tesla K40_80_undirected_idempotence_markpred
gunrock_version_compare_bfs_Tesla V100_all
gunrock_version_compare_bfs_Tesla V100_undirected_idempotence_markpred

@jowens
Copy link
Contributor Author

jowens commented May 28, 2020

gunrock_version_compare_pr_Tesla K40_80_all
gunrock_version_compare_pr_Tesla V100_all
gunrock_version_compare_pr_Tesla V100_undirected_pull

@jowens
Copy link
Contributor Author

jowens commented May 28, 2020

gunrock_version_compare_sssp_Tesla K40_80_all
gunrock_version_compare_sssp_Tesla K40_80_undirected_markpred
gunrock_version_compare_sssp_Tesla V100_all
gunrock_version_compare_sssp_Tesla V100_undirected_markpred

jowens added a commit to gunrock/docs that referenced this issue May 28, 2020
Added gunrock-version comparisons gunrock/io#60
@neoblizz
Copy link
Member

This is great, I really like this. Very easy to see what's going on.
I will run PR on K40c on Luigi and get you some results for a baseline.

@jowens
Copy link
Contributor Author

jowens commented May 28, 2020

I can't believe how many tests you ran, and still these plots are pretty sparse. Ah well. We have plenty of data points to explore though, and they're ~the same as the ones we found a couple of months back that @crozhon is looking at, so that's a great outcome.

@neoblizz
Copy link
Member

neoblizz commented May 28, 2020

This also helps me see what datasets we are missing for some primitives. I would really like to fully automate this process using actions and have some sort of feedback/report system that can inform is if things go wrong or stuff is missing.

It's a bit of effort to get that all set up though. But worth it.

@jowens
Copy link
Contributor Author

jowens commented May 28, 2020

Yeahhhhh I just worry that it's a lot of time to set up and it's not going to repay the effort it takes to set it up. I mean, we don't check in new results all that often. And every time we add something new I have to go muck with the scripts to make them support it anyway ... I guess I'd like to know that someone is going to use this a fair bit before we invest that time.

@jowens
Copy link
Contributor Author

jowens commented Jun 8, 2020

The thing we could do that's perhaps a little more straightforward is a non-graph-based python script that basically prints out a comparison. If the script was to input "here are the new files you checked in", it could perhaps print out a summary for each experiment ("the fastest previous run you did was X"). That would require, however, that we'd have presumably have a local copy of gunrock-output somewhere; not sure it would be kind to slurp tens of thousands of files from github every time we ran the script. I think this would be more useful than a graph, frankly. But, it's not particularly high priority.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants