Added Centrality Algorithms Example Jupyter Notebook #386

kedarghule · 2023-05-31T22:00:46Z

This PR adds an example jupyter notebook for centrality algorithms on the airline travel reachability network dataset. The notebook explores eigenvector centrality, betweenness centrality & degree centrality.

This is in reference to my comment in the issue: #254

Thank you :)

netlify · 2023-05-31T22:00:52Z

✅ Deploy Preview for neo4j-graph-data-science-client canceled.

Name	Link
🔨 Latest commit	`34d5ea1`
🔍 Latest deploy log	https://app.netlify.com/sites/neo4j-graph-data-science-client/deploys/64915023b2c88b000835a802

kedarghule · 2023-06-05T17:55:13Z

Umm. I don't know why one check is not running :/

Mats-SX · 2023-06-08T08:05:52Z

@kedarghule The check not running is normal. For security reasons we do not run PR builds from non-members. There have been several attacks made on our public repositories to attempt to expose secrets from our build system.

I apologise for the delayed review. We are in the process of releasing GDS 2.4.0 which is taking up most of my time lately. We are grateful for your contribution, and you are not forgotten :)

All the best
Mats

kedarghule · 2023-06-09T20:29:28Z

@kedarghule The check not running is normal. For security reasons we do not run PR builds from non-members. There have been several attacks made on our public repositories to attempt to expose secrets from our build system.

I apologise for the delayed review. We are in the process of releasing GDS 2.4.0 which is taking up most of my time lately. We are grateful for your contribution, and you are not forgotten :)

All the best Mats

Alright @Mats-SX and sure, no problem :)

Mats-SX

Thanks a lot for this contribution! It's a great notebook. I only have a few comments.

I will push a commit that clears output cells and applies our code style rules to the Python code. For reference, we have two scripts in the scripts directory: checkstyle and makestyle that can be used (just run them as scripts/checkstyle without arguments) to manage this in future contributions.

I will also create a sibling PR to allow our CI to run.

Mats-SX · 2023-06-19T08:48:20Z

examples/centrality-algorithms.ipynb

+    "gds.run_cypher(\n",
+    "    \"UNWIND $nodes AS node CREATE (n:City {node_id: node.node_id, name: node.name, population: node.metro_pop})\",\n",
+    "    params={\"nodes\": nodes_info_df.to_dict(\"records\")},\n",
+    ")\n",
+    "\n",
+    "gds.run_cypher(\n",
+    "    \"\"\"\n",
+    "    UNWIND $rels AS rel \n",
+    "    MATCH (source:City {node_id: rel.Origin}), (target:City {node_id: rel.Destination}) \n",
+    "    CREATE (source)-[:HAS_FLIGHT_TO]->(target)\n",
+    "    \"\"\",\n",
+    "    params={\"rels\": routes_df.to_dict(\"records\")},\n",
+    ")"


This method of importing data to the database is a good choice for a graph of this size (small).

Mats-SX · 2023-06-19T08:50:21Z

examples/centrality-algorithms.ipynb

+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Next, we load this data (nodes and edges) into a Graph Database and a GDS graph."


Relating to the comment on the next section, I think it would be good to add a small note here that for graphs of larger size (like, 100k to millions of nodes) other importing measures should be considered (batching, neo4j-admin import or Arrow CREATE DATABASE).

We could just have something like

Since this graph is very small, a straight-forward Cypher UNWIND query is the simplest way to create our graph in the database.
Larger graphs may need a more sophisticated importing technique.

Alright. Done :)

Mats-SX · 2023-06-19T08:52:52Z

examples/centrality-algorithms.ipynb

+    "# We can use convenience methods on `G` to check if the projection looks correct\n",
+    "print(f\"Graph '{G.name()}' node count: {G.node_count()}\")\n",
+    "print(f\"Graph '{G.name()}' node labels: {G.node_labels()}\")"


I think it would be nice to also print the relationshipCount here, and maybe comment that the counts match the expected counts from the SNAP website

Mats-SX · 2023-06-19T08:54:25Z

examples/centrality-algorithms.ipynb

+    "if eigenvector_centrality_result.didConverge: \n",
+    "    print(f\"The number of iterations taken by Eigenvector Centrality to run is {eigenvector_centrality_result.ranIterations}.\")\n",
+    "else:\n",
+    "    print(\"Algorithm did not converge!\")"


very elegant!

Mats-SX · 2023-06-19T08:55:39Z

examples/centrality-algorithms.ipynb

+    {
+     "data": {
+      "text/plain": [
+       "{'p99': 0.08469437807798386,\n",
+       " 'min': 0.0012630745768547058,\n",
+       " 'max': 0.0856785699725151,\n",
+       " 'mean': 0.041094380038741385,\n",
+       " 'p90': 0.07575749605894089,\n",
+       " 'p50': 0.0386042520403862,\n",
+       " 'p999': 0.0856785699725151,\n",
+       " 'p95': 0.08167456835508347,\n",
+       " 'p75': 0.05822562426328659}"
+      ]
+     },


It's better if we commit the notebook without cell outputs. Could you clear these and commit with empty?

Oh I think you already did this for me. Thank you!

Yeah I noticed that our makestyle script does this automatically :)

Mats-SX · 2023-06-19T08:56:48Z

examples/centrality-algorithms.ipynb

+   "source": [
+    "gds.graph.nodeProperties.write(G, [\"eigenvectorCentrality\"])"
+   ]


we should have a small markdown cell before this one that tells the reader that we are about to write our results back to the database

Mats-SX · 2023-06-19T08:59:07Z

examples/centrality-algorithms.ipynb

+   "source": [
+    "gds.graph.nodeProperties.write(G, [\"betweennessCentrality\"])"
+   ]


let's also add a small sentence about this operation

Mats-SX · 2023-06-19T09:01:19Z

examples/centrality-algorithms.ipynb

+   "source": [
+    "### References\n",
+    "- For the network:\n",
+    "Brendan J. Frey and Delbert Dueck. \"Clustering by passing messages between data points.\" Science 315.5814 (2007): 972-976.\n",
+    "\n",
+    "- For the city metadata (metropolitan population, latitude, and longitude):\n",
+    "Austin R. Benson, David F. Gleich, and Jure Leskovec. \"Higher-order Organization of Complex Networks.\" Science, 353.6295 (2016): 163–166.\n",
+    "\n",
+    "- Notebook contributed by [Kedar Ghule](https://github.com/kedarghule)"
+   ]


Good list! Maybe let's also add the link to the SNAP dataset page that we had in the introduction for completeness

kedarghule · 2023-06-19T17:13:08Z

@Mats-SX I have addressed your comments for the PR and made a new commit. Please let me know if anything else is needed or if I missed anything.

Also the latest commit won't run a check again :/

Mats-SX · 2023-06-20T07:07:37Z

examples/centrality-algorithms.ipynb

+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
  "language_info": {
-   "name": "python"
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.10.11"


This stuff was added back in, which is normal when the notebook is run. I will clear it out again.

Mats-SX · 2023-06-20T07:47:47Z

@kedarghule Thank you! I've kicked off another CI build.

Mats-SX self-assigned this Jun 1, 2023

Mats-SX approved these changes Jun 19, 2023

View reviewed changes

kedarghule added 4 commits June 19, 2023 14:00

Started centrality algos notebook

9991e3e

Added eigenvector centrality code

a35f08c

Added betweenness centrality code

d7aa881

Added degree centrality code

f79a36c

Mats-SX force-pushed the centrality-notebook-kedar branch from 2dc053c to 2fdd333 Compare June 19, 2023 12:00

Apply makestyle on notebook

5bbbb46

Mats-SX force-pushed the centrality-notebook-kedar branch from 2fdd333 to 5bbbb46 Compare June 19, 2023 12:20

Mats-SX mentioned this pull request Jun 19, 2023

PR for CI of #386 #403

Merged

addressed PR comments

4234bcc

Clear notebook metadata

34d5ea1

Mats-SX enabled auto-merge June 20, 2023 07:07

Mats-SX reviewed Jun 20, 2023

View reviewed changes

Mats-SX merged commit a410c14 into neo4j:main Jun 20, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Added Centrality Algorithms Example Jupyter Notebook #386

Added Centrality Algorithms Example Jupyter Notebook #386

kedarghule commented May 31, 2023

netlify bot commented May 31, 2023 •

edited

Loading

kedarghule commented Jun 5, 2023

Mats-SX commented Jun 8, 2023

kedarghule commented Jun 9, 2023

Mats-SX left a comment

Mats-SX Jun 19, 2023

Mats-SX Jun 19, 2023

Mats-SX Jun 19, 2023

kedarghule Jun 19, 2023

Mats-SX Jun 19, 2023

kedarghule Jun 19, 2023

Mats-SX Jun 19, 2023

Mats-SX Jun 19, 2023

kedarghule Jun 19, 2023

Mats-SX Jun 20, 2023

Mats-SX Jun 19, 2023

kedarghule Jun 19, 2023

Mats-SX Jun 19, 2023

Mats-SX Jun 19, 2023

kedarghule Jun 19, 2023

kedarghule commented Jun 19, 2023

Mats-SX Jun 20, 2023

Mats-SX commented Jun 20, 2023

Added Centrality Algorithms Example Jupyter Notebook #386

Added Centrality Algorithms Example Jupyter Notebook #386

Conversation

kedarghule commented May 31, 2023

netlify bot commented May 31, 2023 • edited Loading

✅ Deploy Preview for neo4j-graph-data-science-client canceled.

kedarghule commented Jun 5, 2023

Mats-SX commented Jun 8, 2023

kedarghule commented Jun 9, 2023

Mats-SX left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kedarghule commented Jun 19, 2023

Choose a reason for hiding this comment

Mats-SX commented Jun 20, 2023

netlify bot commented May 31, 2023 •

edited

Loading