Generate new network graphs #93

willcl-ark · 2023-09-07T11:46:10Z

Generate networkx.random_internet_as_graph-type graphs using CLI.

e.g.

$ warcli graph create n=100 --outfile=random_internet_as_graph_n100.graphml --random

willcl-ark · 2023-09-07T11:50:11Z

~~@pinheadmz if you review changed some cli commands:~~

~~warcli network start -> warcli network create <from-file [graph_file] | random>~~

willcl-ark · 2023-09-07T11:51:35Z

The weighted tags mean that there will likely be more nodes with recent version numbers.

We might want to add a more accurate distribution later, or allow users to pass their own distribution function into this

willcl-ark · 2023-09-09T00:50:29Z

@pinheadmz @josibake should we also allow node versionc constriants, "e.g >=0.21.1" or just leave as weighted random over all supported tags?

vercel · 2023-09-12T15:01:48Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
warnet	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Nov 2, 2023 1:26pm

pinheadmz · 2023-09-12T15:50:33Z

should we also allow node versionc constriants, "e.g >=0.21.1" or just leave as weighted random over all supported tags?

bitcoind should be free to make these decisions. I think once we have the p2p dns seeder (which can provide service flags) this will evolve on its own

I dont love removing start, would prefer it if create is a new command that once executed either:

also starts the network as if start had also been executed
save the graph file somewhere and tell the user its name so the user can manully call start

this latter option i think is preferable but then i also think well need a directory of graph files and a "list" method for them ?

pinheadmz · 2023-09-12T15:51:50Z

README.md

@@ -129,17 +129,17 @@ Once `warnet` is running it can be interacted with using the cli tool `warcli`.
 All `warcli` commands accept a `--network` option, which allows you to specify the warnet you want to control.
 This is set by default to `--network="warnet"` to simplify default operation.

-To start an example warnet, with your venv active and the server running, run the following command to use the default graph and network:
+To start a random warnet, with your venv active and the server running, run the following command to create 20 nodes where each edge has a 30% chance of being connected:


add this instead of replace the example graph method?

pinheadmz · 2023-09-12T15:52:19Z

warcli network start -> warcli network create <from-file [graph_file] | random>

a little confused. you didnt remove/replace the start command ?

willcl-ark · 2023-09-12T17:28:26Z

should we also allow node versionc constriants, "e.g >=0.21.1" or just leave as weighted random over all supported tags?

bitcoind should be free to make these decisions. I think once we have the p2p dns seeder (which can provide service flags) this will evolve on its own

Whilst the p2p dns seeder will certainly help in making connections, the bitcoind container cannot decide it's own binary version number it's going to run? This PR is about generating a graph file of randomised (or not) node:versions...

My question was about whether we should have warcli network create random take a min_version param (or similar), in the case that people might not want v16.3 nodes for example

I dont love removing start, would prefer it if create is a new command that once executed either:
* also starts the network as if `start` had also been executed

* save the graph file somewhere and tell the user its name so the user can manully call `start`
this latter option i think is preferable but then i also think well need a directory of graph files and a "list" method for them ?

Should we simply remove the grammar at this point, if we are not going to use it for everything? Currently we have:

  debug      Debug commands
   ├─   generate-compose  Generate the docker-compose file for a given...
   ├─   update-dns-seed   Update the dns seed database using a <graph_file> on...
  debug-log  Fetch the Bitcoin Core debug log from <node> in [network]
  help       Display help information for the given command.
  messages   Fetch messages sent between <node_a> and <node_b> in <network>
  network    Network commands
   ├─   down   Run 'docker-compose down on a warnet named <--network> (default:...
   ├─   start  Start a warnet with topology loaded from a <graph_file> into...
   ├─   up     Run 'docker-compose up' on a warnet named <--network> (default:...
  rpc        Call bitcoin-cli <method> <params> on <node> in <--network>
  scenarios  Scenario commands
   ├─   active  List running scenarios on <--network> (default=warnet) as...
   ├─   list    List available scenarios in the Warnet Test Framework
   ├─   run     Run <scenario> from the Warnet Test Framework on <--network>...
   ├─   stop    Stop scenario with <pid> from running on <--network>
  stop       Stop warnetd.

@josibake felt like network was un-intuative here and would prefer start and stop as top-level. I'd argue that create belongs in network, and you now think that start and stop belong in network but not create :P

I am very happy to go with the majority, but I think we should spec out exactly how we want this to operate so that we can build it just so!

My personal perference would be to retain the sub-commands (but use them in all cases), but happy to be out-voted on this one...

edit: perhaps on a re-read now, I see that you would be for having create (as a network sub-command?) but just have it generate the graph file? I could live with that!)

pinheadmz · 2023-09-12T17:47:06Z

ok yeah we got wires crossed.

Create a network with random bitcoind versions: great, got it now. yeah weighting is a cool idea here based on current user agent distribution. This is essentially an automatic warnet generation function and mine as well have this feature. Maybe it can be behind a switch where default is weighted random versions but --allversions=25.0 or something is a flag
grammar:
I DO LIKE:
warcli network create random1
warlci network start random1
warlci network stop random1
I think we could also add, example:

$ warcli network list
  [
    'random1',
    'example_20nodes`,
    'sensitive_relay'
  ]

josibake · 2023-09-12T17:52:16Z

Just poppin in to say ignore what I was saying earlier, been playing around with the CI more and got a feel for it. I do think more detailed help messages could alleviate any "un-intuitiveness"

willcl-ark · 2023-09-12T18:11:44Z

OK how about this push? New sub-command graph, which will do graph things. Like generate a random graph!

try

warcli graph generate 20 0.3 to see one spat out to the terminal. or add --file=/path/to/file to have it dumped to a file?

willcl-ark · 2023-09-12T18:13:09Z

* Create a network with random bitcoind versions: great, got it now. yeah weighting is a cool idea here based on current user agent distribution. This is essentially an automatic warnet generation function and mine as well have this feature. Maybe it can be behind a switch where default is weighted random versions but `--allversions=25.0` or something is a flag

Yeah I think we could add an --allversions flag very easily!

* grammar:
  I DO LIKE:
  `warcli network create random1`
  `warlci network start random1`
  `warlci network stop random1`
  I think we could also add, example:

See latest force-push, again, happy to be overruled on this, but perhaps graph sub-command makes even more sense (but takes more steps to fire up?)

pinheadmz · 2023-09-12T18:35:39Z

src/warnet/warnetd.py

+    graph = create_graph_with_probability(num_nodes, probability)
+    if file:
+        file = Path(file)
+        networkx.write_graphml(graph, file)


are there any options for this? the file works but is a little different than im used to seeing thus far which might make manual editing a bit more confusing e.g.

<key id="d1" for="node" attr.name="bitcoin_config" attr.type="string" /> <key id="d0" for="node" attr.name="version" attr.type="string" /> ... <data key="d0">24.1</data> <data key="d1" />

pinheadmz · 2023-09-12T18:44:12Z

src/warnet/cli/network.py

 @network.command()
 @click.argument("graph_file", default=EXAMPLE_GRAPH_FILE, type=click.Path())
 @click.option("--force", default=False, is_flag=True, type=bool)
 @click.option("--network", default="warnet", show_default=True)
-def start(
+def from_graph(


It might be time to get rid of this default graph, especially for a command called "from graph", i think it should require an argument.

If we do include a new directory src/graphs with files like sens_relay.graphml and all_released_versions.graphml etc they are more than just examples they are like standards. Would it be insane to call them "examples" and then have commands
warcli network list_examples and
warcli network from_example sens_relay ?
This would be IN ADDITION to warcli network from_graph <whatever> which would explicitly require an external file provided by the user

pinheadmz · 2023-09-12T18:46:28Z

src/warnet/cli/network.py

    graph_file: Path = EXAMPLE_GRAPH_FILE, force: bool = False, network: str = "warnet"
 ):
    """
-    Start a warnet with topology loaded from a <graph_file> into <--network> (default: "warnet")
+    Create a warnet with topology loaded from a <graph_file> into <--network> (default: "warnet")


might need to drop the default network as well. for example just now i ran a warnet then closed it, then tried to start a new warnet from a new graph and got this:

Config dir /Users/matthewzipkin/.warnet/warnet/warnet already exists, not overwriting existing warnet without --force

Maybe the dir could be something like...

~/.warnet/networks/<graphml file name?>

pinheadmz · 2023-09-12T19:20:16Z

src/warnet/utils.py

@@ -31,6 +31,8 @@
    "0.16.3",
    "0.15.2",
 ]
+WEIGHTED_TAGS = [tag for index, tag in enumerate(reversed(SUPPORTED_TAGS)) for _ in range(index + 1)]


...or you could just list the versions in SUPPORTED_TAGS in order of weight?

pinheadmz · 2023-09-12T19:24:06Z

src/warnet/warnetd.py

-        except Exception as e:
-            logger.error(f"Exception {e}")
+@jsonrpc.method()
+def update_dns_seeder(graph_file: str, network: str = "warnet") -> str:


its a bit tricky to review moved code, did this function change?

willcl-ark · 2023-10-31T12:33:00Z

@pinheadmz updated this for you

delete random AS graphs of size `n` using either default tag (latest) or random tags from SUPPORTED_TAGS

pinheadmz · 2023-10-31T18:34:50Z

need to add lxml to requirements

pinheadmz · 2023-10-31T18:35:23Z

src/warnet/cli/graph.py

+
+
+@graph.command()
+@click.argument("params", nargs=-1, type=str)


lets just require --n=<integer> right? there's no other params

Yep agree.

Although... if we ever extend it to other graph types which do have more params, I think this is useful to leave in.

Personally, I feel like it's not much code, which does no harm currently, and I'd prefer to leave it as it's written up now for the future?

Easily persuaded that's not the best course of action here though.

it's just weird to me to suddenly have an arg without the --. And also, I think filename should either be required, or there should be a flag to pipe the output right back in to network start. Would anyone need a 100 node graph file printed to the console?

pinheadmz · 2023-10-31T18:36:12Z

src/warnet/utils.py

+
+    logger.debug(f"Parsed params: {kwargs}")
+
+    try:


IMO most of the try/catch you have in this function should be removed so the error gets thrown back to the RPC caller. Otherwise you get a nice log message but all the CLI shows is "server error"

Hmmm, this is an attempt. I actually don't recall why now I needed it at all. IIRC during some of my testing some graphs were breaking without it. But could have been an artefact of the nargs optional cli args thingy (i.e. dodgy input coming from terminal?)

Perhaps if we want to actually return exceptions back to the user via CLI (do we though!?) we should have a refactor PR which moves all try/except into server.py? Each server function wraps function calls with them and can then either returns success, or any bubbled up exception (can also remove our dodgy decorator at the same time?)

just execute warcli graph create with no arguments.

User gets this:
Error generating graph: Server error

Logs get all this. Which starts off making sense (missing an arg) but ends up in a really weird place...

2023-11-01 17:42:21,811 - utils - ERROR - Failed to create graph: random_internet_as_graph() missing 1 required positional argument: 'n' 2023-11-01 17:42:21,846 - warnet.server - ERROR - unexpected error Traceback (most recent call last): File "/opt/homebrew/lib/python3.11/site-packages/flask_jsonrpc/site.py", line 135, in handle_dispatch_except return self.dispatch(req_json) ^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/homebrew/lib/python3.11/site-packages/flask_jsonrpc/site.py", line 184, in dispatch resp_view = current_app.ensure_sync(view_func)(**params) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/homebrew/lib/python3.11/site-packages/typeguard/__init__.py", line 1033, in wrapper retval = func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/Users/matthewzipkin/Desktop/work/warnet/src/warnet/server.py", line 291, in graph_generate nx.write_graphml(graph, bio) File "/opt/homebrew/lib/python3.11/site-packages/networkx/utils/decorators.py", line 766, in func return argmap._lazy_compile(__wrapper)(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "<class 'networkx.utils.decorators.argmap'> compilation 9", line 5, in argmap_write_graphml_lxml_5 import itertools ^^^^^^^^ File "/opt/homebrew/lib/python3.11/site-packages/networkx/readwrite/graphml.py", line 176, in write_graphml_lxml writer = GraphMLWriterLxml( ^^^^^^^^^^^^^^^^^^ File "/opt/homebrew/lib/python3.11/site-packages/networkx/readwrite/graphml.py", line 745, in __init__ self.add_graph_element(graph) File "/opt/homebrew/lib/python3.11/site-packages/networkx/readwrite/graphml.py", line 751, in add_graph_element if G.is_directed(): ^^^^^^^^^^^^^ AttributeError: 'str' object has no attribute 'is_directed'

But I hear ya, we might need an RPC overhaul to clean up some try's. I think we are inconsistent with it.

willcl-ark · 2023-10-31T20:14:11Z

need to add lxml to requirements

₿ pip uninstall warnet; and pip freeze | xargs pip uninstall; and pip install -e .
...
₿ warcli graph create n=100 --outfile=random_internet_as_graph_n100.graphml --random
Generated graph written to file: random_internet_as_graph_n100.graphml

Works for me without that (tm) 🤷🏼‍♂️

Graph CI won't create a network and errors heavily on teardown. Add a flag to TestBase which can bypass this. Also add a thread to read server stdout into test logs. And tear this thread down in the correct order to not block.

willcl-ark · 2023-10-31T22:23:13Z

OK ready FR FR

pinheadmz · 2023-11-01T21:53:26Z

hm yeah i didnt get the lxml error this time... ooooohhhkkkaaayyyyyy

test/test_graph.py

pinheadmz · 2023-11-02T17:25:43Z

YES.

willcl-ark force-pushed the generate_graph branch from 7ef1868 to a5be661 Compare September 7, 2023 11:48

willcl-ark force-pushed the generate_graph branch from a5be661 to 7d22709 Compare September 8, 2023 21:20

willcl-ark force-pushed the generate_graph branch 2 times, most recently from 066ec46 to 3e4b516 Compare September 11, 2023 19:25

willcl-ark requested review from pinheadmz and josibake September 11, 2023 19:25

willcl-ark added this to the Thursday milestone Sep 12, 2023

willcl-ark force-pushed the generate_graph branch from 3e4b516 to 19a49c3 Compare September 12, 2023 15:01

vercel bot temporarily deployed to Preview September 12, 2023 15:02 Inactive

pinheadmz reviewed Sep 12, 2023

View reviewed changes

willcl-ark force-pushed the generate_graph branch from 19a49c3 to 0140d56 Compare September 12, 2023 18:10

vercel bot temporarily deployed to Preview September 12, 2023 18:10 Inactive

willcl-ark force-pushed the generate_graph branch from 0140d56 to f14f15d Compare September 12, 2023 18:14

vercel bot temporarily deployed to Preview September 12, 2023 18:14 Inactive

pinheadmz reviewed Sep 12, 2023

View reviewed changes

willcl-ark force-pushed the generate_graph branch from f14f15d to 6bdba96 Compare September 13, 2023 12:32

vercel bot temporarily deployed to Preview September 13, 2023 12:32 Inactive

vercel bot temporarily deployed to Preview September 13, 2023 13:49 Inactive

willcl-ark force-pushed the generate_graph branch from a03aa42 to bbded03 Compare September 13, 2023 13:54

vercel bot temporarily deployed to Preview October 25, 2023 08:01 Inactive

willcl-ark force-pushed the generate_graph branch from 9d8998a to 2176876 Compare October 25, 2023 11:38

vercel bot temporarily deployed to Preview October 25, 2023 11:39 Inactive

willcl-ark force-pushed the generate_graph branch from 2176876 to b2126dd Compare October 31, 2023 12:28

vercel bot temporarily deployed to Preview October 31, 2023 12:29 Inactive

willcl-ark force-pushed the generate_graph branch from b2126dd to 5e9c202 Compare October 31, 2023 12:31

willcl-ark changed the title ~~Lookup and generate graphs using networkx generators~~ Generate new network graphs Oct 31, 2023

vercel bot temporarily deployed to Preview October 31, 2023 12:32 Inactive

graph: add generator

5ead3a2

delete random AS graphs of size `n` using either default tag (latest) or random tags from SUPPORTED_TAGS

willcl-ark force-pushed the generate_graph branch from 5e9c202 to 5ead3a2 Compare October 31, 2023 12:34

vercel bot temporarily deployed to Preview October 31, 2023 12:35 Inactive

pinheadmz reviewed Oct 31, 2023

View reviewed changes

test: only teardown networks when created

bea13f7

Graph CI won't create a network and errors heavily on teardown. Add a flag to TestBase which can bypass this. Also add a thread to read server stdout into test logs. And tear this thread down in the correct order to not block.

vercel bot temporarily deployed to Preview October 31, 2023 22:12 Inactive

willcl-ark force-pushed the generate_graph branch from c68d1a0 to 819143e Compare October 31, 2023 22:14

vercel bot temporarily deployed to Preview October 31, 2023 22:15 Inactive

pinheadmz reviewed Nov 1, 2023

View reviewed changes

test/test_graph.py Outdated Show resolved Hide resolved

pinheadmz reviewed Nov 1, 2023

View reviewed changes

test/test_graph.py Outdated Show resolved Hide resolved

graph: add simple test to check graph can be generated successfully

d3c63c6

willcl-ark force-pushed the generate_graph branch from 819143e to d3c63c6 Compare November 2, 2023 13:25

vercel bot temporarily deployed to Preview November 2, 2023 13:26 Inactive

willcl-ark mentioned this pull request Nov 2, 2023

Threaded logging #164

Closed

pinheadmz merged commit 0a006c0 into bitcoin-dev-project:main Nov 2, 2023
6 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Generate new network graphs #93

Generate new network graphs #93

willcl-ark commented Sep 7, 2023 •

edited

Loading

willcl-ark commented Sep 7, 2023 •

edited

Loading

willcl-ark commented Sep 7, 2023

willcl-ark commented Sep 9, 2023

vercel bot commented Sep 12, 2023 •

edited

Loading

pinheadmz commented Sep 12, 2023

pinheadmz Sep 12, 2023

pinheadmz commented Sep 12, 2023

willcl-ark commented Sep 12, 2023 •

edited

Loading

pinheadmz commented Sep 12, 2023

josibake commented Sep 12, 2023

willcl-ark commented Sep 12, 2023

willcl-ark commented Sep 12, 2023

pinheadmz Sep 12, 2023

pinheadmz Sep 12, 2023

pinheadmz Sep 12, 2023

pinheadmz Sep 12, 2023

pinheadmz Sep 12, 2023

willcl-ark commented Oct 31, 2023

pinheadmz commented Oct 31, 2023 •

edited

Loading

pinheadmz Oct 31, 2023

willcl-ark Oct 31, 2023

pinheadmz Nov 1, 2023

pinheadmz Oct 31, 2023

willcl-ark Oct 31, 2023

pinheadmz Nov 1, 2023

pinheadmz Nov 1, 2023

willcl-ark commented Oct 31, 2023

willcl-ark commented Oct 31, 2023

pinheadmz commented Nov 1, 2023

pinheadmz commented Nov 2, 2023



		@graph.command()
		@click.argument("params", nargs=-1, type=str)

Generate new network graphs #93

Generate new network graphs #93

Conversation

willcl-ark commented Sep 7, 2023 • edited Loading

willcl-ark commented Sep 7, 2023 • edited Loading

willcl-ark commented Sep 7, 2023

willcl-ark commented Sep 9, 2023

vercel bot commented Sep 12, 2023 • edited Loading

pinheadmz commented Sep 12, 2023

Choose a reason for hiding this comment

pinheadmz commented Sep 12, 2023

willcl-ark commented Sep 12, 2023 • edited Loading

pinheadmz commented Sep 12, 2023

josibake commented Sep 12, 2023

willcl-ark commented Sep 12, 2023

willcl-ark commented Sep 12, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

willcl-ark commented Oct 31, 2023

pinheadmz commented Oct 31, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

willcl-ark commented Oct 31, 2023

willcl-ark commented Oct 31, 2023

pinheadmz commented Nov 1, 2023

pinheadmz commented Nov 2, 2023

willcl-ark commented Sep 7, 2023 •

edited

Loading

willcl-ark commented Sep 7, 2023 •

edited

Loading

vercel bot commented Sep 12, 2023 •

edited

Loading

willcl-ark commented Sep 12, 2023 •

edited

Loading

pinheadmz commented Oct 31, 2023 •

edited

Loading