Define Bridge Node Architecture

[carver]

Although we aren't building this now, we're taking small steps toward it, with issues like #314

Here is the architecture that currently makes the most sense to me:

I'm imagining maybe 200 nodes, each with 5MB storage, and only connected to the history network. The Splitter will push data in using the portal_historyStore json-rpc endpoint, which will gossip normally, as if it had received the data on-chain. Then it will return back the list of ENRs that accepted the new data. This can help us monitor whether the data is making it outside of our self-hosted nodes and bootnodes.

#314 will be a scaled down, dumbed down version of this ^, maybe running only 4-8 nodes and doing the dumb thing of just pushing all data to all nodes. Maybe some data gets totally dropped on the floor, which is fine for alpha-quality. Also, the monitoring of recipient ENRs will probably not be included.

Potential Subtasks

This is a brainstorm of some ideas about how to accomplish the above architecture. This assumes that a number of things will already be taken care of by #314 -- so none of these are imminent

• trin
  • if launched with a default port, and 9000 is in-use, automatically cycle to a different port
  • add cli arg for data storage directory?
  • show data storage directory in info logs
  • add a new portal_historyGossip that returns the list of ENRs that the new history was gossiped to (this will make it take longer, as it must block during the gossip phase)
• run a dedicated eth1 node (geth?)
• in bridge-splitter first setup script:
  • auto-generate node IDs with even distribution (configurable by total number of target nodes)
  • create ramdisk for data directories (because ~200 threads trying to hit the disk simultaneously would be a mess)
• in bridge-splitter manager script:
  • before launch: copy data store into ramdisk
  • at launch
    • configure custom json-rpc IPC path with node id
    • configure custom discovery port (aiming for consistency to reduce ENR turnover)
    • configure custom data path to ramdisk
  • after shutdown: copy data from ramdisk back to hard disk (rationale vs using sqlite in-memory: want to save routing tables, ENR sequence number, etc in-between runs)
• in bridge-splitter daemon. mostly the same as Inject headers to alpha network from external source #314, but also:
  • connect to the dedicated eth1 node instead of 3rd-party service
  • generate content id from content key, before storing
  • use ^ content id to choose k nearest bridge nodes to run portal_historyStore on, for that piece of content. I'm guessing 0 < k < 6
  • save the list of ENRs that each piece of content was gossiped to, for later analysis (eg~ are we just gossiping to our own bootnodes, bridge nodes, etc)

[pipermerriam]

I don't understand the reasoning behind "tiny radius" nodes and I also want to clarify whether these are nodes that the bridge runs itself, are they independent nodes, or are they something fancier like a single piece of software managing multiple nodes concurrently on the network.

[carver]

I also want to clarify whether these are nodes that the bridge runs itself, are they independent nodes, or are they something fancier like a single piece of software managing multiple nodes concurrently on the network.

All the Trin Node # circles are nodes that the bridge runs on itself. They are lightweight enough that we can run a bunch on one machine. The nodes are stock trin clients, with configured radius.

Architecture plan:

• Run a bunch of nodes, with evenly spread IDs, in order to quickly gossip data to the right locations in the network.
• Grind private keys to get an even distribution of node IDs across the keyspace.
• Each node will have a high-resolution map of the nodes near its node ID in the network, so gossip will be trivial for it.

In the diagram, Portal Neighbor A (and B) are independent nodes in the routing table of Trin Node 1. So that's the "actual" seeding into the network.

I don't understand the reasoning behind "tiny radius" nodes

The rationale to tune the radius down includes:

1. if there are 200 nodes, even a 5MB cap uses 1GB of space
2. these nodes are not intended to seed/host data on the network, just to pipe into other nodes that can seed the data

[ogenev]

The general idea looks reasonable to me 👍 .

• Grind private keys to get an even distribution of node IDs across the keyspace.

I'm not sure if it is worth spreading those evenly across the keyspace (especially in a testing network) because in practice it is really hard to generate random NodeIDs which fall in a bucket higher than 16-17ish.

[carver]

• Grind private keys to get an even distribution of node IDs across the keyspace.

I'm not sure if it is worth spreading those evenly across the keyspace (especially in a testing network) because in practice it is really hard to generate random NodeIDs which fall in a bucket higher than 16-17ish.

Totally. "Evenly-spaced" is very loose here. If we want 256 nodes, then we could grind until we have node IDs that start with every byte from 0x00 to 0xFF, and not worry much about the rest of the node ID. (maybe a slight preference for the next byte to be closer to 0x80 than to 0x00 or 0xFF, but probably not worrying about it much)

[njgheorghita]

closing as our bridge architecture has evolved beyond this issue