Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Network message #883

Closed
wants to merge 11 commits into from
Closed

Network message #883

wants to merge 11 commits into from

Conversation

Robadob
Copy link
Member

@Robadob Robadob commented Jul 8, 2022

Some of the original ideas here.

Stage 1 - Static environment graph

  • Description
  • Host Set
  • Host Get
  • Host Misc (e.g. from file)
  • Host API element methods to accept N=0 (I may rework this, the solution is imperfect)
    • Host API SWIG elements
  • Agent Get
  • Agent Traverse
    • Edges Leaving
    • Edges Joining
  • Device Curve
  • RTC Curve
  • Submodel support
  • SWIG support (This is main reason CI currently failing)
  • C tests
  • Py tests
  • Refactor to remove nested class
  • Solve Agent Traverse seatbelts
  • Improve HostAPI auto sync (See comment below)
  • Make ID useful? (This adds an extra layer of indirection to lookups, will need to consider perf impact)
  • Remove indexable edge access? (The sort for CSR/CSC mixes up the order)
  • Docs
  • Add ADVANCED_API method to HostAPI to expose CUDA buffers/stream.
  • Do we want host traverse? (Not immediately)
  • (internal) Proper backing graph data structure/sort

Stage 2 - Messages for the graph

  • Everything (or just use bucket?)

@Robadob Robadob self-assigned this Jul 8, 2022
@Robadob Robadob linked an issue Jul 8, 2022 that may be closed by this pull request
@Robadob Robadob force-pushed the network-message branch from 1ce2109 to 278c97e Compare July 8, 2022 12:14
@Robadob
Copy link
Member Author

Robadob commented Jul 8, 2022

I don't really like how inside EnvironmentDirectedGraph::CUDABuffers, a reference to the map of agents is held, so that when buffers are resized curve can be updated in each agent function. Something about this feels wrong.

Alternate options being:

  • A central set of HostCurve objects is maintained in parallel which can be shared.
  • Everytime an agent function is loaded, it re-sets the graph ptrs in curve, although they won't change beyond init in most cases.

@Robadob

This comment was marked as outdated.

@Robadob

This comment was marked as outdated.

@Robadob

This comment was marked as outdated.

@Robadob

This comment was marked as outdated.

@Robadob Robadob force-pushed the network-message branch 2 times, most recently from bee9adf to c16f11b Compare December 1, 2022 12:05
@Robadob

This comment was marked as outdated.

@Robadob Robadob force-pushed the network-message branch 2 times, most recently from 72e20fe to df470e8 Compare January 3, 2023 14:30
@Robadob

This comment was marked as outdated.

@Robadob
Copy link
Member Author

Robadob commented Jan 4, 2023

All current TestEnvironmentDirectedGraph tests are now passing. This was solved by defining edge source-dest pairs in the same order they would be sorted into.

It's assumed subsequent tests will cover the traversal of graphs, so the actual sort will be tested implicitly.

However, we may wish to remove indexable edge access in future, as the sort will lead to confusion??

@Robadob

This comment was marked as outdated.

@Robadob

This comment was marked as outdated.

@Robadob
Copy link
Member Author

Robadob commented Jan 11, 2023

ID currently exists as an automatically added Vertex variable. But it's not used for anything? Whereas other libs/pete use ID inplace where we currently use index.

@Robadob Robadob force-pushed the network-message branch 2 times, most recently from 9e1bfb7 to 25d1f7b Compare January 18, 2023 16:16
@Robadob Robadob mentioned this pull request Jan 18, 2023
@Robadob
Copy link
Member Author

Robadob commented Jan 31, 2023

The HostAPI auto sync could be improved.

If the user calls

std::shared_ptr<flamegpu::detail::CUDAEnvironmentDirectedGraphBuffers> map = FLAMEGPU->environment.getDirectedGraph("map").getCUDABuffers();

The graph will be sync'd, prior to them accessing the CUDA buffers (ignoring the fact this doesn't mark the graph for rebuild). This however would require some proper internal tracking to ensure it persists to the end of the host fn.

@Robadob
Copy link
Member Author

Robadob commented Feb 27, 2023

I don't think it validates that edge src/dest are within range of valid vertex indices.

Robadob added 11 commits May 3, 2023 13:48
Currently it exposes no additional functionality.
Directed graphs can be defined on the Host and accessed and traversed on device (edges in/out of a given vertex).
Vertices and Edges are currently referred to by their index, although Vertices have a mandatory ID property.
The ID property is currently used, it would be possible to switch traversal to use IDs however it adds an extra layer of indirection to all accesses so has not currently been done.
Graphs can be exported/imported to a JSON format compatible with d3.js.
Tests added for all current functionality.
Also fixed a few GLM_ON build issues, likely a side effect of refactors prior to release.
HostAPI::getCUDAStream()

HostEnvironmentDirectedGraph::getCUDABuffers()
Not sure if this change will impact SEATBELTS=0 performance.
@Robadob Robadob force-pushed the network-message branch from 3eac0b9 to 1212131 Compare May 3, 2023 12:48
@Robadob Robadob mentioned this pull request Jul 14, 2023
13 tasks
@Robadob
Copy link
Member Author

Robadob commented Jul 14, 2023

Deprecated by #1089, to avoid breaking FRE dependency during development before the feature is merged into master.

@Robadob Robadob closed this Jul 14, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Feature: Graph/Network-based Communication
1 participant