Per world access lists and utils

[ElliottjPierce]

Objective

One of the big ones is parallelizing extraction phases for sub apps. It's a big bottleneck that currently exists for all sub-app systems since they can't be parallelized. It's not that big of a deal when it's just a render world, but with a physics world and others, too, this could really help performance.
[12:52]Eagster: Another one is for multiplayer architecture. It is often helpful to fully separate the client side from the server side into separate worlds so there aren't any side effects from roll-back, etc. For example, I have a front-end world and a back-end world, where the back end can be optionally synced to another world. This makes it easier (for me) to have a clean separation from front to back.
[12:54]Eagster: Also, it could be useful to have as a utility, and the PR can be made to have practically no performance change. What inspired the PR originally was the utilities in the render world to make it easier to communicate between the main and render world. I wanted to generalize that to be used for other sub-apps and use cases too.

• OP, from Discord

Fixes #16933

Note

This PR proposes a direct solution to #16933. The issue is technically still in design phase, and while there may be better solutions, I wanted to get the conversation going. This is the least complex solution I could think of.

Since this PR is sort of a base-line implementation to create some discussion, I haven't done mush polish. If the community thinks this sort of implementation is worth pursuing, I would want to put more time into polishing it up with more documentation, tests, and maybe an example.

Solution

I made four main changes to facilitate this. These could be split into smaller PRs later.

First, I added universal access lists. See crates/bevy_ecs/src/query/access.rs UniversalAccess. This stores multiple Accesss, one per relevant world. This lets access be tracked over multiple worlds. I used a SmallVec instead of a HashMap for pairing worlds to accesses because I suspect it will perform better, but this could always be changed.

Second, I converted SystemMeta and FilteredAccessSet to use UniversalAccess. I also updated the relevant references. This formed the backbone for the feature since it allows systems to be parallelized across multiple world accesses.

Third, I created an abstraction in crates/bevy_ecs/src/world/links.rs that makes it easy to nest or reference worlds in each other. This is what makes the feature useful.

Fourth, I added SystemParam::update_meta. I needed a way to notify nested parameters of archetype changes. This needed to be within the scope of System::update_archetype_component_access, otherwise SystemMeta may be changed in a way that conflicts with the scheduler or the nested param my not have been fully updated. Using this new interface, I was able to fully connect nested params.

As a note, the Link abstraction uses some sneaky privacy rules to safely access a resource in a way that is traditionally unsafe. Although I don't see any issues with it, I encourage it to be very carefully reviewed in particular.

Testing

I have done minimal testing myself. If the community likes this sort of implementation, I would want more tests for sure. If anyone has any requests for more tests, I'd be happy to add them.

The current CI passes for me. I'm on a M2 Mac.

As far as I can tell, though not polished, the current state of the PR is safe, so please try to break it.

---

Showcase

The following is an excerpt of the test I wrote for a proof on concept. See crates/bevy_ecs/src/world/links.rs for full context and refer to #16933 for the long-term vision.

let mut main = World::new();
main.insert_resource(MyRes);
main.link(NestedWorld(World::new()));
main.peek_link::<NestedWorld>()
    .unwrap()
    .deref_mut()
    .0
    .insert_resource(MyRes);

fn my_system(
    main_res: ResMut<MyRes>,
    main_query: Query<&mut MyComponent>,
    nested_res: Linked<NestedWorld, ResMut<MyRes>>,
    nested_query: Linked<NestedWorld, Query<&'static mut MyComponent>>,
) {
    // would do stuff here
}

main.run_system_once(my_system).unwrap();

let mut schedule = Schedule::default();
schedule.add_systems(my_system);
schedule.run(&mut main);

This kind of interface could make sub app extraction much much easier to work with (and likely faster from parallelization). In the long run, it may also enable multiple sub apps' extractions to be run in parallel. This would be huge for performance of projects with multiple big sub apps, like those using avian physics.

Migration Guide

Most notably, SystemMeta's access data types have changed. If the community is in favor of this kind of implementation, I can work on a more complete migration guide.

Final Note:

This is my first time contributing to bevy or open source in general, so I am extremely open to feedback. I have done my best to follow the contributing guidelines, but let me know if anything needs to be done differently. I'm very excited about this feature, so any feedback or requests are very welcome.

[github-actions]

Welcome, new contributor!

Please make sure you've read our contributing guide and we look forward to reviewing your pull request shortly ✨

[ElliottjPierce]

@PPakalns I would appreciate your input here too since you expressed interest in #16933. I want to walk away from this PR (merged or not) with a solid plan for implementing the feature.

[PPakalns]

@PPakalns I would appreciate your input here too since you expressed interest in #16933. I want to walk away from this PR (merged or not) with a solid plan for implementing the feature.

System query and resource access api looks great.

It would be nice if linked world is SubApp instead of World. This would allow to apply different set of plugins for SubApp/Subworld. And subworld schedule could be executed somewhere in main world schedule manually by taking Full mutable access to Subapp/Subworld or maybe even executed in parallel in far future.

Overall great work and looking forward to this functionality :)

[PPakalns]

And looks like there is still no answer to data extraction question #8143 from subapp and this API would be greatly appreciated.

[ElliottjPierce]

It would be nice if linked world is SubApp instead of World. This would allow to apply different set of plugins for SubApp/Subworld. And subworld schedule could be executed somewhere in main world schedule manually by taking Full mutable access to Subapp/Subworld or maybe even executed in parallel in far future.

I agree with that in the long run. Or even, make some sort of WorldPlugin since most Plugins only use the world anyway and don't set an extraction function or anything. I also want to be able to add these linked worlds at runtime though. I don't think you can create a new subapp for the main app once it starts.

Its also worth noting that you can, very carefully, use a raw pointer to a world as a WorldLink. The linked world doesn't have to live in the main world. You could potentially insert a pointer to a subapp's world into the main world as a WorldLink from the subapp's extraction phase. You would also need to create some form of synchronization or maybe a custom schedule to keep the access safe, but it should be possible to use this interface with subapps.

[PPakalns]

I also want to be able to add these linked worlds at runtime though. I don't think you can create a new subapp for the main app once it starts.

Agree, It is important that linked SubApp/World can be recreated/removed/replaced during runtime.

Some questions about planned interface that I thought of.

1. Is it possible to get exclusive access to world. Like fn system(Linked<NestedWorld, &mut World>) {}
2. Is it possible to create Optional Linked world query (If linked world currently is not linked). Like fn system(Option<Linked<NestedWorld, ...>>) And to group multiple resource, query access together to simplify matching of the Option: fn system(Option<Linked<NestedWorld, (Res<R1>, Query<R2>)>>) {}
3. Is it possible to take whole SubApp if I would like to execute schedule from the linked Subapp (run SubApp systems). Probably could be done using unsafe and taking exclusive access to linked World and then executing SubApp. But during system execution main App and its subworlds are not accessible if I remember correctly.

[ElliottjPierce]

1. Is it possible to get exclusive access to world. Like fn system(Linked<NestedWorld, &mut World>) {}

Yes, but you need to do it through exclusive access to the main world. For example:

fn nested_exclusive(main: &mut World) {
     let nested = main..peek_link::<NestedWorld>().unwrap()
}

&mut World doesn't count as a normal SystemParam, so it can't be put into Linked. As it happens, if you could do this, it could cause race conditions. See my comments on the Link resource itself for why we can't allow this, yet. The root reason is that we need to provide new_archetype to linked params. currently, there is no way (that I could think of) to synchronize this call in the scheduler. Maybe we could find a better way to do this, but for now, I think using exclusive main access is a reasonable trade off.

2. Is it possible to create Optional Linked world query (If linked world currently is not linked). Like fn system(Option<Linked<NestedWorld, ...>>) And to group multiple resource, query access together to simplify matching of the Option: fn system(Option<Linked<NestedWorld, (Res<R1>, Query<R2>)>>) {}

Absolutely. This should be pretty strait forward to add in the future, but for now, I'm trying to keep this already massive PR as slim as possible. If this is merged, you can expect a number of follow up PRs, adding more utilities/interfaces to make this even more useful.

3. Is it possible to take whole SubApp if I would like to execute schedule from the linked Subapp (run SubApp systems). Probably could be done using unsafe and taking exclusive access to linked World and then executing SubApp. But during system execution main App and its subworlds are not accessible if I remember correctly.

I'm not sure if I am 100% on what you're asking here, but here is something you could do:

struct SubAppLink(*mut World);

impl WorldLink for SubAppLink {
    // insert unsafe pointer magic here
}

fn sub_app_extraction(sub_world: &mut World, main_world: &mut World) {
    main_world.link(SubAppLink(sub_world));
    main_world.run_schedule(MyCustomSchedule);
    main_world.unlink::<SubAppLink>();
}

fn main_system_for_custom_schedule(from_sub_app: Linked<SubAppLink, AnyParamHere>) {
    // whatever work you need on the main world but with reference to the sub app.
    // You could even make this function exclusive and run more schedules on the sub app here. Endless possibilities!
}

I cut some corners, but that's the idea. If you want more free-flowing communication with a custom synchronization system, the Linked interface won't be enough. The per-world-access lists will still be foundational to any interface that does that though.

[ElliottjPierce]

@BenjaminBrienen Thanks! Merging should be good now.

This has gotten positive feedback in both discord and github, but if there is anything you or anyone else wants changed, let me know.

[alice-i-cecile]

Alright, weighing in in-thread as SME-ECS. Sorry for leaving you hanging, and thanks a ton for providing the additional context.

From a technical perspective, this seems well-executed, and I definitely understand why you might want to move in this direction: multiple worlds are useful, and being able to multithread your scheduling across them is a really cool idea.

That said, I'm reluctant to move forward with this right now. As @wrongshoe said on Discord.

This has been discussed a couple times on discord before and the guess was that the extra overhead was probably not worth it since the executor can already be bottlenecked on spawning tasks currently. Having to check all the archetype-component conflicts in 2 worlds would potentially double the overhead.

As it currently stands, increasing multi-threading will likely worsen Bevy's performance. @NthTensor has a secret project to reduce that overhead that will likely change that calculus, and I have long-standing hopes of trying to precompute parallelism strategies through schedules to batch systems, so I don't think that's an unchangeable truth. Just "probably not worth the overhead and complexity right now".

More broadly, there's increasing appetite for real multi-world support in Bevy. The existing SubApp architecture is IMO very unclear, with muddy semantics, poor tools to coordinate across worlds (either in terms of threading or simply sending data between worlds), and no guidance on when / if this should be used.

Providing more tools to support using multiple worlds without a clear design in this space is a recipe for maintainer headaches (whee tech debt) and user frustration. There's no blessed mechanism or documentation for even the simplest of operations (sending data between worlds), and while you can hack around the limitations, intermediate users will struggle terribly and the ecosytem will fragment as advanced users write their own hacks.

Tackling tech debt in this area (both the Decoupled Rendering and the Turtles All the Way Down working groups are doing this) should come first, but after that I would like to spin up a true Multiworld working group, where a design in this vein (cross-world multithreaded scheduling) can be discussed in earnest.

[ElliottjPierce]

Wow. Great points. That's why I asked for an expert!

You're absolutely right that this should at the very least wait until precomputed schedules. I didn't even know that was possible or being discussed, but now I'm excited.

I agree with your opinion on sub-apps too. I have also been thinking about more free-flowing multi-world coordination beyond sub-apps, so I'm glad to hear I'm not the only one.

My end goal with this was to get multiple worlds communicating in one app, and it sounds like that is in Bevy's future, right after bsn, editors, and half a dozen way more important stuff.

Summarizing the discussion as I understand it:

• Bevy as a whole wants some version of this API in the future.
• The current scheduling system can't reasonably support multi-world synchronization performantly enough to be worth it.
• Even if it could, a more elegant solution may be possible in the future that would generalize sub-apps.

I will look forward to the multi-world working group and what they put together. In the meantime, I'm going to let this go cold, and come back to my multi-world project later. Feel free to close the PR when a better solution is proposed down the road.

One last thing in case it interests anyone: In the per-world access lists, the list stores a small vec of normal (one world) access lists. If we restrict that vec's inline capacity to 1, I'm inclined to think there wouldn't really be any additional scheduling overhead since it still only checks one world. That would allow a temporary fix that might provide a more convinient alternative to the hack arounds you mentioned. That said, given the other points you made, I'm not even convinced myself that it would even be worth it. I'll leave it up to your judgement.