Implement new custom ECS (WIP) #404

caelunshun · 2021-03-13T05:09:30Z

Status

Ready
Development
Hold

Description

Implements #402. Work in progress.

Progress

Spawn and despawn entities
Random component access
Static and dynamic queries
Handle Drop components
Dynamic borrow checking and mutable component access
Pass all hecs tests with Miri
Move system executor to new ECS
Switch to the new ECS in Feather (should be trivial; the API is almost the same as hecs)
Integrate with quill

I'm going to hold off on the new event handling system and the bundles discussed in #402 for another PR.

Defman

Looks good, some minor questions.

Could we perhaps use FnvMap for component storage since it is insanely fast. I know we are not using an ArchType based system, but my example is based on it.

struct World {
  // Finding the `ArchTypeId` for a given set of components, 
  // used when spawning a new entity or adding a component to an existing entity.
  // Note btree maps and sets might not be ideal, since they are kinda slow.
  arch_type_sets: BTreeMap<BTreeSet<ComponentId>, ArchTypeId>,
  // When quering we and all components bitset together, 
  // which yields indices into `arch_types`.
  components: FnvHashMap<ComponentId, BitSet>,
  arch_types: Vec<ArchType>,
  // Mapping from entity id to memory location.
  entities: FnvHashMap<EntityId, (ArchTypeId, ArchTypeOffset)>,
}

struct ArchType {
  components: FnvHashMap<ComponentId, MemorySpace>,
}

Querying ~

For each component get the bitset from world.components
And all the bit sets together
Each remaining true ArchTypeId in the bitset contains all the queried components.
Get each component from each archtypes doring the iteration.

Insertion ~

Create set of all component ids
Try getting a archtype id from world.arch_type_sets, if it does not exist push a new archtype and insert the id.
Insert the components at the end of arch_type.components
Create a new entity id by inserting (arch_type_id, offset)

Updating entity

Swap remove from the archtype
push to new archtype
Fix entity id ptrs (this require a reverse lookup table, which I did not add)

Defman · 2021-03-13T20:31:03Z

quill/ecs/src/component.rs

+        Self(ComponentTypeIdInner::Rust(TypeId::of::<T>()))
+    }
+
+    pub fn opaque(id: u64) -> Self {


Maybe 32 bits would be enough?

There's no harm in using a u64 so I disagree, there is only benefit.

Defman · 2021-03-13T20:32:00Z

quill/ecs/src/component.rs

+
+#[derive(Copy, Clone, Debug, PartialEq, Eq, Hash, PartialOrd, Ord)]
+enum ComponentTypeIdInner {
+    Rust(TypeId),


Maybe just use Opaque and use a FnvHashMap to map from type id to opaque id.

How would that be better?

We have to chose a true id at one point, I suggest we chose Opaque as the true id and map TypeId to the true id(Opaque). This indirection have happen at some point.

To my knowledge.

What do you mean by "true id" and why is it needed?

"true id" one universal identifier for components, no distinction between local ie Any::TypeId or remote ids.

And why is it needed?

quill/ecs/src/storage/blob_vec.rs

Defman · 2021-03-13T20:41:47Z

quill/ecs/src/storage/sparse_set.rs

+use super::blob_vec::BlobVec;
+
+/// Stores components in a sparse set.
+pub struct SparseSetStorage {


FnvHashMap is insanely fast, maybe it could be used in stead of SparseSetStorage?

Might require some architectural changes, since for each entity you would lookup its components by their id.

I'm using sparse sets instead of hash tables because we can optimize query iteration by e.g. packing all entities matching a given archetype into the front of the array. See https://skypjack.github.io/2019-03-07-ecs-baf-part-2/

Wouldn't using a HashMap defeat part of the purpose of ECS by getting rid of linear memory reads?

Sparse set already gets rid of 100% linear memory reads. Also FnvHashMap is not just fast, its insanely fast... it even beats bitsets... its crazy.

I mean the memory reads with sparse sets are still linear, just with gaps.
Using a HashMap will make memory reads completely unpredictable.

Hmm now I'm wondering where I saw next_power_of_two being used. RawVec uses the following algorithm

let required_cap = used_cap + needed_extra_cap; let double_cap = self.cap * 2; let new_cap = cmp::max(required_cap, double_cap);

Using next_power_of_two ~

let required_cap = used_cap + needed_extra_cap; let new_cap = required_cap.next_power_of_two();

These two code snippets produce different behaviour. Also you are in the wrong comment thread xD

Everything just seems to be going wrong for me at the moment 😅

Yea, the behavior is slightly different but both ensures push are O(n) amortized. The implementation that does not use next_power_two will double the capacity unless the total required cap is greater than the doubled size. Where the next_power_of_two will double and if the capacity is greater than cap * 2 it will be cap * 2 * 2 it will repeat this pattern until double cap is equal or greater than required cap.

quill/ecs/src/entity.rs

Changes BlobVec to BlobArray; it now has a fixed size. ComponentVec now provides the growable component vector, with the additional guarantee that items are stable in memory. Item drop functions are not yet called, so this has memory leaks. However, the code is sound and passes Miri.

Allocating with size 0 is UB. This uses a dangling pointer when the component size is 0. (Vec<T> does the same for zero-sized T.)

Defman · 2021-03-15T15:54:09Z

quill/ecs/src/ecs.rs

+    /// is overriden.
+    ///
+    /// Time complexity: O(1)
+    pub fn insert<T: Component>(


Maybe return the old component?

quill/ecs/src/entity_builder.rs

This splits query lifetimes in two: 'w is the lifetime of the ECS and 'q is the lifetime of the query. Additionally, QueryDriver::iter now returns a struct instead of an impl Iterator. These changes allow GATs to be removed from the QueryDriver implementation.

It took some wrestling with lifetimes, but static queries now work. They have roughly the same interface as `hecs`.

caelunshun · 2021-03-16T23:05:44Z

I updated the PR with a checklist of remaining work.

There was one false positive never_loop lint.

We previously uses World to store blocks and Ecs to store entities. Now, Level contains blocks and World contains entities. This change is motivated by most Rust ECS crates using World as their entity store. Also, Level is the terminology used internally in Minecraft from what I know.

Miri is slow. Spawning 10,000 entities takes several minutes.

Fixes leaked memory in components with allocations. All tests now pass with Miri.:

Amber came up with this on Discord. 'It's the Feather's backbone.'

Limitation: T must be Copy for soundness, since components are stored unaligned.

Plugins are temporarily disabled until we port Quill to the new ECS.

Defman · 2021-09-06T06:37:16Z

BUMP

Defman · 2021-09-16T20:16:06Z

Bump

quill: Initial commit for ECS

84a0226

caelunshun force-pushed the quill-ecs branch from 940ca7a to 84a0226 Compare March 13, 2021 05:09

caelunshun marked this pull request as draft March 13, 2021 05:10

Defman reviewed Mar 13, 2021

View reviewed changes

caelunshun added 4 commits March 14, 2021 19:17

ecs: Spawn entities, access components

83fc421

ecs: Add support to spawn with a tuple of components, despawn

c1193b2

ecs: Fix undefined behavior with zero-sized components

7e5a97b

Allocating with size 0 is UB. This uses a dangling pointer when the component size is 0. (Vec<T> does the same for zero-sized T.)

Defman reviewed Mar 15, 2021

View reviewed changes

caelunshun added 4 commits March 15, 2021 20:19

ecs: Support removing components, start on borrow flags

f3ee898

ecs: New model for plugin components, implement test query code

52b8b63

ecs: Working implementation of static queries

f5ec9e0

It took some wrestling with lifetimes, but static queries now work. They have roughly the same interface as `hecs`.

caelunshun added 3 commits March 16, 2021 17:14

ecs: Fix panic when query contains unknown components

1ec4e78

ecs: Resolve Clippy lints

686b35b

There was one false positive never_loop lint.

caelunshun force-pushed the quill-ecs branch from 3b9115f to 7c66425 Compare March 17, 2021 00:17

caelunshun added 10 commits March 17, 2021 16:15

ecs: Implement dynamic borrow checking

9b7138e

ecs: Ignore expensive tests when running under Miri

0f6c26b

Miri is slow. Spawning 10,000 entities takes several minutes.

ecs: Invoke component drop functions

0478b7f

Fixes leaked memory in components with allocations. All tests now pass with Miri.:

ecs: Fix memory leaks when EntityBuilder is dropped

f4a87bc

ecs: Rename to vane

c29e05f

Amber came up with this on Discord. 'It's the Feather's backbone.'

ecs: Port system executor and event system to vane.

7abf303

ecs: Add EntityBuilder::get()

94c5a40

Limitation: T must be Copy for soundness, since components are stored unaligned.

ecs: Add EntityRef

e4ce5a2

common, server: Port to vane.

e3a9ebf

Plugins are temporarily disabled until we port Quill to the new ECS.

Delete the old feather-ecs crate, which wrapped hecs.

5b61e10

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement new custom ECS (WIP) #404

Implement new custom ECS (WIP) #404

caelunshun commented Mar 13, 2021 •

edited

Loading

Defman left a comment •

edited

Loading

Defman Mar 13, 2021

ambeeeeee Mar 13, 2021 •

edited

Loading

Defman Mar 13, 2021

Schuwi Mar 13, 2021

Defman Mar 13, 2021 •

edited

Loading

Schuwi Mar 13, 2021

Defman Mar 15, 2021

Schuwi Mar 16, 2021

Defman Mar 13, 2021

caelunshun Mar 13, 2021

Schuwi Mar 13, 2021

Defman Mar 13, 2021

Schuwi Mar 13, 2021

Defman Mar 14, 2021 •

edited

Loading

Schuwi Mar 14, 2021

Defman Mar 14, 2021 •

edited

Loading

Defman Mar 15, 2021

caelunshun commented Mar 16, 2021

Defman commented Sep 6, 2021

Defman commented Sep 16, 2021

Implement new custom ECS (WIP) #404

Are you sure you want to change the base?

Implement new custom ECS (WIP) #404

Conversation

caelunshun commented Mar 13, 2021 • edited Loading

Status

Description

Progress

Defman left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ambeeeeee Mar 13, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Defman Mar 13, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Defman Mar 14, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Defman Mar 14, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

caelunshun commented Mar 16, 2021

Defman commented Sep 6, 2021

Defman commented Sep 16, 2021

caelunshun commented Mar 13, 2021 •

edited

Loading

Defman left a comment •

edited

Loading

ambeeeeee Mar 13, 2021 •

edited

Loading

Defman Mar 13, 2021 •

edited

Loading

Defman Mar 14, 2021 •

edited

Loading

Defman Mar 14, 2021 •

edited

Loading