StreamingDataFrame.merge(): prep work #725

daniil-quix · 2025-01-23T14:48:47Z

This PR adds some prep work for the merge() feature.
No new features have been added yet, and SDFs are expected to work like before.

I want to merge it before implementing the actual merge() to split the review process into steps.

Added support for multiple parents in Stream class to implement the merge() feature.
The Stream.compose() method now uses graphlib.TopologicalSorter to resolve the sequence of operations in the tree of Stream objects.
With multiple parents, each tree may have multiple root Streams (e.g. when merging 2 SDFs will result in a single tree with two starting points). This PR handles that.

Related to #487

quixstreams/core/stream/stream.py

tests/test_quixstreams/test_core/test_stream.py

Co-authored-by: Remy Gwaramadze <[email protected]>

daniil-quix added 7 commits January 23, 2025 14:01

WIP: merging streams together

1a28f5b

Add support for multiple parents to Stream class

4409763

Fix Stream tests

75d2efb

Remove BaseStreaming base class

bb1b35a

Fix StreamingSeries tests

8772444

Fix mypy errors

8562155

Move InvalidTopology exception

667ccbb

daniil-quix changed the title ~~StreamingDataFrame.merge(): prep work #722~~ StreamingDataFrame.merge(): prep work Jan 23, 2025

gwaramadze reviewed Jan 24, 2025

View reviewed changes

daniil-quix and others added 2 commits January 24, 2025 11:26

Update quixstreams/core/stream/stream.py

d73b808

Co-authored-by: Remy Gwaramadze <[email protected]>

Post-review updates

8c938c0

gwaramadze approved these changes Jan 24, 2025

View reviewed changes

daniil-quix merged commit e3795ab into main Jan 24, 2025
3 checks passed

daniil-quix deleted the feature/merge-prep branch January 24, 2025 12:53

Provide feedback