diff --git a/docs/doks/content/en/docs/internals/api.md b/docs/doks/content/en/docs/internals/api.md
index 504d27d..37a2c72 100644
--- a/docs/doks/content/en/docs/internals/api.md
+++ b/docs/doks/content/en/docs/internals/api.md
@@ -13,12 +13,14 @@ weight: 506
toc: true
---
-# ZSets
+## ZSets
-`ZSet`s can be added `a + b`, negated `-a`, `ZSetPython`s can be multiplied `a * n`.
+`ZSet`s can be added `a + b`, negated `-a`, `ZSetPython`s can be multiplied `a * n`
They also have the following methods:
+
+
```python
def iter(
self,
@@ -27,7 +29,9 @@ def iter(
...
```
-Iterates over the value, count of the `ZSet` in no particular order. Optionally filter on a set of values.
+[[src]](https://github.com/search?q=repo%3Aleontrolski%2Fstepping+path%3Asrc+%22def+iter%28%22&type=code) Iterates over the value, count of the `ZSet` in no particular order. Optionally filter on a set of values.
+
+
```python
def iter_by_index(
@@ -38,7 +42,7 @@ def iter_by_index(
...
```
-Iterates over the key, value, count of the indexed `ZSet` in the order defined by the index. Optionally filter on a set of values.
+[[src]](https://github.com/search?q=repo%3Aleontrolski%2Fstepping+path%3Asrc+%22def+_iter_by_index%28%22&type=code) Iterates over the key, value, count of the indexed `ZSet` in the order defined by the index. Optionally filter on a set of values.
# Operators
@@ -50,7 +54,9 @@ st.identity_print(
) -> T
```
-Prints `a`, then returns it.
+[[src]](https://github.com/search?q=repo%3Aleontrolski%2Fstepping+path%3Asrc+%22def+identity_print%28%22&type=code) Prints `a`, then returns it.
+
+
```python
st.ensure_python_zset(
@@ -58,7 +64,8 @@ st.ensure_python_zset(
) -> st.ZSet[T]
```
-Converts `ZSetSQl` -> `ZSetPython`.
+[[src]](https://github.com/search?q=repo%3Aleontrolski%2Fstepping+path%3Asrc+%22def+ensure_python_zset%28%22&type=code) Converts `ZSetSQl` -> `ZSetPython`
+
## Day to day
@@ -70,7 +77,9 @@ st.map(
) -> st.ZSet[V]
```
-Maps function `f` over all the values in `a`.
+[[src]](https://github.com/search?q=repo%3Aleontrolski%2Fstepping+path%3Asrc+%22def+map%28%22&type=code) Maps function `f` over all the values in `a`
+
+
```python
st.map_many(
@@ -80,7 +89,9 @@ st.map_many(
) -> st.ZSet[V]
```
-Maps function `f` over all the values in `a`. `f` returns many values in a `frozenset`, these are unioned together in the returned `ZSet`.
+[[src]](https://github.com/search?q=repo%3Aleontrolski%2Fstepping+path%3Asrc+%22def+map_many%28%22&type=code) Maps function `f` over all the values in `a`. `f` returns many values in a `frozenset`, these are unioned together in the returned `ZSet`
+
+
```python
st.filter(
@@ -90,7 +101,9 @@ st.filter(
) -> st.ZSet[T]
```
-Equivalent to SQL's `WHERE`.
+[[src]](https://github.com/search?q=repo%3Aleontrolski%2Fstepping+path%3Asrc+%22def+filter%28%22&type=code) Equivalent to SQL's `WHERE`
+
+
```python
st.join(
@@ -102,7 +115,9 @@ st.join(
) -> st.ZSet[st.Pair[T, U]]
```
-Equivalent to SQL's `LEFT JOIN`.
+[[src]](https://github.com/search?q=repo%3Aleontrolski%2Fstepping+path%3Asrc+%22def+join_lifted%28%22&type=code) Equivalent to SQL's `JOIN`
+
+
```python
st.outer_join(
@@ -114,7 +129,9 @@ st.outer_join(
) -> st.ZSet[st.Pair[T, Union[U, st.Empty]]]
```
-Equivalent to SQL's `LEFT OUTER JOIN`, with `Empty()` equivalent to `NULL`.
+[[src]](https://github.com/search?q=repo%3Aleontrolski%2Fstepping+path%3Asrc+%22def+outer_join_lifted%28%22&type=code) Equivalent to SQL's `LEFT OUTER JOIN`, with `Empty()` equivalent to `NULL`
+
+
```python
st.distinct(
@@ -122,7 +139,9 @@ st.distinct(
) -> st.ZSet[T]
```
-Equivalent to SQL's `DISTINCT`.
+[[src]](https://github.com/search?q=repo%3Aleontrolski%2Fstepping+path%3Asrc+%22def+distinct_lifted%28%22&type=code) Equivalent to SQL's `DISTINCT`
+
+
```python
st.add(
@@ -131,7 +150,9 @@ st.add(
) -> TAddable
```
-Adds two values, equivalent to SQL's `UNION` when applied to `ZSet`s.
+[[src]](https://github.com/search?q=repo%3Aleontrolski%2Fstepping+path%3Asrc+%22def+add%28%22&type=code) Adds two values, equivalent to SQL's `UNION` when applied to `ZSet`s.
+
+
```python
st.count(
@@ -139,9 +160,9 @@ st.count(
) -> st.ZSet[int]
```
-Counts the number of values (effectively `sum(count for _, count in a.iter())`).
+[[src]](https://github.com/search?q=repo%3Aleontrolski%2Fstepping+path%3Asrc+%22def+count_lifted%28%22&type=code) Counts the number of values (effectively `sum(count for _, count in a.iter())`). Returns a `ZSet` containing a single `int`
-Returns a `ZSet` containing a single `int`.
+
```python
st.first_n(
@@ -152,7 +173,7 @@ st.first_n(
) -> st.ZSet[T]
```
-Similar to SQL's `ORDER BY ... LIMIT n`. The output is unordered (it's still a `ZSet`), but calling:
+[[src]](https://github.com/search?q=repo%3Aleontrolski%2Fstepping+path%3Asrc+%22def+first_n_lifted%28%22&type=code) Similar to SQL's `ORDER BY ... LIMIT n`. The output is unordered (it's still a `ZSet`), but calling:
```python
for key, value, count in z.iter_by_index(index):
@@ -161,15 +182,17 @@ for key, value, count in z.iter_by_index(index):
Will yield values ordered by the index.
+
+
```python
st.transitive_closure(
a: st.ZSet[st.Pair[TIndexable, TIndexable]],
) -> st.ZSet[st.Pair[TIndexable, TIndexable]]
```
-Given a set of edges `left -> right`, returns the [transitive closure](https://en.wikipedia.org/wiki/Transitive_closure#/media/File:Transitive-closure.svg) of all the edges.
+[[src]](https://github.com/search?q=repo%3Aleontrolski%2Fstepping+path%3Asrc+%22def+transitive_closure_lifted%28%22&type=code) Given a set of edges `left -> right`, returns the [transitive closure](https://en.wikipedia.org/wiki/Transitive_closure#/media/File:Transitive-closure.svg) of all the edges.
-Example, given:
+**Example** -- given:
```
left right
@@ -197,6 +220,7 @@ left right
Read the code for ideas implementing other recursive functions.
+
## Group/Reduce
```python
@@ -208,13 +232,15 @@ st.reduce(
) -> st.ZSet[TReducable]
```
-A more general version of `st.count(...)`. In common usage `zero` and `pick_value` will be functions that either:
+[[src]](https://github.com/search?q=repo%3Aleontrolski%2Fstepping+path%3Asrc+%22def+reduce_lifted%28%22&type=code) A more general version of `st.count(...)`. In common usage `zero` and `pick_value` will be functions that either:
- Return `0` and an `int`, thereby implementing SQL's `SUM`.
-- Return an empty `ZSetPython[SomeType]` and a `ZSetPython[SomeType]` respectively, thereby (kinda) implementing SQL's `ARRAY_AGG`.
+- Return an empty `ZSetPython[SomeType]` and a `ZSetPython[SomeType]` respectively, thereby (kinda) implementing SQL's `ARRAY_AGG`
_Note that to handle the removal of rows in the inputted changes, `TReducable` has to implement `__mul__` (which luckily `ZSetPython`s do)._
+
+
```python
st.group_reduce_flatten(
a: st.ZSet[T],
@@ -225,9 +251,11 @@ st.group_reduce_flatten(
) -> st.ZSet[st.Pair[TReducable, K]]
```
-Equivalent to SQL's `SELECT reduce(...) FROM ... GROUP BY ...`. In common usage `zero` and `pick_value` will be functions that return a `ZSetPython[SomeType]`.
+[[src]](https://github.com/search?q=repo%3Aleontrolski%2Fstepping+path%3Asrc+%22def+group_reduce_flatten_lifted%28%22&type=code) Equivalent to SQL's `SELECT reduce(...) FROM ... GROUP BY ...`. In common usage `zero` and `pick_value` will be functions that return a `ZSetPython[SomeType]`
-The output is a `ZSet` of pairs of: the reduced value and the key they were grouped by.
+The output is a `ZSet` of pairs of the reduced value and the key they were grouped by.
+
+
```python
st.group(
@@ -237,7 +265,9 @@ st.group(
) -> st.Grouped[st.ZSet[T], K]
```
-Groups `a` by a key.
+[[src]](https://github.com/search?q=repo%3Aleontrolski%2Fstepping+path%3Asrc+%22def+group%28%22&type=code) Groups `a` by a key.
+
+
```python
st.flatten(
@@ -245,7 +275,8 @@ st.flatten(
) -> st.ZSet[st.Pair[T, K]]
```
-Flattens a `Group` to a more useful `ZSet`.
+[[src]](https://github.com/search?q=repo%3Aleontrolski%2Fstepping+path%3Asrc+%22def+flatten%28%22&type=code) Flattens a `Group` to a more useful `ZSet`
+
## Internal
@@ -255,7 +286,9 @@ st.neg(
) -> TNegable
```
-Return `-a` (remember, applicable to `ZSet`s).
+[[src]](https://github.com/search?q=repo%3Aleontrolski%2Fstepping+path%3Asrc+%22def+neg%28%22&type=code) Returns `-a` (remember, applicable to `ZSet`s).
+
+
```python
st.make_scalar(
@@ -265,7 +298,9 @@ st.make_scalar(
) -> T
```
-Turn a `ZSet` of count = 1 to a scalar value.
+[[src]](https://github.com/search?q=repo%3Aleontrolski%2Fstepping+path%3Asrc+%22def+make_scalar%28%22&type=code) Turn a `ZSet` of count = 1 to a scalar value.
+
+
```python
st.make_set(
@@ -273,8 +308,9 @@ st.make_set(
) -> st.ZSet[T]
```
-Turn a scalar into a `ZSet` of count = 1. SQL implicitly does this if you do `SELECT 1`.
+[[src]](https://github.com/search?q=repo%3Aleontrolski%2Fstepping+path%3Asrc+%22def+make_set%28%22&type=code) Turn a scalar into a `ZSet` of count = 1. SQL implicitly does this if you do `SELECT 1`
+
```python
st.add3(
@@ -284,7 +320,9 @@ st.add3(
) -> TAddable
```
-Add three things.
+[[src]](https://github.com/search?q=repo%3Aleontrolski%2Fstepping+path%3Asrc+%22def+add3%28%22&type=code) Add three things.
+
+
```python
st.haitch(
@@ -293,7 +331,8 @@ st.haitch(
) -> st.ZSet[T]
```
-Used internally by `st.distinct(...)`, efficiently watches for change of sign in counts.
+[[src]](https://github.com/search?q=repo%3Aleontrolski%2Fstepping+path%3Asrc+%22def+haitch%28%22&type=code) Used internally by `st.distinct(...)`, efficiently watches for change of sign in counts.
+
## Delay/Differentiate/Integrate
@@ -303,7 +342,9 @@ st.delay(
) -> T
```
-Returns the previous value it was called with, see [reference page]({{< ref "/docs/internals/how-it-works.md" >}}).
+[[src]](https://github.com/search?q=repo%3Aleontrolski%2Fstepping+path%3Asrc+%22def+delay%28%22&type=code) Returns the previous value it was called with, see [reference page]({{< ref "/docs/internals/how-it-works.md" >}}#delays).
+
+
```python
st.delay_indexed(
@@ -313,7 +354,9 @@ st.delay_indexed(
) -> st.ZSet[T]
```
-Returns the previous value it was called with, with indexes.
+[[src]](https://github.com/search?q=repo%3Aleontrolski%2Fstepping+path%3Asrc+%22def+delay_indexed%28%22&type=code) Returns the previous value it was called with, with indexes.
+
+
```python
st.differentiate(
@@ -321,7 +364,9 @@ st.differentiate(
) -> TAddAndNegable
```
-See [reference page]({{< ref "/docs/internals/how-it-works.md" >}}).
+[[src]](https://github.com/search?q=repo%3Aleontrolski%2Fstepping+path%3Asrc+%22def+differentiate%28%22&type=code) Differentiates input values over time. See [reference page]({{< ref "/docs/internals/how-it-works.md" >}}#differentiation).
+
+
```python
st.integrate(
@@ -329,15 +374,9 @@ st.integrate(
) -> TAddable
```
-See [reference page]({{< ref "/docs/internals/how-it-works.md" >}}).
+[[src]](https://github.com/search?q=repo%3Aleontrolski%2Fstepping+path%3Asrc+%22def+integrate%28%22&type=code) Integrates input values over time. See [reference page]({{< ref "/docs/internals/how-it-works.md" >}}#integration).
-```python
-st.integrate_delay(
- a: TAddable,
-) -> TAddable
-```
-
-See [reference page]({{< ref "/docs/internals/how-it-works.md" >}}).
+
```python
st.integrate_indexed(
@@ -347,9 +386,21 @@ st.integrate_indexed(
) -> st.ZSet[T]
```
-See [reference page]({{< ref "/docs/internals/how-it-works.md" >}}).
+[[src]](https://github.com/search?q=repo%3Aleontrolski%2Fstepping+path%3Asrc+%22def+integrate_indexed%28%22&type=code) Integrates input values over time, adds indexes to the delay node. See [reference page]({{< ref "/docs/internals/how-it-works.md" >}}#integration).
-# Compile
+
+
+```python
+st.integrate_delay(
+ a: TAddable,
+) -> TAddable
+```
+
+[[src]](https://github.com/search?q=repo%3Aleontrolski%2Fstepping+path%3Asrc+%22def+integrate_delay%28%22&type=code) Integrates input values over time, returns the previous value. See [reference page]({{< ref "/docs/internals/how-it-works.md" >}}#integration).
+
+## Compile
+
+
```python
st.compile(
@@ -357,7 +408,9 @@ st.compile(
) -> st.Graph[Any, Any]
```
-Compile a query function to a graph.
+[[src]](https://github.com/search?q=repo%3Aleontrolski%2Fstepping+path%3Asrc+%22def+compile%28%22&type=code) Compile a query function to a graph.
+
+
```python
st.compile_lazy(
@@ -365,7 +418,9 @@ st.compile_lazy(
) -> Callable[[], st.Graph[Any, Any]]
```
-Returns a function with no arguments that compiles the graph, caches it and returns it.
+[[src]](https://github.com/search?q=repo%3Aleontrolski%2Fstepping+path%3Asrc+%22def+compile_lazy%28%22&type=code) Returns a function with no arguments that compiles the graph, caches it and returns it.
+
+
```python
st.compile_typeof(
@@ -373,14 +428,18 @@ st.compile_typeof(
) -> type[T]
```
-Get the resolved type of `t` at query compile time.
+[[src]](https://github.com/search?q=repo%3Aleontrolski%2Fstepping+path%3Asrc+%22compile_typeof%22&type=code) Get the resolved type of `t` at query compile time.
+
+
```python
with st.at_compile_time:
...
```
-Run this code block at query compile time.
+[[src]](https://github.com/search?q=repo%3Aleontrolski%2Fstepping+path%3Asrc+%22at_complie_time%22&type=code) Run this code block at query compile time.
+
+
# Run
@@ -393,7 +452,9 @@ st.iteration(
) -> tuple[Any, ...]
```
-Run a single iteration of a graph, returning resultant changes.
+[[src]](https://github.com/search?q=repo%3Aleontrolski%2Fstepping+path%3Asrc+%22def+iteration%28%22&type=code) Run a single iteration of a graph, returning resultant changes.
+
+
```python
st.actions(
@@ -402,9 +463,11 @@ st.actions(
) -> Any
```
-Return a tuple of helpers to insert, remove, replace. Examples dotted around the docs.
+[[src]](https://github.com/search?q=repo%3Aleontrolski%2Fstepping+path%3Asrc+%22def+actions%28%22&type=code) Return a tuple of helpers to insert, remove, replace. Examples dotted around the docs.
-# Indexes
+
+
+## Indexes
```python
st.pick_index(
@@ -414,7 +477,9 @@ st.pick_index(
) -> st.Index[T, K]
```
-Pick an index of `t`. The index key should be indexable:
+[[src]](https://github.com/search?q=repo%3Aleontrolski%2Fstepping+path%3Asrc+%22def+pick_index%28%22&type=code) Pick an index of `t`. The index key should be indexable:
+
+
```python
IndexableAtom = str | int | float | bool | None | date | datetime | UUID
@@ -423,6 +488,8 @@ Indexable = IndexableAtom | tuple[IndexableAtom, ...]
Optionally, a boolean for ascending can be passed in, this is equivalent to SQL's `ASC`/`DESC`. If the key is `tuple[IndexableAtom, ...]`, `ascending` must be a tuple of bools of the same length.
+
+
```python
st.pick_identity(
t: type[KAtom],
@@ -430,25 +497,31 @@ st.pick_identity(
) -> st.Index[KAtom, KAtom]
```
-Pick an index of the value itself.
+[[src]](https://github.com/search?q=repo%3Aleontrolski%2Fstepping+path%3Asrc+%22def+pick_identity%28%22&type=code) Pick an index of the value itself.
-# Database Connections
+
+
+## Database Connections
```python
with st.connection_postgres(db_url: str) as conn:
...
```
-Context manager for a Postgres connection.
+[[src]](https://github.com/search?q=repo%3Aleontrolski%2Fstepping+path%3Asrc+%22def+connection_postgres%28%22&type=code) Context manager for a Postgres connection.
+
+
```python
with st.connection_sqlite(db_url: pathlib.Path) as conn:
...
```
-Context manager for a SQLite connection.
+[[src]](https://github.com/search?q=repo%3Aleontrolski%2Fstepping+path%3Asrc+%22def+connection_sqlite%28%22&type=code) Context manager for a SQLite connection.
+
+
-# Helpers
+## Helpers
```python
st.annotate_zset(
@@ -456,7 +529,9 @@ st.annotate_zset(
) -> tuple[pydantic.Validator, ...]
```
-Use when it is required to serialize a `ZSetPython` to a store, example:
+[[src]](https://github.com/search?q=repo%3Aleontrolski%2Fstepping+path%3Asrc+%22def+annotate_zset%28%22&type=code) Use when it is required to serialize a `ZSetPython` to a store, example:
+
+
```python
class A(st.Data):
@@ -466,6 +541,8 @@ class A(st.Data):
_At some point, the magic `pydantic` method will be added to make this redundant._
+
+
```python
st.batched(
iterable: list[T],
@@ -473,7 +550,9 @@ st.batched(
) -> Iterator[list[T]]
```
-See [itertools docs](https://docs.python.org/3/library/itertools.html#itertools-recipes).
+[[src]](https://github.com/search?q=repo%3Aleontrolski%2Fstepping+path%3Asrc+%22def+batched%28%22&type=code) See [itertools docs](https://docs.python.org/3/library/itertools.html#itertools-recipes).
+
+
```python
st.write_png(
@@ -484,4 +563,4 @@ st.write_png(
) -> NoneType
```
-Write a graph to a `.png` file using `dot`.
+[[src]](https://github.com/search?q=repo%3Aleontrolski%2Fstepping+path%3Asrc+%22def+write_png%28%22&type=code) Write a graph to a `.png` file using `dot`
diff --git a/docs/doks/content/en/docs/internals/how-it-works.md b/docs/doks/content/en/docs/internals/how-it-works.md
index 8c15b14..fc145f2 100644
--- a/docs/doks/content/en/docs/internals/how-it-works.md
+++ b/docs/doks/content/en/docs/internals/how-it-works.md
@@ -202,9 +202,55 @@ st.Graph[
]
```
-_`Aθ` is just a collection of arguments with length θ._
+_`Aθ` is just a collection of arguments with length θ -- waiting on mypy support for `TypeVarTuple` over here._
-Let's look at a more interesting graph and write a `.png` file with a diagram of it:
+The internal structure of the graph from above is:
+
+```python
+Graph(
+ vertices=[
+ t:input_0 OperatorKind.identity(ZSet[str]) -> ZSet[str],
+ t:delayed OperatorKind.delay(ZSet[str]) -> ZSet[str]],
+ input=[
+ (t:input_0 OperatorKind.identity(ZSet[str]) -> ZSet[str], 0)
+ ],
+ internal={
+ (
+ t:input_0 OperatorKind.identity(ZSet[str]) -> ZSet[str],
+ (t:delayed OperatorKind.delay(ZSet[str]) -> ZSet[str], 0)
+ )
+ },
+ output=[
+ t:delayed OperatorKind.delay(ZSet[str]) -> ZSet[str]
+ ],
+ run_no_output=[]
+)
+```
+
+The graph has a:
+
+- List of all the vertices within it.
+- List of all the inputs. These are a tuple of a `Vertex` and `0` for the first argument, `1` for the second argument (in the case of binary vertices).
+- Set of all the internal edges. These are each a tuple of a `Vertex`, pointing to a (vertex, `0|1`) tuple.
+- List of output vertices.
+- List of vertices that we want to run, but we don't use in the output.
+
+A vertex has a fairly simple type, there are unary and binary ones, let's look at a unary one:
+
+```python
+class VertexUnary(Generic[T, V]):
+ t: type[T]
+ v: type[V]
+ operator_kind: OperatorKind
+ path: Path
+ f: Callable[[T], V]
+```
+
+`t` is the input type, `v` is the output type, operator kind is `add`, `delay`, `filter` etc, path is effectively the unique name of the vertex, f is the function that it runs.
+
+
+
+Now let's look at a more interesting graph and write a `.png` file with a diagram of it:
```python
def query_graph(a: st.ZSet[A], b: st.ZSet[B]) -> st.ZSet[st.Pair[A, B]]:
@@ -467,7 +513,7 @@ Indexes are a generic:
Index[T, K]
```
-Where `T` the same `T` from `ZSet[T]` and `K` is an `Indexable` key of `T` (including tuples of many keys). Indexes also have an `ascending=True|False` for each of the key(s).
+Where `T` is the same `T` from `ZSet[T]` and `K` is an [`Indexable`](https://github.com/search?q=repo%3Aleontrolski%2Fstepping+path%3Asrc+%22Indexable+%22&type=code) key of `T` (including tuples of many keys). Indexes also have an `ascending=True|False` for each of the key(s).
Indexes are used to efficiently implement many things in stepping: joins, limits, grouping, distinct.
diff --git a/scripts/md_api.py b/scripts/md_api.py
index 70baad0..e6ab44f 100644
--- a/scripts/md_api.py
+++ b/scripts/md_api.py
@@ -45,14 +45,21 @@
]:
sig_str = sig_str.replace(a, b)
- by_module[f.__module__].append(sig_str)
-
-for module_name, sig_strs in sorted(by_module.items()):
- print("## " + module_name)
- print()
- for sig_str in sig_strs:
- print("```python")
- print(sig_str)
- # print("\n".join(" " + line for line in sig_str.splitlines()))
- print("```")
- print()
+ # by_module[f.__module__].append(sig_str)
+ import inspect
+ # lineno = inspect.getlineno(f)
+ _, lineno = inspect.findsource(f)
+ filename = f.__module__.replace(".", "/") + ".py"
+ s = f"[[src]](https://github.com/search?q=repo%3Aleontrolski%2Fstepping+path%3Asrc+%22def+{f.__name__}%28%22&type=code)"
+ # s = f"[[source]](https://github.com/leontrolski/stepping/blob/fedca46b2f/src/{filename}#L{lineno})"
+ print(s)
+
+# for module_name, sig_strs in sorted(by_module.items()):
+# print("## " + module_name)
+# print()
+# for sig_str in sig_strs:
+# print("```python")
+# print(sig_str)
+# # print("\n".join(" " + line for line in sig_str.splitlines()))
+# print("```")
+# print()