Skip to content

Commit

Permalink
Update documentation (#663)
Browse files Browse the repository at this point in the history 
Co-authored-by: daniil-quix <[email protected]>
  • Loading branch information
@github-actions @daniil-quix
github-actions[bot] and daniil-quix authored Dec 2, 2024
1 parent e09cd54 commit 3705e9a
Show file tree
Hide file tree
Showing 3 changed files with 60 additions and 50 deletions.
13 changes: 7 additions & 6 deletions docs/api-reference/application.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -482,7 +482,8 @@ See :class:`quixstreams.sources.base.BaseSource` for more details.

- `source`: a :class:`quixstreams.sources.BaseSource` instance
- `topic`: the :class:`quixstreams.models.Topic` instance the source will produce to
Default: the source default
Default - the topic generated by the `source.default_topic()` method.
Note: the names of default topics are prefixed with "source__".

<a id="quixstreams.app.Application.run"></a>

Expand All @@ -494,7 +495,7 @@ Default: the source default
def run(dataframe: Optional[StreamingDataFrame] = None)
```

[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/app.py#L699)
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/app.py#L700)

Start processing data from Kafka using provided `StreamingDataFrame`

Expand Down Expand Up @@ -530,7 +531,7 @@ app.run()
def setup_topics()
```

[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/app.py#L822)
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/app.py#L823)

Validate and create the topics

Expand All @@ -542,7 +543,7 @@ Validate and create the topics
class ApplicationConfig(BaseSettings)
```

[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/app.py#L998)
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/app.py#L999)

Immutable object holding the application configuration

Expand All @@ -565,7 +566,7 @@ def settings_customise_sources(
) -> Tuple[PydanticBaseSettingsSource, ...]
```

[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/app.py#L1033)
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/app.py#L1034)

Included to ignore reading/setting values from the environment

Expand All @@ -579,7 +580,7 @@ Included to ignore reading/setting values from the environment
def copy(**kwargs) -> Self
```

[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/app.py#L1046)
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/app.py#L1047)

Update the application config and return a copy

55 changes: 30 additions & 25 deletions docs/api-reference/quixstreams.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -10486,7 +10486,8 @@ See :class:`quixstreams.sources.base.BaseSource` for more details.

- `source`: a :class:`quixstreams.sources.BaseSource` instance
- `topic`: the :class:`quixstreams.models.Topic` instance the source will produce to
Default: the source default
Default - the topic generated by the `source.default_topic()` method.
Note: the names of default topics are prefixed with "source__".

<a id="quixstreams.app.Application.run"></a>

Expand All @@ -10496,7 +10497,7 @@ Default: the source default
def run(dataframe: Optional[StreamingDataFrame] = None)
```

[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/app.py#L699)
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/app.py#L700)

Start processing data from Kafka using provided `StreamingDataFrame`

Expand Down Expand Up @@ -10528,7 +10529,7 @@ app.run()
def setup_topics()
```

[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/app.py#L822)
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/app.py#L823)

Validate and create the topics

Expand All @@ -10540,7 +10541,7 @@ Validate and create the topics
class ApplicationConfig(BaseSettings)
```

[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/app.py#L998)
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/app.py#L999)

Immutable object holding the application configuration

Expand All @@ -10561,7 +10562,7 @@ def settings_customise_sources(
) -> Tuple[PydanticBaseSettingsSource, ...]
```

[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/app.py#L1033)
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/app.py#L1034)

Included to ignore reading/setting values from the environment

Expand All @@ -10573,7 +10574,7 @@ Included to ignore reading/setting values from the environment
def copy(**kwargs) -> Self
```

[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/app.py#L1046)
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/app.py#L1047)

Update the application config and return a copy

Expand Down Expand Up @@ -11555,6 +11556,8 @@ This method is triggered when the topic is not provided to the source.

The source must return a default topic configuration.

Note: if the default topic is used, the Application will prefix its name with "source__".

<a id="quixstreams.sources.base.source.Source"></a>

### Source
Expand All @@ -11563,7 +11566,7 @@ The source must return a default topic configuration.
class Source(BaseSource)
```

[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/sources/base/source.py#L135)
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/sources/base/source.py#L137)

A base class for custom Sources that provides a basic implementation of `BaseSource`
interface.
Expand Down Expand Up @@ -11621,7 +11624,7 @@ if __name__ == "__main__":
def __init__(name: str, shutdown_timeout: float = 10) -> None
```

[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/sources/base/source.py#L185)
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/sources/base/source.py#L187)

**Arguments**:

Expand All @@ -11637,7 +11640,7 @@ def __init__(name: str, shutdown_timeout: float = 10) -> None
def running() -> bool
```

[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/sources/base/source.py#L199)
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/sources/base/source.py#L201)

Property indicating if the source is running.

Expand All @@ -11651,7 +11654,7 @@ The `stop` method will set it to `False`. Use it to stop the source gracefully.
def cleanup(failed: bool) -> None
```

[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/sources/base/source.py#L207)
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/sources/base/source.py#L209)

This method is triggered once the `run` method completes.

Expand All @@ -11667,7 +11670,7 @@ It flushes the producer when `_run` completes successfully.
def stop() -> None
```

[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/sources/base/source.py#L218)
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/sources/base/source.py#L220)

This method is triggered when the application is shutting down.

Expand All @@ -11681,7 +11684,7 @@ It sets the `running` property to `False`.
def start() -> None
```

[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/sources/base/source.py#L226)
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/sources/base/source.py#L228)

This method is triggered in the subprocess when the source is started.

Expand All @@ -11696,7 +11699,7 @@ It marks the source as running, execute it's run method and ensure cleanup happe
def run()
```

[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/sources/base/source.py#L242)
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/sources/base/source.py#L244)

This method is triggered in the subprocess when the source is started.

Expand All @@ -11714,7 +11717,7 @@ def serialize(key: Optional[object] = None,
timestamp_ms: Optional[int] = None) -> KafkaMessage
```

[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/sources/base/source.py#L250)
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/sources/base/source.py#L252)

Serialize data to bytes using the producer topic serializers and return a `quixstreams.models.messages.KafkaMessage`.

Expand All @@ -11736,7 +11739,7 @@ def produce(value: Optional[Union[str, bytes]] = None,
buffer_error_max_tries: int = 3) -> None
```

[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/sources/base/source.py#L266)
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/sources/base/source.py#L268)

Produce a message to the configured source topic in Kafka.

Expand All @@ -11748,7 +11751,7 @@ Produce a message to the configured source topic in Kafka.
def flush(timeout: Optional[float] = None) -> None
```

[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/sources/base/source.py#L291)
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/sources/base/source.py#L293)

This method flush the producer.

Expand All @@ -11771,12 +11774,14 @@ None use producer default or -1 is infinite. Default: None
def default_topic() -> Topic
```

[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/sources/base/source.py#L309)
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/sources/base/source.py#L311)

Return a default topic matching the source name.

The default topic will not be used if the topic has already been provided to the source.

Note: if the default topic is used, the Application will prefix its name with "source__".

**Returns**:

`quixstreams.models.topics.Topic`
Expand All @@ -11789,7 +11794,7 @@ The default topic will not be used if the topic has already been provided to the
class StatefulSource(Source)
```

[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/sources/base/source.py#L326)
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/sources/base/source.py#L330)

A `Source` class for custom Sources that need a state.

Expand Down Expand Up @@ -11847,7 +11852,7 @@ if __name__ == "__main__":
def __init__(name: str, shutdown_timeout: float = 10) -> None
```

[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/sources/base/source.py#L376)
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/sources/base/source.py#L380)

**Arguments**:

Expand All @@ -11866,7 +11871,7 @@ def configure(topic: Topic,
**kwargs) -> None
```

[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/sources/base/source.py#L386)
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/sources/base/source.py#L390)

This method is triggered before the source is started.

Expand All @@ -11881,7 +11886,7 @@ It configures the source's Kafka producer, the topic it will produce to and the
def store_partitions_count() -> int
```

[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/sources/base/source.py#L405)
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/sources/base/source.py#L409)

Count of store partitions.

Expand All @@ -11896,7 +11901,7 @@ Used to configure the number of partition in the changelog topic.
def assigned_store_partition() -> int
```

[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/sources/base/source.py#L414)
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/sources/base/source.py#L418)

The store partition assigned to this instance

Expand All @@ -11909,7 +11914,7 @@ The store partition assigned to this instance
def store_name() -> str
```

[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/sources/base/source.py#L421)
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/sources/base/source.py#L425)

The source store name

Expand All @@ -11922,7 +11927,7 @@ The source store name
def state() -> State
```

[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/sources/base/source.py#L428)
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/sources/base/source.py#L432)

Access the `State` of the source.

Expand All @@ -11938,7 +11943,7 @@ Important: after each `.flush()` call, a previously returned instance is invalid
def flush(timeout: Optional[float] = None) -> None
```

[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/sources/base/source.py#L447)
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/sources/base/source.py#L451)

This method commit the state and flush the producer.

Expand Down
Loading

0 comments on commit 3705e9a

Please sign in to comment.