Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(datasets): fix SparkStreamingDataSet docstring #236

Merged
merged 10 commits into from
Jun 13, 2023
Merged

Conversation

noklam
Copy link
Contributor

@noklam noklam commented Jun 13, 2023

Description

Development notes

Checklist

  • Opened this PR as a 'Draft Pull Request' if it is work-in-progress
  • Updated the documentation to reflect the code changes
  • Added a description of this change in the relevant RELEASE.md file
  • Added tests to cover my changes

@astrojuanlu
Copy link
Member

This contains commits from gh-234, can be rebased

Copy link
Member

@astrojuanlu astrojuanlu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good but would like to see a test that it works

@@ -20,15 +20,16 @@ class SparkStreamingDataSet(AbstractDataSet):
`YAML API <https://kedro.readthedocs.io/en/stable/data/\
data_catalog.html#use-the-data-catalog-with-the-yaml-api>`_:
.. code-block:: yaml

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, did you test it locally? Maybe open a PR on the framework side cherry picking this commit kedro-org/kedro@9c08611

@AhdraMeraliQB AhdraMeraliQB changed the title Fix SparkStreamingDataSet Docstring fix: Fix SparkStreamingDataSet Docstring Jun 13, 2023
@noklam noklam force-pushed the fix/setup-pickle branch from 0feaaf4 to de107f8 Compare June 13, 2023 13:08
@noklam noklam enabled auto-merge (squash) June 13, 2023 13:22
@deepyaman deepyaman changed the title fix: Fix SparkStreamingDataSet Docstring fix(datasets): fix SparkStreamingDataSet docstring Jun 13, 2023
Copy link
Member

@deepyaman deepyaman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice! I was looking at this yesterday; this seems to cover the two comments Ivan had left on the original PR that weren't addressed. :)

I actually had one more question on this, and maybe you can roll it into the same PR--does it make sense for the "Spark Streaming" docs to be in spark/README.md? It only applies to a single dataset implementation, so it's a bit confusing.

But approving in case not sure how to best handle that/want to do it later.

@noklam
Copy link
Contributor Author

noklam commented Jun 13, 2023

@deepyaman Frankly I don't know what's the README.md for, it's not a standard and only exists for very few datasets. I don't believe it will be render in the Kedro's doc either so I would say it's not very important at all.

Signed-off-by: Nok Chan <[email protected]>
@noklam noklam force-pushed the fix/setup-pickle branch from 6b5a61a to c419202 Compare June 13, 2023 14:18
@noklam noklam disabled auto-merge June 13, 2023 15:01
@noklam
Copy link
Contributor Author

noklam commented Jun 13, 2023

This is still failing - I try to fix the build but it starts complaining the __init__ method.

@astrojuanlu
Copy link
Member

does it make sense for the "Spark Streaming" docs to be in spark/README.md?

Until we significantly rework how our docs are built, we don't have a way to place narrative/prose documentation from this repo on docs.kedro.org (see kedro-org/kedro#2072 (comment))

Signed-off-by: Ahdra Merali <[email protected]>
@noklam
Copy link
Contributor Author

noklam commented Jun 13, 2023

kedro-org/kedro#2679

I created this PR in kedro , so whenever you made changes in this PR you can re-trigger the CI to see if the build is success.

Ahdra Merali and others added 7 commits June 13, 2023 17:42
@noklam noklam merged commit 031b737 into main Jun 13, 2023
@noklam noklam deleted the fix/setup-pickle branch June 13, 2023 19:52
@noklam noklam restored the fix/setup-pickle branch June 14, 2023 13:42
afaqueahmad7117 pushed a commit to afaqueahmad7117/kedro-plugins that referenced this pull request Jun 19, 2023
* fix malform docstring in SparkStreamingDataSet

Signed-off-by: Nok Chan <[email protected]>

* change indent

Signed-off-by: Nok Chan <[email protected]>

* Test without docstring

Signed-off-by: Ahdra Merali <[email protected]>

* Add back docstring

Signed-off-by: Ahdra Merali <[email protected]>

* Format docstring

Signed-off-by: Ahdra Merali <[email protected]>

* Fix typo

Signed-off-by: Ahdra Merali <[email protected]>

* Fix typo

* Lint

Signed-off-by: Ahdra Merali <[email protected]>

---------

Signed-off-by: Nok Chan <[email protected]>
Signed-off-by: Ahdra Merali <[email protected]>
Co-authored-by: Ahdra Merali <[email protected]>
Co-authored-by: Ahdra Merali <[email protected]>
Signed-off-by: Afaque Ahmad <[email protected]>
afaqueahmad7117 pushed a commit to afaqueahmad7117/kedro-plugins that referenced this pull request Jun 19, 2023
* fix malform docstring in SparkStreamingDataSet

Signed-off-by: Nok Chan <[email protected]>

* change indent

Signed-off-by: Nok Chan <[email protected]>

* Test without docstring

Signed-off-by: Ahdra Merali <[email protected]>

* Add back docstring

Signed-off-by: Ahdra Merali <[email protected]>

* Format docstring

Signed-off-by: Ahdra Merali <[email protected]>

* Fix typo

Signed-off-by: Ahdra Merali <[email protected]>

* Fix typo

* Lint

Signed-off-by: Ahdra Merali <[email protected]>

---------

Signed-off-by: Nok Chan <[email protected]>
Signed-off-by: Ahdra Merali <[email protected]>
Co-authored-by: Ahdra Merali <[email protected]>
Co-authored-by: Ahdra Merali <[email protected]>
Signed-off-by: Afaque Ahmad <[email protected]>
@merelcht merelcht deleted the fix/setup-pickle branch August 15, 2023 13:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants