Python implementation

[jsa34]

There have been various discussions happening regarding an official Cucumber implementation in Python.

I thought it would be good to converge and discuss a plan forward and anyone who might be interested in supporting in this forum.

Options previously discussed

• Fork/bring under Cucumber space behave: https://github.com/behave/behave
• Fork/bring under Cucumber space Pytest-BDD: https://github.com/pytest-dev/pytest-bdd
• cherry pick from Pytest-BDD-NG features and backport to Pytest-BDD: https://github.com/elchupanebrej/pytest-bdd-ng
• start from scratch

Existing discussions

(please let me know of any others and I can append here as I know there have been others)

• behave discussion: Adopt under the cucumber org? behave/behave#1059

@mattwynne @Johnlon @The-BDD-Coach @blaisep @elchupanebrej @youtux @kieran-ryan

[The-BDD-Coach]

We should also ask for input from @youtux and @kieran-ryan.

My personal opinions are:

1. It doesn't make sense to fork Behave. As much as I appreciate the work that went into Behave, and even though I have used it extensively, I think that the code base is too old and too complex to be a good base for the next generation.
2. Pytest-BDD would make a good base; it gets a steady stream of updates from Pytest, which has a very large and active community.
3. We definitely want the official Cucumber parser and the other upgrades that Kostya added when he forked Pytest-BDD to create Pytest-BDD-NG.

If we accept those three premises, then I see these possible paths forward:
A) Back-port the Pytest-BDD-NG changes into Pytest-BDD.
B) After back-porting the NG changes, adopt Pytest-BDD as the official Cucumber implementation for Python.
OR
C) Adopt Pytest-BDD-NG as the official Cucumber implementation for Python. I don't like this approach because we would then have to catch NG up with all of the changes that have been made to Pytest-BDD since the fork, and would then have to continually merge in changes from Pytest and Pytest-BDD in order to maintain compatibility with those projects. I think that it is very important to maintain compatibility with Pytest; that community is very large and very active, and we want everyone in that community to be able to easily adopt the Cucumber implementation.

Thoughts?

[jsa34]

We should also ask for input from @youtux and @kieran-ryan.

My personal opinions are:

1. It doesn't make sense to fork Behave. As much as I appreciate the work that went into Behave, and even though I have used it extensively, I think that the code base is too old and too complex to be a good base for the next generation.
2. Pytest-BDD would make a good base; it gets a steady stream of updates from Pytest, which has a very large and active community.
3. We definitely want the official Cucumber parser and the other upgrades that Kostya added when he forked Pytest-BDD to create Pytest-BDD-NG.

If we accept those three premises, then I see these possible paths forward: A) Back-port the Pytest-BDD-NG changes into Pytest-BDD. B) After back-porting the NG changes, adopt Pytest-BDD as the official Cucumber implementation for Python. OR C) Adopt Pytest-BDD-NG as the official Cucumber implementation for Python. I don't like this approach because we would then have to catch NG up with all of the changes that have been made to Pytest-BDD since the fork, and would then have to continually merge in changes from Pytest and Pytest-BDD in order to maintain compatibility with those projects. I think that it is very important to maintain compatibility with Pytest; that community is very large and very active, and we want everyone in that community to be able to easily adopt the Cucumber implementation.

Thoughts?

Hello!

Thanks!

Sorry, I joined the conversation towards the end and just wanted to make sure the good ideas and momentum were not lost. I will update with pleasure any other people that should be involved in the discussion and any other ideas that have been suggested. I have added both mentioned to the members referred to in the discussion topic.

[Johnlon]

option A and b sounds sensible and healthier

[elchupanebrej]

Agree, but the second integration of the official parser will totally repeat efforts done by pytest-bdd-ng (and will make same internal/external APIs changes). Maybe a good idea to use pytest-bdd-ng as a big major release of pytest-bdd. Direct back-merge is not possible, but user APIs are left very close to the original pytest-bdd

[The-BDD-Coach]

I am a competent Python programmer and could assist, but I have no knowledge of the internals of Pytest-BDD-NG, Pytest-BDD, or Pytest, so I can't guide the project. Who can take the lead?

I think that there are a few completely independent tasks to back-port support for:
T1) The official Cucumber parser into Pytest-BDD. (I think this is the most important change.)
T2) Cucumber Expressions
T3) Standard JSON output for reporting
T4) The Cucumber Messages protocol (also for reporting)

That is, I think that each of these could be implemented by a different person on independent schedules. Does anyone know why they wouldn't be independent?

[youtux]

I think those can be independently implemented indeed

[elchupanebrej]

Please add to list:
T5) Integrate https://pypi.org/project/ci-environment/
T6) Move pygments / sphinx-gherkindoc (https://pygments.org/languages/ ) to use official parser

[elchupanebrej]

T3) on the cucumber community end could be continued from https://github.com/orgs/cucumber/discussions/2202#discussioncomment-10510928

[youtux]

Please add to list:

T5) Integrate https://pypi.org/project/ci-environment/

T6) Move pygments / sphinx-gherkindoc (https://pygments.org/languages/ ) to use official parser

To me, these don't seem to be part of the scope of this project.

[Johnlon]

same re python and pytest. but i didn't know anything about godog internals either before I tweaked that. I'm on hols so will take a look when I get back cheers John

…

On Mon, 26 Aug 2024, 4:27 pm The-BDD-Coach, ***@***.***> wrote: I am a competent Python programmer and could assist, but I have no knowledge of the internals of Pytest-BDD-NG, Pytest-BDD, or Pytest, so I can't guide the project. Who can take the lead? I think that there are a few completely independent tasks to back-port support for: T1) The official Cucumber parser into Pytest-BDD. (I think this is the most important change.) T2) Cucumber Expressions T3) Standard JSON output for reporting T4) The Cucumber Messages protocol (also for reporting) That is, I think that each of these could be implemented by a different person on independent schedules. Does anyone know why they wouldn't be independent? — Reply to this email directly, view it on GitHub <#2202 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAGMFGE7WJSVTM4PW575B2TZTNCMPAVCNFSM6AAAAABNB22442VHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTANBVGMZTENQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[youtux]

I would love to see pytest-bdd using the official parser. I just don’t have enough time myself to do it, but if somebody feels like starting the endeavour, I’m happy to offer my time in reviewing and make sure it gets merged.

[The-BDD-Coach]

It has already been done in Pytest-BDD-NG; if we use that as a guide and get some direction from Kostya I think that shouldn't be too difficult. He just messaged me and said that he expects to have some time this weekend to join this conversation.

@elchupanebrej, can you give us an estimate of the time it would take you to do each of these tasks? What about the time it would take for someone unfamiliar with the code bases to do the task (but with Pytest-BDD-NG as a guide)? Those estimates would give us at least an upper and lower bound on the size of the effort.

[elchupanebrej]

Hi @The-BDD-Coach, @youtux,

I hope this message finds you well.

T1) Official Cucumber Parser Integration into Pytest-BDD

• The current model of pytest-bdd is quite complex and would need to be restructured to directly use the Cucumber parser. This would require introducing an adapter and making API changes to support the following points, necessitating major version releases. (My comment here went unanswered). I implemented this in my fork using the attrs library, which is used in pytest. When I previously attempted to contribute to the parser/components, my PRs were not addressed for months. Repeating this effort could take approximately one man-month.

T2) Cucumber Expressions

• These are almost usable in their current state, though not all features are available. Some model adjustments are needed, but they are not extensive. However, this still requires the refactoring mentioned in T1.

T3) Standard JSON Output for Reporting
T4) The Cucumber Messages Protocol (also for reporting)

• In pytest-bdd-ng, I used a generated model from the JSON schema, which works with minor patches. The most significant effort was adapting hooks to the order used by the reporter. If I recall correctly, it took about three months of dedicated work to complete all these points.

The pytest-bdd-ng fork was created to address these issues and solve some day-to-day problems I encountered while using pytest-bdd. Almost all changes made there are necessary for the correct usage of Cucumber components. A few modules could be dropped. I have supported the original test suite of pytest-bdd and extended it for Cucumber cases. Additionally, step context (and step definitions in agreement with the Cucumber parser) was added. If starting from scratch without context, I would propose a “facelift” for pytest-bdd-ng, adding support for the latest Python and pytest versions, and possibly some CI tools (I have ported all major changes made by @youtux). However, a few issues could hinder this:

• pytest-bdd-ng was not rewritten with speed in mind (some step selections could take longer than in pytest-bdd).
• The API changes are substantial - I kindly ask @youtux to review, try to launch the project/tests, and provide feedback.
  Selection of release/test tools.
• Cucumber parser releases - The latest release I tested (25.0-25.1) was unusable, and I had to revert changes.
• The release process for Cucumber components is quite complex - be prepared to spend time understanding all parts.

[youtux]

The API changes are substantial - I kindly ask @youtux to review, try to launch the project/tests, and provide feedback.
Selection of release/test tools.

When you say that API changes are substantial, do you mean that users would have to change their code deeply in order to run BDD scenarios with pytest?

I'm not sure why would that be necessary, if the only change is the parser, and only users using the pytest-bdd parser (which is undocumented and a internal anyway, so not part of the public API) should be affected

[elchupanebrej]

Users would have to change their code deeply

Let's split users into two categories: writing tests and developing plugins.
For the first category, everything must be passed smoothly (if address T1 only). There are extra features in the Gherkin that are not fully covered by pytest-bdd: steps contexts, data tables, and parametrized steps. Of course, they could be omitted, but the community asks for that features.
For the second category that change won't be complex. Some hooks would be changed, but feel-and-look will be the same

[The-BDD-Coach]

Thank you @elchupanebrej - those estimates are larger than I had hoped, but the information is valuable.

@youtux @kieran-ryan, what are your thoughts about the API changes?

@mattwynne Matt, who is the expert on the Cucumber parser releases? Please invite them into this discussion (or ask them to start a side discussion with Kostya about release 25.1).

[elchupanebrej]

I've tested pytest-bdd-ng against gherkin-official 29.0(https://pypi.org/project/gherkin-official/29.0.0/) yesterday, no obvious defects were found. So no work for now here

[Johnlon]

Hi does the work change the user facing API substantially? Is it possible moving the code to cucumber and then making a sequence of improvements rather than building dependencies?

…

On Sat, 31 Aug 2024, 10:40 pm The-BDD-Coach, ***@***.***> wrote: Thank you @elchupanebrej <https://github.com/elchupanebrej> - those estimates are larger than I had hoped, but the information is valuable. @youtux <https://github.com/youtux> @kieran-ryan <https://github.com/kieran-ryan>, what are your thoughts about the API changes? @mattwynne <https://github.com/mattwynne> Matt, who is the expert on the Cucumber parser releases? Please invite them into this discussion (or ask them to start a side discussion with Kostya about release 25.1). — Reply to this email directly, view it on GitHub <#2202 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAGMFGGKHFFLNEXP272WYDLZUIZ4TAVCNFSM6AAAAABNB22442VHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTANJQHEYTMOI> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[elchupanebrej]

A basic Python implementation for messages was created here: cucumber/messages#165 . This wasn’t an end-to-end implementation in terms of continuous delivery (CD) because I couldn't put such effort to investigate and implement all the necessary parts that time. Instead, I released a package, cuke-messages https://pypi.org/project/cuke-messages/, to demonstrate that pytest-bdd-ng works directly with these models. The community can help by taking the approach used in that repository/PR and integrating it into the Cucumber release process. I can assist with code generation.

• It might be a good idea to regenerate the latest model and use it as-is, then move to CD in the stream with other projects.
• cuke-messages will be yanked just after official release
• pytest-bdd-ng produces correct JSON reports using cuke-messages!

[elchupanebrej]

@The-BDD-Coach cucumber/messages#165 was updated. Please help to address this PR. Also, help with the release part is needed there(polyglot releases?)

[The-BDD-Coach]

Please schedule a Zoom call for the two of us so we can discuss this; I would be glad to help if I can, but need to understand exactly what you are asking me to do. Makefiles are not my area of expertise.

[Johnlon]

I think big breaking changes to the user API equate to this effectively being a new product. obviously new apis for new features is a different matter.

…

On Mon, 2 Sept 2024, 9:20 pm Kostiantyn Goloveshko, < ***@***.***> wrote: Users would have to change their code deeply Let's split users into two categories: writing tests and developing plugins. For the first category, everything must be passed smoothly (if address T1 only). There are extra features in the Gherkin that are not fully covered by pytest-bdd: steps contexts, data tables, and parametrized steps. Of course, they could be omitted, but the community asks for that features. For the second category that change won't be complex. Some hooks would be changed, but feel-and-look will be the same — Reply to this email directly, view it on GitHub <#2202 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAGMFGDIVNO4ALUJHAZ3VPTZUTCARAVCNFSM6AAAAABNB22442VHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTANJSGQ3TQNI> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[Johnlon]

thinking of my own goals..

what id like to see is a well supported and widely adopted native python impl with a feature set equivalent to at least cucumber jvm, including a scheme for concurrency at the scenario/outline level.

so that when I'm switching between jvm to golang to python teams I can expect and promote equivalent capabilities and not have to make apologies.

--+

annoyingly
at present one has to hand crank attachments and also concurrency into behave and so I've encouraged folk to choose golang or jvm if possible but obviously I think we need a native Impl with equivalence to cukejvm/godog (btw to close the gaps between jvm/go/python I added attachments to godog recently, and I also added ambiguous step detection, and I started adding message support but no pr yet for messages).

[The-BDD-Coach]

@youtux @kieran-ryan Whose agreement do we need to get in order to move Pytest-BDD into the official Cucumber project world?

I would like to email everyone to begin a conversation that I think should be taken offline; I don't have email addresses for @youtux or @Johnlon. If you would please email me through TheBDDCoach.com I would greatly appreciate it.

[youtux]

@youtux @kieran-ryan Whose agreement do we need to get in order to move Pytest-BDD into the official Cucumber project world?

Do you mean to move pytest-bdd under the gherkin org, instead of pytest?

I would like to email everyone to begin a conversation that I think should be taken offline; I don't have email addresses for @youtux or @Johnlon. If you would please email me through TheBDDCoach.com I would greatly appreciate it.

We can discuss on discord

[The-BDD-Coach]

@youtux Yes, I am wondering whether we should move Pytest-BDD under the 'gherkin org, instead of pytest'. There are advantages and disadvantages, but that is only a small part of what I want to discuss via email. In this case I believe that email has some advantages over Discord, but you are welcome to DM me on Discord and I can explain why.

[The-BDD-Coach]

I started a thread in the new '#funding' channel; @kieran-ryan, @Johnlon, @blaisep do you have access to that channel?

[youtux]

understood, DM'd you

[The-BDD-Coach]

A few additional questions:

• Who updates Pytest-BDD when the underlying Pytest gets updates?
  • How often does this happen?
• Does Pytest-Dev have a mechanism for funding projects?

Perhaps we should have a short Zoom or Discord video conference to discuss this?

[youtux]

Who updates Pytest-BDD when the underlying Pytest gets updates?

I do, usually every few months, or if there's a bug report.