Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Divide operations by content type #19473

Open
wants to merge 2 commits into
base: master
Choose a base branch
from

Conversation

altro3
Copy link
Contributor

@altro3 altro3 commented Aug 28, 2024

Fixed #17877

OpenAPI allow set to one operation multiple content types for request body, but in main frameworks we need to divide this operation to multiple methods - with the same path, but different request body class and different consumed content type.

Enabled it for Java and Kotlin generators.

See isssue to details: #17877

PR checklist

  • Read the contribution guidelines.
  • Pull Request title clearly describes the work in the pull request and Pull Request description provides details about how to validate the work. Missing information here may result in delayed response from the community.
  • Run the following to build the project and update samples:
    ./mvnw clean package 
    ./bin/generate-samples.sh ./bin/configs/*.yaml
    ./bin/utils/export_docs_generators.sh
    
    (For Windows users, please run the script in Git BASH)
    Commit all changed files.
    This is important, as CI jobs will verify all generator outputs of your HEAD commit as it would merge with master.
    These must match the expectations made by your contribution.
    You may regenerate an individual generator by passing the relevant config(s) as an argument to the script, for example ./bin/generate-samples.sh bin/configs/java*.
    IMPORTANT: Do NOT purge/delete any folders/files (e.g. tests) when regenerating the samples as manually written tests may be removed.
  • File the PR against the correct branch: master (upcoming 7.6.0 minor release - breaking changes with fallbacks), 8.0.x (breaking changes without fallbacks)
  • If your PR is targeting a particular programming language, @mention the technical committee members, so they are more likely to review the pull request.

@altro3
Copy link
Contributor Author

altro3 commented Aug 28, 2024

@wing328 Could you review this PR?

@altro3
Copy link
Contributor Author

altro3 commented Aug 29, 2024

@cachescrubber (2022/02) @welshm (2022/02) @MelleD (2022/02) @atextor (2022/02) @manedev79 (2022/02) @javisst (2022/02) @borsch (2022/02) @banlevente (2022/02) @Zomzog (2022/09) @martin-mfg (2023/08)

@altro3
Copy link
Contributor Author

altro3 commented Sep 5, 2024

@wing328 @cachescrubber (2022/02) @welshm (2022/02) @MelleD (2022/02) @atextor (2022/02) @manedev79 (2022/02) @javisst (2022/02) @borsch (2022/02) @banlevente (2022/02) @Zomzog (2022/09) @martin-mfg (2023/08) ping

@wing328
Copy link
Member

wing328 commented Sep 9, 2024

thanks for the PR. general speaking i like the idea of dividing operations by content type

since this PR updates both default codegen and generator, it may take more time to review

at the first glance, my first feedback is to add more comments (e.g. docstring explaining what the function does, comments above code blocks to help other contributors easily understand what these do)

@altro3 altro3 force-pushed the divide-operations-by-content-type branch 3 times, most recently from 7710ce5 to a3a9fbb Compare September 9, 2024 10:02
@altro3
Copy link
Contributor Author

altro3 commented Sep 10, 2024

@wing328 thanks for the answer!

I had to make changes to the core because there is no way to do this at the level of a custom code generator, because I needed to change the logic of processing operations.

So, the changes are quite simple, what I do:

  1. I check the divideOperationsByContentType flag - by default for generators it is false , so that the logic of all generators that currently exist does not change. Anyone who wants to can set this flag to true and then the operations will be divided by content type.

  2. I go through all the operations and process those with several content types

  3. Next, if we have several operations with different content types, I put all the additional operations in extensions, so that I can then get them into DefaultGenerator

I just did not see a more correct mechanism for how this could be built into the current code

@altro3 altro3 force-pushed the divide-operations-by-content-type branch from a3a9fbb to f9e5320 Compare September 26, 2024 06:57
@martin-mfg
Copy link
Contributor

Hi @altro3, thanks for the PR!

In my tests using the Java generator, I can't disable the new "divideOperationsByContentType" feature. Apparently because the below two new lines always enables it:

GlobalSettings.setProperty(CodegenConstants.DIVIDE_OPERATIONS_BY_CONTENT_TYPE, "true");

GlobalSettings.setProperty(CodegenConstants.DIVIDE_OPERATIONS_BY_CONTENT_TYPE, "true");

It also seems you need to update the samples (following bullet point 3 in the PR checklist) again. Other than that this PR looks good to me. 👍

@slabiakt
Copy link

thanks for the PR! I have a question what about multiple content types for response body? example is here #17877 (comment)
in the test in this MR there are such content types for request body:

       content:
          application/json:
            schema:
              $ref: '#/components/schemas/coordinates'
          application/xml:
            schema:
              $ref: '#/components/schemas/coordinates'

if we define also such content types for response body, I think second function that is generated should have not only different params but also different response type, correct? now it's not working like that

@altro3
Copy link
Contributor Author

altro3 commented Oct 23, 2024

@martin-mfg @slabiakt Hi guys! Sorry for the long answer. Ok, about your questins:

  1. You can't turn it off because it's not a setting, it's just a flag for the generator. The idea was that if any of the generators in other languages ​​needed to split operations into different methods, it would just add this line to its generator and that's it.

I mean that no sane Java or Kotlin developer would turn off this feature, because otherwise the generated code would be incorrect from the point of view of Java or Kotlin frameworks.

So my idea was to not give the user the ability to enable or disable this setting. And this flag is needed exclusively for communication between the DefaultCodegen and DefaultGenerator classes.

  1. @slabiakt So, regarding different responses with different content types: you see, the content type for a request is used in routing within the spring / micronaut framework, i.e. for requests with different content types, the framework itself will be able to determine which method should be called, depending on the incoming request.

As far as I know, the content type of the response does not participate in routing, so the problem is that you have one method that can return responses with different content types. And you can't 100% correctly divide this logic into different methods. Of course, I may be wrong.

In addition, the question arises, how will you correctly divide these methods if you simultaneously have an incoming request that can be xml or json, and at the same time the response can be xml or json?

If you add division by response, then you should have 4 methods generated, not 2. But this is no longer a more correct generation, but an excessive generation. In my opinion, division only by incoming content type is quite sufficient

@altro3 altro3 force-pushed the divide-operations-by-content-type branch from f9e5320 to 127a9ae Compare October 23, 2024 05:39
@slabiakt
Copy link

@altro3 you have header "Accept" which is used for negotiation about response, in spring if you have such two endpoints:

@RestController
@RequestMapping("/api/example")
public class ExampleController {

    @GetMapping(produces = MediaType.APPLICATION_JSON_VALUE)
    public ResponseEntity<MyDto> getJson() {
        MyDto response = new MyDto();
        response.setMessage("JSON response");
        return ResponseEntity.ok(response);
    }

    @GetMapping(produces = MediaType.APPLICATION_XML_VALUE)
    public ResponseEntity<MyDto> getXml() {
        MyDto response = new MyDto();
        response.setMessage("XML response");
        return ResponseEntity.ok(response);
    }
}

depending what you will provide in Accept header, request will hit either getJson or getXml method.
For number of methods to generate, you are correct that it will be 4 methods to handle all possible cases, when you have xml and json in requet content type and xml and json in response content type, but I think no one is giving json in request and asking for xml for response, 90% of users will expect in response what they gave in request so I would generate only methods where request body content type has it's corresponding content-type in response body.

@altro3
Copy link
Contributor Author

altro3 commented Oct 23, 2024

It's just that your solution only covers special cases. I'm not sure you're right about 90%...

In general, the only solution I see for this case is to generate all possible combinations by default and add a setting - to compare the content of the type request and response. It seems to me that such a solution will cover 100% of implementations

@slabiakt
Copy link

yes, I agree that it would be better to just generate all combinations 👍

@altro3 altro3 force-pushed the divide-operations-by-content-type branch from 127a9ae to 92f38a8 Compare October 26, 2024 10:58
@altro3
Copy link
Contributor Author

altro3 commented Oct 26, 2024

@martin-mfg @slabiakt Ok, i've added two new options groupByResponseContentType and groupByRequestAndResponseContentType.

I think these options are enough. Maybe you should think about more readable method naming.

Sample openapi spec:

openapi: 3.0.3
info:
  version: "1"
  title: Multiple Content Types for same request
paths:
  /multiplecontentpath:
    post:
      operationId: myOp
      requestBody:
        content:
          application/json:
            schema:
              $ref: '#/components/schemas/coordinates'
          application/xml:
            schema:
              $ref: '#/components/schemas/coordinates'
          multipart/form-data:
            schema:
              type: object
              properties:
                coordinates:
                  $ref: '#/components/schemas/coordinates'
                file:
                  type: string
                  format: binary
          application/yaml:
            schema:
              $ref: '#/components/schemas/MySchema'
          text/json:
            schema:
              $ref: '#/components/schemas/MySchema'
      responses:
        201:
          description: Successfully created
          headers:
            Location:
              schema:
                type: string
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/coordinates'
            application/xml:
              schema:
                $ref: '#/components/schemas/coordinates'
            text/json:
              schema:
                $ref: '#/components/schemas/MySchema'
components:
  schemas:
    coordinates:
      type: object
      required:
        - lat
        - long
      properties:
        lat:
          type: number
        long:
          type: number
    MySchema:
      type: object
      required:
        - lat
      properties:
        lat:
          type: number

Micronaut samples (the same will be for spring)

  1. groupByRequestAndResponseContentType = true and groupByResponseContentType = true (Default behaviour):
    @Post("/multiplecontentpath")
    Mono<HttpResponse<@Valid Coordinates>> myOp(
        @Body @Nullable @Valid Coordinates coordinates
    );

    @Post("/multiplecontentpath")
    @Consumes("application/xml")
    @Produces("application/xml")
    Mono<HttpResponse<@Valid Coordinates>> myOp_1(
        @Body @Nullable @Valid Coordinates coordinates
    );

    @Post("/multiplecontentpath")
    @Consumes({"application/json", "application/xml", "text/json"})
    @Produces("multipart/form-data")
    Mono<HttpResponse<@Valid Coordinates>> myOp_2(
        @Nullable @Valid Coordinates coordinates,
        @Nullable byte[] file
    );

    @Post("/multiplecontentpath")
    @Consumes("text/json")
    @Produces({"application/yaml", "text/json"})
    Mono<HttpResponse<@Valid MySchema>> myOp_3(
        @Body @Nullable @Valid MySchema mySchema
    );
  1. groupByRequestAndResponseContentType = true and groupByResponseContentType = false
    @Post("/multiplecontentpath")
    Mono<HttpResponse<@Valid Coordinates>> myOp(
        @Body @Nullable @Valid Coordinates coordinates
    );

    @Post("/multiplecontentpath")
    @Consumes("application/xml")
    @Produces("application/xml")
    Mono<HttpResponse<@Valid Coordinates>> myOp_1(
        @Body @Nullable @Valid Coordinates coordinates
    );

    @Post("/multiplecontentpath")
    @Consumes({"application/json", "application/xml", "text/json"})
    @Produces("multipart/form-data")
    Mono<HttpResponse<@Valid Coordinates>> myOp_2(
        @Nullable @Valid Coordinates coordinates,
        @Nullable byte[] file
    );

    @Post("/multiplecontentpath")
    @Consumes({"application/json", "application/xml", "text/json"})
    @Produces("application/yaml")
    Mono<HttpResponse<@Valid Coordinates>> myOp_3(
        @Body @Nullable @Valid MySchema mySchema
    );

    @Post("/multiplecontentpath")
    @Consumes("text/json")
    @Produces("text/json")
    Mono<HttpResponse<@Valid MySchema>> myOp_4(
        @Body @Nullable @Valid MySchema mySchema
    );
  1. groupByRequestAndResponseContentType = false and groupByResponseContentType = true (all combinations, but groupping only by response content-type)
    @Post("/multiplecontentpath")
    @Consumes("application/xml")
    Mono<HttpResponse<@Valid Coordinates>> myOp(
        @Body @Nullable @Valid Coordinates coordinates
    );

    @Post("/multiplecontentpath")
    Mono<HttpResponse<@Valid Coordinates>> myOp_1(
        @Body @Nullable @Valid Coordinates coordinates
    );

    @Post("/multiplecontentpath")
    @Consumes("text/json")
    Mono<HttpResponse<@Valid MySchema>> myOp_2(
        @Body @Nullable @Valid Coordinates coordinates
    );

    @Post("/multiplecontentpath")
    @Consumes("text/json")
    @Produces({"application/yaml", "text/json"})
    Mono<HttpResponse<@Valid MySchema>> myOp_3(
        @Body @Nullable @Valid MySchema mySchema
    );

    @Post("/multiplecontentpath")
    @Consumes("application/xml")
    @Produces("application/xml")
    Mono<HttpResponse<@Valid Coordinates>> myOp_4(
        @Body @Nullable @Valid Coordinates coordinates
    );

    @Post("/multiplecontentpath")
    @Produces("application/xml")
    Mono<HttpResponse<@Valid Coordinates>> myOp_5(
        @Body @Nullable @Valid Coordinates coordinates
    );

    @Post("/multiplecontentpath")
    @Consumes("text/json")
    @Produces("application/xml")
    Mono<HttpResponse<@Valid MySchema>> myOp_6(
        @Body @Nullable @Valid Coordinates coordinates
    );

    @Post("/multiplecontentpath")
    @Consumes("application/xml")
    @Produces("multipart/form-data")
    Mono<HttpResponse<@Valid Coordinates>> myOp_7(
        @Nullable @Valid Coordinates coordinates,
        @Nullable byte[] file
    );

    @Post("/multiplecontentpath")
    @Produces("multipart/form-data")
    Mono<HttpResponse<@Valid Coordinates>> myOp_8(
        @Nullable @Valid Coordinates coordinates,
        @Nullable byte[] file
    );

    @Post("/multiplecontentpath")
    @Consumes("text/json")
    @Produces("multipart/form-data")
    Mono<HttpResponse<@Valid MySchema>> myOp_9(
        @Nullable @Valid Coordinates coordinates,
        @Nullable byte[] file
    );

    @Post("/multiplecontentpath")
    @Consumes("application/xml")
    @Produces({"application/yaml", "text/json"})
    Mono<HttpResponse<@Valid Coordinates>> myOp_10(
        @Body @Nullable @Valid MySchema mySchema
    );

    @Post("/multiplecontentpath")
    @Produces({"application/yaml", "text/json"})
    Mono<HttpResponse<@Valid Coordinates>> myOp_11(
        @Body @Nullable @Valid MySchema mySchema
    );
  1. groupByRequestAndResponseContentType = false and groupByResponseContentType = false (all combinations)
    @Post("/multiplecontentpath")
    @Consumes("application/xml")
    Mono<HttpResponse<@Valid Coordinates>> myOp(
        @Body @Nullable @Valid Coordinates coordinates
    );

    @Post("/multiplecontentpath")
    Mono<HttpResponse<@Valid Coordinates>> myOp_1(
        @Body @Nullable @Valid Coordinates coordinates
    );

    @Post("/multiplecontentpath")
    @Consumes("text/json")
    Mono<HttpResponse<@Valid MySchema>> myOp_2(
        @Body @Nullable @Valid Coordinates coordinates
    );

    @Post("/multiplecontentpath")
    @Consumes("text/json")
    @Produces("application/yaml")
    Mono<HttpResponse<@Valid MySchema>> myOp_3(
        @Body @Nullable @Valid MySchema mySchema
    );

    @Post("/multiplecontentpath")
    @Consumes("application/xml")
    @Produces("text/json")
    Mono<HttpResponse<@Valid Coordinates>> myOp_4(
        @Body @Nullable @Valid MySchema mySchema
    );

    @Post("/multiplecontentpath")
    @Produces("text/json")
    Mono<HttpResponse<@Valid Coordinates>> myOp_5(
        @Body @Nullable @Valid MySchema mySchema
    );

    @Post("/multiplecontentpath")
    @Consumes("text/json")
    @Produces("text/json")
    Mono<HttpResponse<@Valid MySchema>> myOp_6(
        @Body @Nullable @Valid MySchema mySchema
    );

    @Post("/multiplecontentpath")
    @Consumes("application/xml")
    @Produces("application/xml")
    Mono<HttpResponse<@Valid Coordinates>> myOp_7(
        @Body @Nullable @Valid Coordinates coordinates
    );

    @Post("/multiplecontentpath")
    @Produces("application/xml")
    Mono<HttpResponse<@Valid Coordinates>> myOp_8(
        @Body @Nullable @Valid Coordinates coordinates
    );

    @Post("/multiplecontentpath")
    @Consumes("text/json")
    @Produces("application/xml")
    Mono<HttpResponse<@Valid MySchema>> myOp_9(
        @Body @Nullable @Valid Coordinates coordinates
    );

    @Post("/multiplecontentpath")
    @Consumes("application/xml")
    @Produces("multipart/form-data")
    Mono<HttpResponse<@Valid Coordinates>> myOp_10(
        @Nullable @Valid Coordinates coordinates,
        @Nullable byte[] file
    );

    @Post("/multiplecontentpath")
    @Produces("multipart/form-data")
    Mono<HttpResponse<@Valid Coordinates>> myOp_11(
        @Nullable @Valid Coordinates coordinates,
        @Nullable byte[] file
    );

    @Post("/multiplecontentpath")
    @Consumes("text/json")
    @Produces("multipart/form-data")
    Mono<HttpResponse<@Valid MySchema>> myOp_12(
        @Nullable @Valid Coordinates coordinates,
        @Nullable byte[] file
    );

    @Post("/multiplecontentpath")
    @Consumes("application/xml")
    @Produces("application/yaml")
    Mono<HttpResponse<@Valid Coordinates>> myOp_13(
        @Body @Nullable @Valid MySchema mySchema
    );

    @Post("/multiplecontentpath")
    @Produces("application/yaml")
    Mono<HttpResponse<@Valid Coordinates>> myOp_14(
        @Body @Nullable @Valid MySchema mySchema
    );

PS Where @Consumes or @Produces annotaions lost - that means using single default content-type - apllication/json.

@altro3 altro3 force-pushed the divide-operations-by-content-type branch from 92f38a8 to 6ddd97b Compare October 26, 2024 12:22
var codegen = new JavaMicronautClientCodegen();
String outputPath = generateFiles(codegen, "src/test/resources/3_0/java/multiple-content-types.yaml", CodegenConstants.APIS, CodegenConstants.MODELS);

// Micronaut declarative http client should use the provided path separator
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This comment is probably copied from elsewhere and doesn't apply here, right?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed

var codegen = new JavaMicronautServerCodegen();
String outputPath = generateFiles(codegen, "src/test/resources/3_0/java/multiple-content-types.yaml", CodegenConstants.APIS, CodegenConstants.MODELS);

// Micronaut declarative http client should use the provided path separator
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This comment is probably copied from elsewhere and doesn't apply here, right?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed

@@ -193,6 +194,8 @@ public abstract class AbstractJavaCodegen extends DefaultCodegen implements Code
public AbstractJavaCodegen() {
super();

GlobalSettings.setProperty(CodegenConstants.DIVIDE_OPERATIONS_BY_CONTENT_TYPE, "true");
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So my idea was to not give the user the ability to enable or disable this setting. And this flag is needed exclusively for communication between the DefaultCodegen and DefaultGenerator classes.

I see, that makes sense. In this case, please don't use GlobalSettings though, as it is not really intended for such use cases and would cause problems. Instead, please follow the example of isTypeErasedGenerics(). I.e. add a new method (e.g. named "supportsDividingOperationsByContentType") to CodegenConfig, make it return false in DefaultCodegen and make it return true in AbstractJavaCodegen and AbstractKotlinCodegen. In DefaultGenerator you should then be able to retrieve the appropriate value via config.supportsDividingOperationsByContentType().

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed

cliOptions.add(CliOption.newBoolean(CodegenConstants.GROUP_BY_RESPONSE_CONTENT_TYPE,
CodegenConstants.GROUP_BY_RESPONSE_CONTENT_TYPE_DESC).defaultValue(Boolean.TRUE.toString()));
cliOptions.add(CliOption.newBoolean(CodegenConstants.GROUP_BY_REQUEST_AND_RESPONSE_CONTENT_TYPE,
CodegenConstants.GROUP_BY_REQUEST_AND_RESPONSE_CONTENT_TYPE_DESC).defaultValue(Boolean.TRUE.toString()));
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since these options are only relevant for Generators which support your new feature, could you put an if around these 4 lines here and execute them only of the currently selected Generator supports the feature? (i.e. if supportsDividingOperationsByContentType() mentioned in my other comment is true)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed

}
var firstEntry = content.entrySet().iterator().next();
var mediaTypesToRemove = new ArrayList<String>();
var withoutContent = false;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This variable can be removed, together with lines 1230-1232.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed

+ "For example, when openapi operation produces one of \"application/json\" and \"application/xml\" content types "
+ "will be generated only one method for both content types. Otherwise for each content type will be generated different method.";

public static final String GROUP_BY_REQUEST_AND_RESPONSE_CONTENT_TYPE = "groupByRequestAndResponseContentType";
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you please add a unit test (or sample config) for the two new options groupByResponseContentType and groupByRequestAndResponseContentType?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed

@@ -487,4 +486,46 @@ public void doRepeatOperationForAllTags() {
.hasAnnotation("JacksonXmlProperty", Map.of("localName", "\"item\""))
.hasAnnotation("JacksonXmlElementWrapper", Map.of("localName", "\"activities-array\""));
}

@Test
public void testMultipleContentTypesToPath() {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The new unit test here and in JavaMicronautClientCodegenTest both fail with a NullPointerException when I run them. Is it different for you?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed

@altro3
Copy link
Contributor Author

altro3 commented Oct 27, 2024

@martin-mfg Yes, of course, I will fix everything and adjust the tests. I just wanted to show the logic of dividing the operation into different methods and a proposal for new settings. If everyone is satisfied with the logic, then I can start testing and correct your comments.

@altro3 altro3 force-pushed the divide-operations-by-content-type branch 3 times, most recently from 4a9d594 to e9040a0 Compare October 27, 2024 09:48
@altro3 altro3 force-pushed the divide-operations-by-content-type branch from e9040a0 to 5a00a57 Compare October 27, 2024 09:57
@altro3
Copy link
Contributor Author

altro3 commented Oct 28, 2024

I think, I need to rework my solution. It was designed only for splitting by request body. but splitting by response body complicated the logic. The current solution is not very effective, I have already come up with a new solution. but I need to find free time to implement it. So for now do not merge my changes

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
4 participants