Skip to content

Commit

Permalink
Merging to release-5.7: [TT-13021]Transfer encoding fix (#6770) (#6782)
Browse files Browse the repository at this point in the history 
### **User description**
[TT-13021]Transfer encoding fix (#6770)

### **User description**
<!-- Provide a general summary of your changes in the Title above -->

TASK: https://tyktech.atlassian.net/browse/TT-13021

## Description

<!-- Describe your changes in detail -->

## Related Issue

<!-- This project only accepts pull requests related to open issues. -->
<!-- If suggesting a new feature or change, please discuss it in an
issue first. -->
<!-- If fixing a bug, there should be an issue describing it with steps
to reproduce. -->
<!-- OSS: Please link to the issue here. Tyk: please create/link the
JIRA ticket. -->

## Motivation and Context

<!-- Why is this change required? What problem does it solve? -->

## How This Has Been Tested

<!-- Please describe in detail how you tested your changes -->
<!-- Include details of your testing environment, and the tests -->
<!-- you ran to see how your change affects other areas of the code,
etc. -->
<!-- This information is helpful for reviewers and QA. -->

## Screenshots (if appropriate)

## Types of changes

<!-- What types of changes does your code introduce? Put an `x` in all
the boxes that apply: -->

- [ ] Bug fix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Refactoring or add test (improvements in base code or adds test
coverage to functionality)

## Checklist

<!-- Go over all the following points, and put an `x` in all the boxes
that apply -->
<!-- If there are no documentation updates required, mark the item as
checked. -->
<!-- Raise up any additional concerns not covered by the checklist. -->

- [ ] I ensured that the documentation is up to date
- [ ] I explained why this PR updates go.mod in detail with reasoning
why it's required
- [ ] I would like a code coverage CI quality gate exception and have
explained why


___

### **PR Type**
Bug fix, Enhancement


___

### **Description**
- Fixed an issue with reading and resetting the request body in the URL
rewrite middleware to ensure downstream handlers can process it
correctly.
- Enhanced regex matching logic in the URL rewrite middleware, including
improved context data handling.
- Extended the tracing functionality to support OAS definitions by
adding a new field in the trace request structure and implementing
extraction logic.
- Improved error handling for malformed or incomplete trace requests.



___



### **Changes walkthrough** 📝
<table><thead><tr><th></th><th align="left">Relevant
files</th></tr></thead><tbody><tr><td><strong>Bug
fix</strong></td><td><table>
<tr>
  <td>
    <details>
<summary><strong>mw_url_rewrite.go</strong><dd><code>Improve request
body handling and regex matching in URL rewrite
</code><br><code>middleware</code></dd></summary>
<hr>

gateway/mw_url_rewrite.go

<li>Added error handling for reading the request body.<br> <li> Reset
the request body to allow downstream handlers to read it.<br> <li>
Enhanced regex matching logic and updated context data handling.<br>


</details>


  </td>
<td><a

href="https://github.com/TykTechnologies/tyk/pull/6770/files#diff-84a6a5c810334aaa8702669f2aebf0284f116d83e8a55ec9d1d5b8bae87f1be6">+20/-1</a>&nbsp;
&nbsp; </td>

</tr>
</table></td></tr><tr><td><strong>Enhancement</strong></td><td><table>
<tr>
  <td>
    <details>
<summary><strong>tracing.go</strong><dd><code>Extend tracing
functionality to support OAS definitions</code>&nbsp; &nbsp;
</dd></summary>
<hr>

gateway/tracing.go

<li>Extended trace request structure to include OAS definitions.<br>
<li> Added logic to extract and log OAS definitions into API
definitions.<br> <li> Improved error handling for missing or malformed
trace requests.<br>


</details>


  </td>
<td><a

href="https://github.com/TykTechnologies/tyk/pull/6770/files#diff-0069987d730b02812808925a17e1434ca7558a4dfc8661beb27ccd11afb8c77d">+8/-2</a>&nbsp;
&nbsp; &nbsp; </td>

</tr>
</table></td></tr></tr></tbody></table>

___

> 💡 **PR-Agent usage**: Comment `/help "your question"` on any pull
request to receive relevant information

---------

Co-authored-by: lghiur <[email protected]>

[TT-13021]:
https://tyktech.atlassian.net/browse/TT-13021?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ


___

### **PR Type**
Bug fix, Enhancement


___

### **Description**
- Fixed an issue with reading and resetting the request body in the URL
rewrite middleware to ensure downstream handlers can process it
correctly.
- Enhanced regex matching logic in the URL rewrite middleware, including
improved context data handling.
- Added test cases to validate payload trigger functionality and ensured
reading payload for triggers does not break the request.
- Updated JSON validation logic to improve request body handling by
replacing `ioutil.ReadAll` with `io.ReadAll`.
- Extended the tracing functionality to support OAS definitions by
adding a new field in the trace request structure and implementing
extraction logic.
- Improved error handling for malformed or incomplete trace requests.



___



### **Changes walkthrough** 📝
<table><thead><tr><th></th><th align="left">Relevant
files</th></tr></thead><tbody><tr><td><strong>Bug
fix</strong></td><td><table>
<tr>
  <td>
    <details>
<summary><strong>mw_url_rewrite.go</strong><dd><code>Improve request
body handling and regex matching in URL rewrite
</code><br><code>middleware</code></dd></summary>
<hr>

gateway/mw_url_rewrite.go

<li>Improved error handling for reading the request body.<br> <li> Reset
the request body to allow downstream handlers to read it.<br> <li>
Enhanced regex matching logic and updated context data handling.<br>
<li> Updated context data with modified map after regex operations.<br>


</details>


  </td>
<td><a
href="https://github.com/TykTechnologies/tyk/pull/6782/files#diff-84a6a5c810334aaa8702669f2aebf0284f116d83e8a55ec9d1d5b8bae87f1be6">+18/-2</a>&nbsp;
&nbsp; </td>

</tr>

<tr>
  <td>
    <details>
<summary><strong>mw_validate_json.go</strong><dd><code>Update JSON
validation to improve request body handling</code>&nbsp; &nbsp;
</dd></summary>
<hr>

gateway/mw_validate_json.go

<li>Replaced <code>ioutil.ReadAll</code> with <code>io.ReadAll</code>
for reading request body.<br> <li> Added
<code>nopCloseRequestBody</code> to handle request body reset.<br>


</details>


  </td>
<td><a
href="https://github.com/TykTechnologies/tyk/pull/6782/files#diff-0f0c6b9ac40c5e01908a5b24b1d03111c8d8b4dbc1ddc0251d17c3c1b5328ab5">+3/-2</a>&nbsp;
&nbsp; &nbsp; </td>

</tr>
</table></td></tr><tr><td><strong>Tests</strong></td><td><table>
<tr>
  <td>
    <details>
<summary><strong>mw_url_rewrite_test.go</strong><dd><code>Add tests for
payload triggers and URL rewrite functionality</code></dd></summary>
<hr>

gateway/mw_url_rewrite_test.go

<li>Added test cases to validate payload trigger functionality.<br> <li>
Ensured reading payload for triggers does not break the request.<br>
<li> Enhanced test coverage for URL rewrite triggers.<br>


</details>


  </td>
<td><a
href="https://github.com/TykTechnologies/tyk/pull/6782/files#diff-b5d21238a0fb61d153d824ded42acf4d4a553e962ca8fa015bbff31027b428a5">+48/-5</a>&nbsp;
&nbsp; </td>

</tr>
</table></td></tr><tr><td><strong>Enhancement</strong></td><td><table>
<tr>
  <td>
    <details>
<summary><strong>tracing.go</strong><dd><code>Extend tracing
functionality to support OAS definitions</code>&nbsp; &nbsp;
</dd></summary>
<hr>

gateway/tracing.go

<li>Extended trace request structure to include OAS definitions.<br>
<li> Added logic to extract and log OAS definitions into API
definitions.<br> <li> Improved error handling for malformed trace
requests.<br>


</details>


  </td>
<td><a
href="https://github.com/TykTechnologies/tyk/pull/6782/files#diff-0069987d730b02812808925a17e1434ca7558a4dfc8661beb27ccd11afb8c77d"></a></td>

</tr>
</table></td></tr></tr></tbody></table>

___

> 💡 **PR-Agent usage**: Comment `/help "your question"` on any pull
request to receive relevant information

Co-authored-by: andrei-tyk <[email protected]>
  • Loading branch information
@buger @andrei-tyk
buger and andrei-tyk authored Dec 17, 2024
1 parent 2d1a8fc commit efa9ab2
Show file tree
Hide file tree
Showing 3 changed files with 69 additions and 9 deletions.
20 changes: 18 additions & 2 deletions gateway/mw_url_rewrite.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@ package gateway

import (
"fmt"
"io/ioutil"
"io"
"net/http"
"net/textproto"
"net/url"
Expand Down Expand Up @@ -696,22 +696,38 @@ func checkContextTrigger(r *http.Request, options map[string]apidef.StringRegexM

func checkPayload(r *http.Request, options apidef.StringRegexMap, triggernum int) bool {
contextData := ctxGetData(r)
bodyBytes, _ := ioutil.ReadAll(r.Body)

nopCloseRequestBody(r)
// Read the entire request body
bodyBytes, err := io.ReadAll(r.Body)
if err != nil {
log.WithError(err).Error("error reading request body")
return false
}

// Perform regex matching on the request body
matched, matches := options.FindAllStringSubmatch(string(bodyBytes), -1)

if matched {
kn := buildTriggerKey(triggernum, "payload")

if len(matches) == 0 {
// If there are no matches, simply return true
return true
}

// Store the first match in the context data
contextData[kn] = matches[0][0]

// Iterate over all matches and add them to the context data
for i, match := range matches {
if len(match) > 0 {
addMatchToContextData(contextData, match, triggernum, "payload", i)
}
}

// Update the context data with the modified map
ctxSetData(r, contextData)
return true
}

Expand Down
53 changes: 48 additions & 5 deletions gateway/mw_url_rewrite_test.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@ package gateway

import (
"bytes"
"io"
"net/http"
"net/http/httptest"
"testing"
Expand Down Expand Up @@ -156,11 +157,12 @@ func BenchmarkRewriter(b *testing.B) {

func TestRewriterTriggers(t *testing.T) {
type TestDef struct {
name string
pattern, to string
in, want string
triggerConf []apidef.RoutingTrigger
req *http.Request
name string
pattern, to string
in, want string
triggerConf []apidef.RoutingTrigger
req *http.Request
payloadTrigger bool
}

ts := StartTest(nil)
Expand Down Expand Up @@ -192,6 +194,7 @@ func TestRewriterTriggers(t *testing.T) {
},
},
r,
false,
}
},
func() TestDef {
Expand All @@ -218,6 +221,7 @@ func TestRewriterTriggers(t *testing.T) {
},
},
r,
false,
}
},
func() TestDef {
Expand Down Expand Up @@ -248,6 +252,7 @@ func TestRewriterTriggers(t *testing.T) {
},
},
r,
false,
}
},
func() TestDef {
Expand Down Expand Up @@ -278,6 +283,7 @@ func TestRewriterTriggers(t *testing.T) {
},
},
r,
false,
}
},
func() TestDef {
Expand Down Expand Up @@ -308,6 +314,7 @@ func TestRewriterTriggers(t *testing.T) {
},
},
r,
false,
}
},
func() TestDef {
Expand Down Expand Up @@ -353,6 +360,7 @@ func TestRewriterTriggers(t *testing.T) {
},
},
r,
false,
}
},
func() TestDef {
Expand Down Expand Up @@ -383,6 +391,7 @@ func TestRewriterTriggers(t *testing.T) {
},
},
r,
false,
}
},
func() TestDef {
Expand Down Expand Up @@ -413,6 +422,7 @@ func TestRewriterTriggers(t *testing.T) {
},
},
r,
false,
}
},
func() TestDef {
Expand All @@ -437,6 +447,7 @@ func TestRewriterTriggers(t *testing.T) {
},
},
r,
false,
}
},
func() TestDef {
Expand All @@ -461,6 +472,7 @@ func TestRewriterTriggers(t *testing.T) {
},
},
r,
false,
}
},
func() TestDef {
Expand Down Expand Up @@ -488,6 +500,7 @@ func TestRewriterTriggers(t *testing.T) {
},
},
r,
false,
}
},
func() TestDef {
Expand Down Expand Up @@ -518,6 +531,7 @@ func TestRewriterTriggers(t *testing.T) {
},
},
r,
false,
}
},
func() TestDef {
Expand Down Expand Up @@ -548,6 +562,7 @@ func TestRewriterTriggers(t *testing.T) {
},
},
r,
false,
}
},
func() TestDef {
Expand Down Expand Up @@ -582,6 +597,7 @@ func TestRewriterTriggers(t *testing.T) {
},
},
r,
false,
}
},
func() TestDef {
Expand Down Expand Up @@ -616,6 +632,7 @@ func TestRewriterTriggers(t *testing.T) {
},
},
r,
false,
}
},
func() TestDef {
Expand Down Expand Up @@ -650,6 +667,7 @@ func TestRewriterTriggers(t *testing.T) {
},
},
r,
false,
}
},
func() TestDef {
Expand All @@ -673,6 +691,7 @@ func TestRewriterTriggers(t *testing.T) {
},
},
r,
true,
}
},
func() TestDef {
Expand All @@ -696,6 +715,7 @@ func TestRewriterTriggers(t *testing.T) {
},
},
r,
true,
}
},
func() TestDef {
Expand All @@ -719,6 +739,7 @@ func TestRewriterTriggers(t *testing.T) {
},
},
r,
true,
}
},
func() TestDef {
Expand All @@ -742,6 +763,7 @@ func TestRewriterTriggers(t *testing.T) {
},
},
r,
true,
}
},
func() TestDef {
Expand Down Expand Up @@ -771,6 +793,7 @@ func TestRewriterTriggers(t *testing.T) {
},
},
r,
true,
}
},
func() TestDef {
Expand Down Expand Up @@ -800,6 +823,7 @@ func TestRewriterTriggers(t *testing.T) {
},
},
r,
true,
}
},
func() TestDef {
Expand All @@ -823,6 +847,7 @@ func TestRewriterTriggers(t *testing.T) {
},
},
r,
false,
}
},
func() TestDef {
Expand Down Expand Up @@ -852,6 +877,7 @@ func TestRewriterTriggers(t *testing.T) {
},
},
r,
true,
}
},
func() TestDef {
Expand Down Expand Up @@ -881,6 +907,7 @@ func TestRewriterTriggers(t *testing.T) {
},
},
r,
true,
}
},
func() TestDef {
Expand All @@ -904,6 +931,7 @@ func TestRewriterTriggers(t *testing.T) {
},
},
r,
false,
}
},
func() TestDef {
Expand All @@ -927,6 +955,7 @@ func TestRewriterTriggers(t *testing.T) {
},
},
r,
false,
}
},
func() TestDef {
Expand Down Expand Up @@ -956,6 +985,7 @@ func TestRewriterTriggers(t *testing.T) {
},
},
r,
false,
}
},
func() TestDef {
Expand Down Expand Up @@ -985,6 +1015,7 @@ func TestRewriterTriggers(t *testing.T) {
},
},
r,
false,
}
},
func() TestDef {
Expand Down Expand Up @@ -1015,6 +1046,7 @@ func TestRewriterTriggers(t *testing.T) {
},
},
r,
false,
}
},
func() TestDef {
Expand Down Expand Up @@ -1042,6 +1074,7 @@ func TestRewriterTriggers(t *testing.T) {
},
},
r,
false,
}
},
func() TestDef {
Expand Down Expand Up @@ -1071,6 +1104,7 @@ func TestRewriterTriggers(t *testing.T) {
},
},
r,
false,
}
},
func() TestDef {
Expand All @@ -1094,6 +1128,7 @@ func TestRewriterTriggers(t *testing.T) {
},
},
r,
false,
}
},
}
Expand All @@ -1111,6 +1146,14 @@ func TestRewriterTriggers(t *testing.T) {
if err != nil {
t.Error("compile failed:", err)
}

//added check to ensure that reading the payload to check for the trigger does not break the request
if tc.payloadTrigger {
body, err := io.ReadAll(tc.req.Body)
assert.NotEqual(t, "", string(body))
assert.NoError(t, err)
}

if got != tc.want {
t.Errorf("rewrite failed, want %q, got %q", tc.want, got)
}
Expand Down
5 changes: 3 additions & 2 deletions gateway/mw_validate_json.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ package gateway
import (
"errors"
"fmt"
"io/ioutil"
"io"
"net/http"

"github.com/TykTechnologies/gojsonschema"
Expand Down Expand Up @@ -48,8 +48,9 @@ func (k *ValidateJSON) ProcessRequest(w http.ResponseWriter, r *http.Request, _
}
}

nopCloseRequestBody(r)
// Load input body into gojsonschema
bodyBytes, err := ioutil.ReadAll(r.Body)
bodyBytes, err := io.ReadAll(r.Body)
if err != nil {
return err, http.StatusBadRequest
}
Expand Down

0 comments on commit efa9ab2

Please sign in to comment.