Expose all HAR "entry" fields in each corresponding Task2 instance

[xinke2411]

Add plugin to filter entries in HAR files

Save HAR from chrome will save all requests to HAR.
Sometimes locustfile is not necessary include static files requests(js, css) and cached requests, add plugin to allow user filter entries.

[tortila]

Hey @xinke2411, have you seen our blacklisting mechanism? Using it you can define which HAR entries should be ignored when translating to Locustfile. Can you try it out and see whether it solves your use case?

[xinke2411]

@tortila Thanks for your suggestion. This is part of HAR content is i recorded from chrome developer tools, some requests has attribute _fromCache if it be cached. I don't want this cached requests in Locustfile. I think is a better way to add a plugin function allow customer filter requests from HAR.

{
  "log": {
    "version": "1.2",
    "creator": {
      "name": "WebInspector",
      "version": "537.36"
    },
    "pages": [
      {
        }
      }
    ],
    "entries": [
      {
        "startedDateTime": "2019-03-06T12:17:05.060Z",
        "time": 1085.550000000012,
        "request": {
          "method": "GET",
          "url": "https://www.google.com/",
          "httpVersion": "http/2.0",
          "headers": [
          ],
          "queryString": [],
          "cookies": [
          ],
          "headersSize": -1,
          "bodySize": 0
        },
        "response": {
        },
        "cache": {},
        "timings": {
          "blocked": 3.176000000082975,
          "dns": -1,
          "ssl": -1,
          "connect": -1,
          "send": 0.18399999999999972,
          "wait": 450.8650000000957,
          "receive": 631.3249999998334,
          "_blocked_queueing": 1.318000000082975,
          "_blocked_proxy": 0.85
        },
        "serverIPAddress": "61.185.200.11",
        "_initiator": {
          "type": "other"
        },
        "_priority": "VeryHigh",
        "_resourceType": "document",
        "connection": "12725",
        "pageref": "page_2"
      },
      {
        "startedDateTime": "2019-03-06T12:17:05.960Z",
        "time": 0.03399999991415825,
        "request": {
          "method": "GET",
          "url": "https://www.google.com/images/branding/googlelogo/1x/googlelogo_color_272x92dp.png",
          "httpVersion": "http/2.0",
          "headers": [],
          "queryString": [],
          "cookies": [],
          "headersSize": -1,
          "bodySize": 0
        },
        "response": {
        },
        "cache": {},
        "timings": {
          "blocked": -1,
          "dns": -1,
          "ssl": -1,
          "connect": -1,
          "send": 0,
          "wait": 0.029000000040468876,
          "receive": 0.004999999873689376,
          "_blocked_queueing": -1
        },
        "serverIPAddress": "61.185.200.11",
        "_initiator": {
          "type": "parser",
          "url": "https://www.google.com/",
          "lineNumber": 63
        },
        "_priority": "VeryLow",
        "_resourceType": "image",
        "_fromCache": "memory",
        "connection": "12725",
        "pageref": "page_2"
      }
      
    ]
  }
}

[tortila]

Thank you for clarification, I think it's a valid concern. Until it's fixed on the Transformer side, what you can do to work around it is to start recording in a browser without any cookies, e.g. "Guest window" in Chrome.

[thilp]

As discussed together with @tortila, our current understanding is:

• Today, it is not possible to write a Transformer plugin filtering tasks based on arbitrary fields of the corresponding log.entries[*] HAR object (like _fromCache in @xinke2411's example), because that information is not exposed by the Request or Task2 classes.
• If that information was exposed by the Task2 class, writing a plugin to remove _fromCache requests would be quite easy: an OnScenario plugin can easily filter Scenario.children depending on the data newly exposed.
• We should expose all entry fields read from the HAR, including the ones already extracted in Request. This would give access to the original HAR entry, so people don't need to add back request.url or any other field you would usually find in a HAR entry.
• We must make it very clear that this additional data is exposed in a read-only fashion: any changes to that data would not be reflected anywhere else in Transformer.
• As a follow-up in Provide helper methods for Transformer plugins #24, we should add a OnScenario helper to simplify filtering tasks on arbitrary HAR fields. Nice to have, but not required for this issue.

[thilp]

I will take the liberty of renaming this issue to better reflect the work that needs to be done. Feel free to edit too.

[tortila]

Hey @xinke2411, new version of Transformer: 1.1.2 was just published and it should contain the changes you requested. Here's a relevant piece of the documentation: https://transformer.readthedocs.io/en/latest/dev.html#transformer.request.Request.har_entry. Let us know if you spot any issues with that or anything else in the project. Happy coding 🤓