Form variables

[jeremicmilan]

Form variables are a feature to configure the variables you want to be forwarded from scraping the form-submit page to scraping the main page for sensor data. A common use case is to populate the X-Login-Token header which is the result of the login.

Example:

multiscrape:
  - resource: "https://website-api.airvisual.com/v1/users/65a28a0cec1ff309a74ba414/devices/avo_65a2bd77c2f7aeabcd715393?units.system=metric&AQI=US&language=en"
    form_submit:
      submit_once: True
      resource: "https://website-api.airvisual.com/v1/auth/signin/by/email"
      input:
        email: "<email>"
        password: "<password>"
      variables:
        - name: token
          value_template: "{{ (value | from_json).loginToken }}"
    headers:
      X-Login-Token: "{{ token }}"
    sensor:
      - name: AirVisual Outdoor AQI
        value_template: "{{ (value | from_json).current.aqi.value }}"
        unit_of_measurement: "AQI US"

Log into https://dashboard.iqair.com/personal/devices, select the device to get the <device_id> in the URL. After that analyze network traffic and find the name starting with <device_id>. That will contain the entire path in the example including <user_id> (there's probably an easier way to get <user_id>, but this works),

[jeremicmilan]

This is continuation of #327. Please refer to it for history.

@danieldotnl, please review.

[jeremicmilan]

@danieldotnl, debugging the extension does not work for me. It worked when I initially worked on the first PR (header mappings). Does it work for you?

My setup is WSL, I enable debugpy and attach locally. The debugger attaches, but it refuses to hit breakpoints (where before it did). Very weird. Do you know what's going on?
BTW, I implemented the form variables without the debugger, it was a little bit annoying. :D

[danieldotnl]

Very nice that you picked this up and implemented variables! Thank you!

[jeremicmilan]

@danieldotnl, it took a while to get some time for this, but I finally addressed all the feedback. Please review again.

[danieldotnl]

It's taking some time but we are getting close!!

[danieldotnl]

@danieldotnl, fun trivia, the token that I started using months ago did not yet expire. It usually should expire in 90 days. So, I did not need to update manually the token on my production home assistant installation. 😄

Good for you as this took way longer than 90 days 😅

[danieldotnl]

Let's try to finish this soon now, as people are waiting for the cookies PR to be merged, but I don't want to bother you with merging that one as well in this PR. So I would like to merge you PR first.

[jeremicmilan]

@danieldotnl , I'll have a lot more time in July. Probably won't make it before that.

[jeremicmilan]

@danieldotnl, I addressed comments and I'm waiting for your guidance on some of them. Please review.

Also, I would love to add tests for these cases, so as they do not get broken over time. I tried adding a test and did not succeed after half an hour. I saw you doing stuff in that space recently. Do you plan to add tests (examples I could follow)?

[danieldotnl]

@danieldotnl, I addressed comments and I'm waiting for your guidance on some of them. Please review.

Also, I would love to add tests for these cases, so as they do not get broken over time. I tried adding a test and did not succeed after half an hour. I saw you doing stuff in that space recently. Do you plan to add tests (examples I could follow)?

Yes, I would love to add tests to multiscrape as well, but time is really limited for me at the moment. So let's not wait for that before merging :)

[jeremicmilan]

@danieldotnl, I did final checks, and I'm fine for this to be merged (and of course, I'll add tests once you add some initial tests). So, please review, I'm happy addressing more issues (if any). :)
When merging, I suggest doing a squash merge (not sure how to specify that on GitHub, as this is my first PR here), as I have a bunch of small commits which would only pollute your main branch,

[jeremicmilan]

@danieldotnl, can we finish the PR this week? From next week on, I'm again very tight on time (and it might be weeks before I'm able to address comments).

[danieldotnl]

Jeremic, I was on vacation but good news! I tested the PR again and I'm happy with the changes! So I merged it and create a pre-release. Big thanks to you!!
Please install it in your production system and test it again.

There is one warning we still need to fix about a parser which is not specified. I think it's not provided to the form scraper as a parameter.

[jeremicmilan]

@danieldotnl, thanks for merging and thanks for support on the PR!

I tested on my production server, and everything works great!

The only minor issue I have is that I need to scrape two different URLs that require the same form submit. As both logins go at the same time one of them is rejected by the server. The next retry starts working correctly. But... This is something I can live with. :)

What is the warning you are referring to? I'm not seeing it...

[danieldotnl]

@jeremicmilan I'm planning to release this in a non-prerelease soon. Is it still working fine for you?
By the way, I fixed debugging and added the first automated tests!!

PS: You can find me also on discord if you want to chat. (instead of via this PR)

[jeremicmilan]

Yep, works great. Feel free to publish.

I added a note to myself regarding adding tests for the changes I made. But I'm unsure when I'll have time to do this.

Sure, I'll ping you once I start writing tests.