Sems Portal Nightly Midnight Zero is double dipping upsetting HA Recorder

[davidsonimagerygmailcom]

Hi Guys, Firstly hello and thanks for writing this great API. It's been at the heart of dashboards we have running on wall mounted displays in our home.
I appreciate this is likely a SemsPortal issue not this HACS integration, but wanted to pick brains.
So over Christmas break noticed that figures for exported and imported kwh weren't updating at midnight at all.
That seemingly got resolved, but now find once it resets, a minute later sems portal reverts to the previous days reading ... for roughly 7 minutes, before zero'ing a second time. This of course causes Home Assistant to see fresh kwh amounts of consumption (both imported and exported power) ... and adds phantom data to the recorder. Screenshots attached.
As a work around, is there a sensible way to get this integration to simply **not ** perform any queries between 00:00 and 00.15? (ie if it doesn't do a query during these times, it never gets exposed to the transient dodgy data?
I did fire an email off to GoodWe support but I'm not sure it is reaching the team whom actually administrate the accounting part of semsportal.

[davidsonimagerygmailcom]

So sorry for the edits, git hub "post" button wasn't functonal (known issue with github) and I was trying to get post to work with less text (which worked).
Screenies ;

[skreza]

I'm also seeing this exact same issue since 28th Dec.
Agree with @davidsonimagerygmailcom that it's likely an issue on the SEMS portal side, maybe they need to configure NTP on their servers?

Unfortunately I can't see a way to use an automation to disable the SEMS service for a period of time, as that would have been a quick workaround within HA. Only other thing I can thing of as a workaround for now is to create a new template sensor in HA that copies the value of the original sensor from this integration and stop this sensor from updating between 00:00 - 00:15, you would then have to add this sensor to the energy dashboard for it to reflect correctly. Hardly ideal but would achieve the desired outcome!

Any help from @TimSoethout would be greatly appreciated!

[davidsonimagerygmailcom]

As a workaround I've done a few things, as yet I am not 100% sure if it's sorted it (and its only a bandaid).

So the issue as ew know is the double dip ... the import kwh and export kwh all go to zero around midnight, then dip back to some form of the previous days numbers but not exactly (all of which is data coming from Sems Portal, yet doens't show up in the reports you can run from sems portal CSV export) . SemsPortal accounting bug of some sort which came in as soon as they fixed the bigger bug which was KWH metrics weren't resetting at all (for days, even on semsportal).

So the issue is HA seeing that zero'ing, followed by data - and thinks its new usage (hooray for generating power from the moon right??).

I did initially try using an automation that does a Utlility meter reset (as pictured below) ... but even so the "consumed" power would still show (and util meter calibrate doesnt apply to that).

So I went a step further and actually turned off the recorder from 11.59pm to 1230am, my idea being not recording any energy data at all therefore it disregards what it sees. In theory turning off the recorder should actually do the whole trick but i added in the zero'ing automations as a bit of a scatter-gun approach.

If this doesnt work, another idea I had was I might just disallow HA from being able to talk to the internet between midniht ad 1230 ... so it never gets exposed to the data from the double dip (it's back to zero by 1215 usually).

I've not yet confirmed if this actually works (before anyone goes following down this line).

[skreza]

Thanks for the update @davidsonimagerygmailcom

I've come up with a similar workaround for now and will update tomorrow if it has helped.
Essentially I've disabled update polling on the integration itself and setup an automation to update the entities every minute between 00:15:00 through to 00:00:30 which hopefully gets the 0kWh at midnight but skips everything between 12:01AM to 12:15AM

YAML for the automation below, you must have disabled polling on the integration first (Open integration, click 3 dot menu, system options, turn off 'Enable polling for updates') Also untested so don't do this if you're unsure, I'll provide an update tomorrow advising if I had any luck with this or not.

Automation is triggered every minute to update the entities in the list. You'll need to change these to match your entity ids, I just added all entities associated with the integration.

alias: GoodWe SEMS Portal Polling
description: ""
trigger:
  - platform: time_pattern
    seconds: "00"
condition:
  - condition: time
    after: "00:15:00"
    before: "00:00:30"
action:
  - service: homeassistant.update_entity
    data: {}
    target:
      entity_id:
        - sensor.homekit_xyz
        - sensor.homekit_xyz_export
        - sensor.homekit_xyz_import
        - sensor.inverter_xyz
        - sensor.inverter_xyz_energy
mode: single

[davidsonimagerygmailcom]

You could even apply the logic of missing the first zero'ing entirely (as you'll get it after 1215 anyway) ... kinda why I went for 1159pm (bypass the whole shemozzle)

[skreza]

I wasn't sure if it needed to reset to zero at midnight for the energy dashboard to calculate it correctly.
Assuming it doesn't, then absolutely the condition could be changed to 12:15am through to 11:59pm

condition:
  - condition: time
    after: "00:15:00"
    before: "23:59:00"

[davidsonimagerygmailcom]

Just to add spanner to the works, I notice the first zero'ing was about 00:01:22 ... then 00:02:24 back up to 10.5kwh ... 00:06:02 back to 0kwh

The kwh readings aren't the same either so not sure what's going on back end.

[davidsonimagerygmailcom]

I wasn't sure if it needed to reset to zero at midnight for the energy dashboard to calculate it correctly. Assuming it doesn't, then absolutely the condition could be changed to 12:15am through to 11:59pm

condition:
  - condition: time
    after: "00:15:00"
    before: "23:59:00"

Good point. Not sure!
Might depend if it's total_increasing or total ... I've never deeply explored/understood

[davidsonimagerygmailcom]

alias: GoodWe SEMS Portal Polling
description: ""
trigger:

• platform: time_pattern
  seconds: "00"
  condition:
• condition: time
  after: "00:15:00"
  before: "00:00:30"
  action:
• service: homeassistant.update_entity
  data: {}
  target:
  entity_id:
  - sensor.homekit_xyz
  - sensor.homekit_xyz_export
  - sensor.homekit_xyz_import
  - sensor.inverter_xyz
  - sensor.inverter_xyz_energy
  mode: single

Where / how do you define your YAMl code? is it on the Automation itself, or into a file and then calling it somehow?

[skreza]

Create a new automation as normal, then press three dots in top right and select Edit in YAML.
Paste the code replacing anything already visible and then change it back to Edit in visual editor from that top menu again.

[davidsonimagerygmailcom]

How would we define every 30 seconds for polling instead of minute?

I wonder if ;

trigger:

• platform: time_pattern
  seconds: "00"
  seconds: "30"

is permissible

(please excuse my amateurism!)

[skreza]

Ok, so based on the fact that we don't really know when the data will spike after zeroing the first time and that the first zeroing may not be exactly at midnight I've modified things slightly and added a second automation. The first automation is same as above however I've removed the condition so it'll run every minute no matter what. The second automation will trigger when the entity hits 0, it will then disable the first automation wait fifteen minutes and then enable it again. This way we don't have to worry about exact timing from the SEMS side.

Automation 1

alias: GoodWe SEMS Portal Polling
description: ""
trigger:
  - platform: time_pattern
    seconds: "00"
condition: []
action:
  - service: homeassistant.update_entity
    data: {}
    target:
      entity_id:
        - sensor.homekit_xyz
        - sensor.homekit_xyz_export
        - sensor.homekit_xyz_import
        - sensor.inverter_xyz
        - sensor.inverter_xyz_energy
mode: single

Automation 2

alias: GoodWe SEMS Portal Polling Controller
description: ""
trigger:
  - platform: state
    entity_id:
      - sensor.homekit_xyz_export
    to: "0"
condition: []
action:
  - service: automation.turn_off
    target:
      entity_id: automation.goodwe_sems_portal_polling
    data:
      stop_actions: true
  - delay:
      hours: 0
      minutes: 15
      seconds: 0
      milliseconds: 0
  - service: automation.turn_on
    target:
      entity_id: automation.goodwe_sems_portal_polling
    data: {}
mode: single

[skreza]

Re 30 second polling, you would just need a second action, one covers 00 seconds the other covers 30 seconds.

trigger:
  - platform: time_pattern
    seconds: "00"
  - platform: time_pattern
    seconds: "30"

[davidsonimagerygmailcom]

Done and working well, i might just to go straight to using your method and see what happens tonight (it's 3pm here).

[micheldejongh]

Another option:

trigger:
  - platform: time_pattern
    seconds: /30

[davidsonimagerygmailcom]

Skreza's workaround worked perfectly

Stopped polling a bit before midnight... oblivious to the double dipping being done at Goodwe's Cloud , and then at 1220 polls again to observe a zero and move on

[skreza]

Glad to hear it worked @davidsonimagerygmailcom Now I need to figure out why it didn't work for me! Hahaha.
Did you end up making it time based entirely or did you use the last method I posted which looked for 0 kWh and then disabled the polling automation for 15 minutes?
I tried this way and it did hit 0 however it didn't seem to trigger that automation for some reason, last triggered should be around 12 hours ago not 19! Hmmmm

[davidsonimagerygmailcom]

Hey Mate, I just went with your first solution ... the simple one.
It just polls all day long (every 30 seconds), except betweeen 23:59 and 00:20 ... worked a treat.

[skreza]

Thanks for that, I might have made it more complex than it needed to be haha!
Reverted it back to time based with the condition, hopefully all good tonight!

[davidsonimagerygmailcom]

Fingers Crossed!
All crusing along on my side. I guess ew won't actually know if the double dip gets fixed until we go back to polling during 2359 > 0020hrs

[justinmaiuto]

Just found this one, seems similar issue (possibly the same?)

home-assistant/core#106677

[TimSoethout]

Hi all, great that you've found a workaround. I'm currently limited in time and energy to work on this, but I'm happy to merge a fix if someone provides it. :)

[davidsonimagerygmailcom]

Just found this one, seems similar issue (possibly the same?)

home-assistant/core#106677

Yeeepp that's what first caught my eye ... I was perplexed how I'd made so much power from the Moon, and who was running around pulling huge loads at midnight :)

[skreza]

Ok went back to my original suggestion and worked perfectly last night.

So for those looking for a workaround, the YAML code below works well, create a new automation and use this, make sure you change the sensors in the entity_id list to match your sensors. You also need to disable polling on the 'GoodWe SEMS PV API' integration (Open integration, click 3 dot menu, system options, turn off 'Enable polling for updates')

alias: GoodWe SEMS Portal Polling
description: ""
trigger:
  - platform: time_pattern
    seconds: "00"
condition:
  - condition: time
    after: "00:20:00"
    before: "23:59:00"
action:
  - service: homeassistant.update_entity
    data: {}
    target:
      entity_id:
        - sensor.homekit_xyz
        - sensor.homekit_xyz_export
        - sensor.homekit_xyz_import
        - sensor.inverter_xyz
        - sensor.inverter_xyz_energy
mode: single

[chicaneau]

Thanks for the automation to resolve the midnight double dip issue @skreza. I have also noticed that at around the same time this issue started, i am now seeing a gap in my house consumption between 10pm and midnight, is anyone else seeing this too? it's happening on my main SEMS homekit integration sensor sensor.homekit_91000hkuxxxxxxx. Any thoughts?

It was happening prior to disabling to my polling and moving to the automation for updating entity. I should note that my grid import stats are unaffected between 10pm and midnight which makes this ever stranger.

House Consumption

Grid Import

[justinmaiuto]

Thanks for the automation to resolve the midnight double dip issue @skreza. I have also noticed that at around the same time this issue started, i am now seeing a gap in my house consumption between 10pm and midnight, is anyone else seeing this too? it's happening on my main SEMS homekit integration sensor sensor.homekit_91000hkuxxxxxxx. Any thoughts?

It was happening prior to disabling to my polling and moving to the automation for updating entity. I should note that my grid import stats are unaffected between 10pm and midnight which makes this ever stranger. House Consumption

Grid Import

Im also seeing this, but it is for the 'current consumption' value, not the running daily total which the energy dashboard uses. So its not impacting the $ calc on the enegy dashboard, but is annoying that the other dashboard I setup has a gap in data

[skreza]

Confirming same issue for me also, however my data is blank from 9pm - midnight, I wonder if this is time zone based? I'm in Melb so currently +11 UTC.

I hadn't noticed this as I don't look at house consumption all that much, I have another integration that pulls data from a Powerpal on my meter and tend to look more at grid import power rather than ongoing house consumption. As @justinmaiuto mentioned, the total consuming is still correct though so data in energy dashboard should still be pretty accurate.

Data in SEMS portal looks fine, pretty weird issues but unfortunately I'm no dev, so trying to fix the source code is above my expertise!

[chicaneau]

Confirming same issue for me also, however my data is blank from 9pm - midnight, I wonder if this is time zone based? I'm in Melb so currently +11 UTC.

I hadn't noticed this as I don't look at house consumption all that much, I have another integration that pulls data from a Powerpal on my meter and tend to look more at grid import power rather than ongoing house consumption. As @justinmaiuto mentioned, the total consuming is still correct though so data in energy dashboard should still be pretty accurate.

Data in SEMS portal looks fine, pretty weird issues but unfortunately I'm no dev, so trying to fix the source code is above my expertise!

so you lose 3 hours of data? weird! Im brisbane so im +10 UTC

[davidsonimagerygmailcom]

No issue here on that 9pm to midnight bit
Perth GMT+8

The double dip still occurs though (for humour last night I turned the polling back on between 2359 > 0015, double dipping still there).

[davidsonimagerygmailcom]

Right I'm seeing weird things were house load is reading the same value for hours on end. THought it was HA, but sems portal shows the same thing. Yet, the graph is in sems portal shows the correct house load changing. Am I missing something here? Only seemed to break down today. Not a polling issue because the consumtion counters are increasing (and in HA too)

[justinmaiuto]

Right I'm seeing weird things were house load is reading the same value for hours on end. THought it was HA, but sems portal shows the same thing. Yet, the graph is in sems portal shows the correct house load changing. Am I missing something here? Only seemed to break down today. Not a polling issue because the consumtion counters are increasing (and in HA too)

Yep, I noticed the same in SEMS portal. The graph is ok, but the text value gets stuck. This coincides with what happens in HA.

[davidsonimagerygmailcom]

The double dip issue is still there, tested last night.

Goodwe support (when they choose to respond to repeated followups) simply say "I can see from my end that the inverter is back online and the readings are showing.
And regarding the double dip issue is it still presisting ? because from my end i am seeing that it doesn't do that and that it restarts to zero at midnight with no problem"

Without any regard to the extensive outlining of my information prior to them. The issue doesn't appear to SemsPortal because whatever this "zero'ing" procedure is that they're doing, it's not being ingested by the sems graph / portal data.

Have repeatedly asked to have the issue escalated to the team whom actually look after the back end databases but falls on deaf ears.

back to leaving the polling off between 2359 and 0020

[davidsonimagerygmailcom]

Deleted the last comment about 1130pm uptick, as realised internet had been down and homekit wasn't reporting usage for several hours. Disregard for anyone that read it before I deleted.

[BrutalDeluxe20]

Thanks everyone for your input into this. @skreza I've implemented your automation and data looks spot on today, so thanks for sharing.

[BrutalDeluxe20]

Hi @TimSoethout, I seem to be having issues with this after daylight savings has started here in Melbourne Australia. I've changed the integration to stop upating the entities now from 1am - 1:10am (ie. what was previous 12am - 12:10am before daylight savings), but it doesn't seem to be working?
Any ideas on what is going on? Unless whereever I'm pulling the data from has started daylight savings at the same time therefor the time difference is further skewed? Thanks a lot

Before daylight savings started

Today

[davidsonimagerygmailcom]

Hmm, no DLS here in WA, but I'm seeing similiar weird things with big spikes 3am, then nothing either side.
Might not be the original issue this thread refers to, but usual SemsPortal antics?
It really shouldn't be this difficult but whoever is handling the mathematical / database rolling and management really seem to struggle.

[skreza]

@BrutalDeluxe20 I'm Melb based and haven't had any issues post daylight savings.

Based on previous comments, you mentioned you changed to automation polling rather than using the SEMS polling in the integration. How have you got it configured currently? Mine is still on the automation as per: #94 (comment)

[BrutalDeluxe20]

Hi @skreza, yes I have it setup as you do. So have disabled polling in the GoodWe SEMS API integration and then use an automation to run every minute to manually update the various entities with the break around midnight. I did only have 10 minutes I think, so have extended that to 20 as per comment #94 and will see how it goes tonight. Just strange it seemed to stop working for me when DLS started?

[TimSoethout]

Hi @TimSoethout, I seem to be having issues with this after daylight savings has started here in Melbourne Australia. I've changed the integration to stop upating the entities now from 1am - 1:10am (ie. what was previous 12am - 12:10am before daylight savings), but it doesn't seem to be working? Any ideas on what is going on? Unless whereever I'm pulling the data from has started daylight savings at the same time therefor the time difference is further skewed? Thanks a lot

Before daylight savings started

Today

It also could be ralated to the component only recovering after some time, and counting all the previous hours to the current hour. Sometimes you get graphs like that then.