Homekit sensors going backwards, daily

[gemmaker]

I'm using two integrations for separate Goodwe components. I have a Goodwe Inverter, and a Goodwe Homekit (which is correctly showing in SEMS - which means with the PV output, export to grid, and grid input, the difference is how household load is derived).
The values from my inverter SEMS are working, and showing in the HA energy page correctly:

However the values for both PV export, and Grid Import, while generally tracking correctly both go 'backwards' daily, then correct themselves. This doesn't appears in the SEMS itself. The jumps backwards (which relates directly) to the previous day's total) results in crazy graphs and totals within the HA energy page, every day.

[TimSoethout]

The same was reported here:
#59 (comment)

I'm not sure what happens and can't debug myself because I don't have HomeKit.

Maybe we can work around it bij taking the previous value if it drops or something?

[jay-oswald]

I'm getting a similar problem, heres my data import from HomeKit, was just installed last week, which is why the graph starts flat.

You can see it drops, but dosen't recover fully or at all somtimes, and it makes the graph goes weird

The data seams to be correct except for the drops

[jay-oswald]

Maybe we can work around it bij taking the previous value if it drops or something?

This would solve the original problem, but it would just break mine, since mine seams to drop, but not recover

[gemmaker]

@jay-oswald is your increasing total matching the value you're getting in SEMS/total production on the inverter itself? Mine is (except for the nightly decrease/increase)

[jay-oswald]

@jay-oswald is your increasing total matching the value you're getting in SEMS/total production on the inverter itself? Mine is (except for the nightly decrease/increase)

@gemmaker I'm not sure, the data does not seam to match. Its almost as if the data is total energy usage, then every night at midnight the solar generation is removed to drop down to import, instead of total. Not sure why it would do that, but its the only thing that explains it

[jay-oswald]

Here's powerpal monitoring, which has been accurate for months, showing auctal import

[Hufflepuff-source]

Hello,

Seems like you are using Powerful and GoodWe like myself. Could you tell me how you integrated the Goodwe inverter and HomeKit energy consumption to HA and what entities you are using to get the energy dashboard up and running? I am really struggling here and any help would be greatly appreciated.

[philipbrennan]

I have the same issue with the totals moving every 24hours, but also one of them seems to be wrong.

I have homekit, inverter is not goodwe.
My focus is currently on the total generated, because I have 3 other sources to cross check with ( SEMS api, the inverter itself, and counter in the fusebox )
Goodwe SEMS site does show the correct total, to within one Kwh. This amount is not exposed currently by the integration

I was using this total from the integration:

300 kwh, which is wrong.

SEMS shows 386.07

and after a small code change to read this from the kpi section of the response from SEMS

I will see if this fluctuates daily, hopefully not.
I tried opening a PR for this but I don't have permission.

[philipbrennan]

Couple more things:
I've had to go my own way on this a bit since I have homekit and no goodwe inverter.
The values reference in the documentation eday, iday, etc are not present in the api response for me.
So I used these values instead:

But they seem to have the same issue as you guys had.
One thing I'm going to try, is to use the new total_increasing sensor class in home assistant.
I defined my sensors from scratch using the new template sensor format.
I'm going to change those totals to total_increasing and hopefully this will avoid the issue with the dip during the night.

I think that the values going backwards every day is most likely a quirk of the SEMS API. If we were to look at the SEMS API in the middle of the night I think we would see the issue.

Here's what I added to configuration.yaml:

template: 
  - sensor:
      - name: "PV Front"
        state: '{{ states.sensor.solar_pv_91000hku21600330.attributes.pv }}'
        unit_of_measurement: 'W'
        state_class: "measurement"
  - sensor:
      - name: "Grid"
        state: '{{ states.sensor.solar_pv_91000hku21600330.attributes.grid }}'
        unit_of_measurement: 'W'
        state_class: "measurement"
  - sensor:
      - name: "House Consumption"
        state: '{{ states.sensor.solar_pv_91000hku21600330.attributes.load }}'
        unit_of_measurement: 'W'
        state_class: "measurement"
  - sensor:
      - name: "PV Status"
        state: '{{ states.sensor.solar_pv_91000hku21600330.attributes.PowerFlowDirection }}'
        unit_of_measurement: 'W'
  - sensor:
      - name: "PV generation total"
        state: '{{ states.sensor.solar_pv_91000hku21600330.attributes.all_time_generation }}'
        unit_of_measurement: 'kWh'
        state_class: "total"
        device_class: "energy"
  - sensor:
      - name: "PV export total"
        state: '{{ states.sensor.solar_pv_91000hku21600330.attributes.Totals_sell }}'
        unit_of_measurement: 'kWh'
        state_class: "total"
        device_class: "energy"
  - sensor:
      - name: "PV import total"
        state: '{{ states.sensor.solar_pv_91000hku21600330.attributes.Totals_buy }}'
        unit_of_measurement: 'kWh'
        state_class: "total"
        device_class: "energy"
  - sensor:
      - name: "PV generation today"
        state: '{{ states.sensor.solar_pv_91000hku21600330.attributes.Charts_sum }}'
        unit_of_measurement: 'kWh'
        state_class: "measurement"
        device_class: "energy"
  - sensor:
      - name: "PV export today"
        state: '{{ states.sensor.solar_pv_91000hku21600330.attributes.Charts_sell }}'
        unit_of_measurement: 'kWh'
        state_class: "measurement"
        device_class: "energy"
  - sensor:
      - name: "PV import today"
        state: '{{ states.sensor.solar_pv_91000hku21600330.attributes.Charts_buy }}'
        unit_of_measurement: 'kWh'
        state_class: "measurement"
        device_class: "energy"
  - sensor:
      - name: "PV Self Use Today"
        state: '{{ states.sensor.solar_pv_91000hku21600330.attributes.Charts_selfUseOfPv   }}'
        unit_of_measurement: 'kWh'
        state_class: "measurement"
        device_class: "energy"

[gemmaker]

I have the same issue with the totals moving every 24hours, but also one of them seems to be wrong.

I have homekit, inverter is not goodwe. My focus is currently on the total generated, because I have 3 other sources to cross check with ( SEMS api, the inverter itself, and counter in the fusebox ) Goodwe SEMS site does show the correct total, to within one Kwh. This amount is not exposed currently by the integration

I was using this total from the integration:

300 kwh, which is wrong.

SEMS shows 386.07

and after a small code change to read this from the kpi section of the response from SEMS

I will see if this fluctuates daily, hopefully not. I tried opening a PR for this but I don't have permission.

I'm excited to see if it works! Thanks! I tried a sensor template before but I was struggling with the syntax of the new templating.

[jay-oswald]

I found a node red flow online that pulls the stats in, without this integration, and they come in properly without the issue. So that kpi/total_power I can confirm works as it should.

However I also have a HomeKit monitor to monitor grid in/out, and those numbers still seam to suffer from the same issue with it dropping, and there does not seam to be an alternative that works from looking through the full API response.

If the kpi/total_power is used would just mean you wouldn't get the data per inverter, just overall data, but IMO overall data that works is better than broken down data that doesn't. Especially because most people only have a single inverter.

Here is my grid import sensor, and you can see even if it was only incrementing and didn't allow drops it would still have issues since it can sometimes take hours to recover to where it was at the midnight drop

[philipbrennan]

Hi Jay, Yeah this is why I only brought total power in as a separate attribute and didn't replace anything existing with it.
Looking more closely at the export and import, they are rubbish. need to log an issue with goodwe.
meanwhile node red flow, Ok I had a quick look. what apis does your flow call? if its the sems api id be interested to see why it doesnt have issues with import and export.

[philipbrennan]

what I've found. the daily totals look fine. they reset at 12:02am.
I'm going to use them as the driver for my total import and export ( and you could use them for total generation as well )
this will only be possible by defining your sensors as template sensors, see my config above
and using the total_increasing class.
the last problem will be to set the starting point, so that I'm not starting from zero today. ha doesnt seem to give a nice way of doing that but add a constant in the sensor definition . I'll report back.

[jay-oswald]

Hey @philipbrennan I've got a ticket open with GoodWe at the moment about it, but pretty sure there's been lots of people create tickets about it and they haven't done anything. I'm mainly doing it so that GoodWe can admit it dosen't work and Im going to see if I can get my money back and replace the HomeKit with an IoTaWatt instead.

For me I don't mind much about starting from 0 today, as far as im concerned all the old data I've got is useless anyway.

So your template sensors worked properly overnight? It didnt drop and bounce back overnight? If you look at my graph above it takes till about 9am to get back above where the drop was, those spikes throughout the morning all seam to be recovering the loss at midnight, I don't have anything running overnight, not even hot water or anything, just my 200-250w baseload

For my node red flow I call the main get endpoint v2/PowerStation/GetMonitorDetailByPowerstationId with ofcourse my id in the params passed into it.
I'm getting the generate from data.kpi.total_power
Then im getting buy/sell from data.energeStatisticsTotals.buy and ofcourse x.sell

Ther dosen't seam to be any alternative to the energeStatisticsTotals for the buy/sell from homekit, like the inverter has

[philipbrennan]

Hi Jay,
Yeah, that's not too surprising re goodwe.. this isn't a small issue.
Sorry I wasn't clear. My current template sensors track the total data which we've seen is wrong.
My trial using total_increasing -
last night:

previous night -

Its hard to say that its really any better. the inconsistency in the incoming data is not consistent. Its very bad.
So what I'm going to try now is just get HA to sum the daily totals, that is what total_increasing is really meant for anyway, a utility meter that periodically resets to zero
The daily total data looks good - here's the past 6 days

always resets cleanly to zero just after 12.
I hear you about changing to something else but I think that would be a long battle with my solar installer. I'm happy that the total generation is very accurate compared to my fuseboard so with these wrinkles ironed out I'll be happy enough.

[philipbrennan]

so what I've done is

changed:

- sensor:
      - name: "PV export total"
        state: '{{ states.sensor.solar_pv_91000hku21600330.attributes.Totals_sell }}'
        unit_of_measurement: 'kWh'
        state_class: "total"
        device_class: "energy"
  - sensor:
      - name: "PV import total"
        state: '{{ states.sensor.solar_pv_91000hku21600330.attributes.Totals_buy }}'
        unit_of_measurement: 'kWh'
        state_class: "total"
        device_class: "energy"

to this:

 - sensor:
      - name: "PV export total"
        state: '{{ states.sensor.solar_pv_91000hku21600330.attributes.Charts_sell + 211 }}'
        unit_of_measurement: 'kWh'
        state_class: "total_increasing"
        device_class: "energy"
  - sensor:
      - name: "PV import total"
        state: '{{ states.sensor.solar_pv_91000hku21600330.attributes.Charts_buy + 198.5 }}'
        unit_of_measurement: 'kWh'
        state_class: "total_increasing"
        device_class: "energy"

I didn't get the adjustments quite right so there is still a small bump for today ( I don't know how to reset sensors completely without recreating them completely ). But I think it will look fine over next few days.

[philipbrennan]

So I've done the changeover, actually its fine for export since that hadn't start moving yet today :)

import, small bump from the switchover, the dip is from the usual problem last night. hopefully all good from now on

Question for @ once this is working, I'd like to PR in my sensor templates. I'll change the total generation to this approach as well as it is a direct replacement into the current structure and would work for multiple inverters, and add comments explaining how to do this part of it.
I think I'll just add a new folder and link to it from the readme?

[philipbrennan]

so this approach doesn't quite work. On further reading, total_increasing tolerates 10% fluctuations in the value.. and since I applied an offset of ~ 200 units , the nightly reset is not being interpreted as a reset. I'll need some better logic to only apply the offset when outside of the nightly reset range. Will update when I have it working.

[philipbrennan]

I think for this to work I'd need a bit more logic, to build a template sensor which smoothly increases in output over time, without any resets or spikes. I'm sure others have come across this so I'll take a broader look.

[gemmaker]

Perhaps Goodwe has changed something its end. This morning my usual two 'correction' spikes are missing and I appear to have correct data!
Will monitor :)

[jay-oswald]

I still had the spikes on all 3 overnight. I might be using the wrong sensor for the solar generation still, I've got it in there 3 ways, but I did see the code get pushed for the kpi version. But the import/export still has the spikes for me. GoodWe support was responding within a couple of hours initially while they figurd out who I was, my serial numbers etc, now that the ticket is about these spikes/drops I haven't gotten a response in almost 2 weeks

[philipbrennan]

Mine still has the spikes. I think they are fairly severe bugs in goodwe code.. they can't figure out how to keep a rolling total without spikes :(

[sammcewan]

Seems nuts you can't even report on the homekit data from their portal.

I posted this in the other issue and it's definitely calmed things down with import export. #73 (comment)

Export is now sweet as if I check if the inverter's doing anything.

Only issue now is my template logic is still allowing that second drop for some reason.

[gemmaker]

Thanks to @Denifia this issue is fixed!
For the first time ever I have a non broken graph.
Confirming that the homekit output does have one 'reversal' spike upon implementing the new version of the Integration, but it has behaved perfectly since.
This means a lot to me as it is the use of this integration (with Goodwe homekit hardware) combined with the Solcast prediction integration that is the reason I started using HA in the first place - to only run appliances when I have excess power and predicted excess power to reduce grid/import reliance.
Thanks everyone for helping!

Day 1 after new version- one reversal spike after midnight:

Day 2 (partial) - no spikes: