Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Decision Proposal 361 - Energy LCCD Phase 2 #361

Open
CDR-API-Stream opened this issue Dec 18, 2024 · 31 comments
Open

Decision Proposal 361 - Energy LCCD Phase 2 #361

CDR-API-Stream opened this issue Dec 18, 2024 · 31 comments
Assignees
Labels
Category: API A proposal for a decision to be made for the API Standards made Category: CX A proposal for a decision to be made for the User Experience Standards Industry: Electricity This proposal impacts the electricity industry sector Status: Decision Made A determination on this decision has been made

Comments

@CDR-API-Stream
Copy link
Contributor

CDR-API-Stream commented Dec 18, 2024

Update: February 7th, 2025

Feedback for this consultation has closed. Thank you to everyone who contributed. Feedback will be reviewed and responded to before a data standard is made by the Chair.

--

Original post: December 18th, 2024
This decision proposal outlines and seeks feedback on the proposed standard changes for incorporating the Last Consumer Change Date (LCCD) into the CDR standards for the energy sector.

The proposal covers changes to the standards for:

  • Sharing of historical electricity usage data based on LCCD
  • Include sharing of the LCCD via CDR
  • CX considerations
  • Future Dated Obligations

The proposal for consultation is attached below:
Decision-Proposal-361-Energy LCCD-Phase-2.pdf

Community views are being sought on the draft standards and positions outlined in this paper. Feedback received will be considered before standards are proposed to the Chair to be made binding.

This consultation period will close on Friday Tuesday 4th February 2025. Friday 7th February 2025

Decision made

The Chair's decision is linked from this comment.

The changes approved in the decision will be published in Standards v1.34.0.

@CDR-API-Stream CDR-API-Stream added the Industry: Electricity This proposal impacts the electricity industry sector label Dec 18, 2024
@CDR-API-Stream CDR-API-Stream changed the title [Placeholder] For upcoming LCCD proposal Decision Proposal 361 - Energy LCCD Phase 2 Dec 20, 2024
@CDR-API-Stream CDR-API-Stream added the Status: Open For Feedback Feedback has been requested for the decision label Dec 20, 2024
@mattyp
Copy link

mattyp commented Dec 23, 2024

Good work, guys. 👍

QQ - if there a particular reason the lastConsumerChangeDate field is proposed as optional rather than conditional upon the LCCD field existing for the Service Point in question?

@HemangCDR
Copy link
Contributor

@mattyp the optional fields in the standards must to be shared if the DH holds the value. See "Mandatory/Optional Fields" in Payload Conventions. In other words, they are conditional to the data being held.

In other words, if the lastConsumerChangeDate is available for a given NMI/Service Point, it must be shared.

Hope this clarifies.

@mattyp
Copy link

mattyp commented Dec 24, 2024

Ah, yes, thanks for the reminder 😝. So, does that mean that some retailers may not have access to the LCCD to populate lastConsumerChangeDate? If so, then ADRs cannot rely on that property existing and so cannot create capability around it. If all retailers have access to it, then it should be conditional on there being a date. conditional or mandatory properties provide significantly more value to ADRs than optional ones. In any case, I think the real value is in the more accurate historical usage data rather than the date.

@HemangCDR
Copy link
Contributor

@mattyp, below are few points which further clarifiy the proposal of making the lastConsumerChangeDate field optional:

  • The lastConsumerChangeDate is populated by AEMO (not by retailers) as its a field in their MSATS database. AEMO, as the data holder for standing data in CDR, will provide the value. The retailer would only be passing it along like any other data AEMO provides.
  • The field is being populated gradually by the industry (outside of CDR) and, I believe, the intent is that eventually it should be fully populated for all connection points.
  • Defining the lastConsumerChangeDate as optional in CDR makes it conditional - i.e. if AEMO holds the value, they must provide it. This will allow the value to be provided where available while the gradual population of it continues.

@mattyp
Copy link

mattyp commented Dec 27, 2024

Thanks, @HemangCDR. It's good to have these points available for all to find 👍. So, by virtue of the DH architecture coupled with the definition of optional the property is implicitly conditional (on the LCC date having been populated in MSATS for the servicepoint in question). We may as well just explicitly make it conditional 😉. I won't labour on it longer (as there are bigger issues to resolve in CDR's energy sector atm, and there must be good reason for a preference of 'optional' over 'conditional'), but my preference would be to have this conditional rather than optional so as to provide explicit confidence to ADRs and their reps that this property will be set if the date exists, regardless of any future DH changes that might otherwise impact on an implicitly conditional property.

@Sbendat
Copy link

Sbendat commented Jan 7, 2025

LCCD is an exciting step forward! Thanks Hemang and team.

I would second mattyp's point about optional fields being a bit of a thorn for ADRs. Constantly having to check if certain fields exist adds to the complexity of understanding if no data in a response was intended or is a legitimate issue that needs to be flagged. On many occasions we at SolvingZero have had to follow up through the various channels to figure out if the missing field is a legitimate issue or the case of an optional field not present. Primarly because, we feel, DHs will use optional fields very liberally.

As one example, DHs will populate optional fields with data that is effectively the same as it not existing, eg end_date can be in the year 9999 or not exist, both imply the date is basically infinite. In these cases the optional fields are providing too much wiggle room and the data quality suffers as a result.

Getting back to LCCD. Adjacent to LCCD, being able to track the energy tariffs a customer had during their historical usage across multiple DHs would be a significant benefit to ADRs and the end consumer. This will allow for the usage to be translated into dollar values (costs), which by and far is the preferred unit of measure for consumers, rather than kWh or some other energy-based measure of units. Perhaps this is out of the scope of LCCD, but this would be a huge next step to build upon the work of LCCD for the benefit of the consumer.

@mattyp
Copy link

mattyp commented Jan 7, 2025

Adjacent to LCCD, being able to track the energy tariffs a customer had during their historical usage across multiple DHs would be a significant benefit to ADRs and the end consumer. This will allow for the usage to be translated into dollar values (costs), which by and far is the preferred unit of measure for consumers, rather than kWh or some other energy-based measure of units.

Sounds like a good suggestion, worthy of its own maintenance issue, @Sbendat 👍. Another use-case for that data would be to know if a consumer is eligible for a tariff change (under the standard one tariff change per year DNSP rule). I can outline that against the dedicated maintenance issue.

@TaylorWoods1
Copy link

Hi Team, to what extent does the data request predate the LCCD? Are there guidelines for a fixed duration? % of historic data available? other?

@HemangCDR
Copy link
Contributor

HemangCDR commented Jan 22, 2025

@TaylorWoods1 the rules limit the sharing of historical energy usage/metering data to a maximum of 2 years from the date of request.

@TaylorWoods1
Copy link

Thanks Hemang,

To confirm, is the intent of the 2-year data-sharing rule to provide all available usage data from the smart meter for the past 2 years, regardless of the customer’s tenure with their current retailer?

For example, if a customer has had a smart meter for 2 years but has only been with the current retailer for one week, would they still receive the full 2 years of data upon providing consent?

Regards,

Taylor

@AGL-ENERGY-CDR
Copy link

Hi @HemangCDR ,

AGL is currently in the process of reviewing the Decision Proposal 361 in detail. To determine whether we may need to provide further feedback to this consultation, we are seeking the DSB's clarification as to:

  1. How the Participant ID will be used as a part of the Get Usage functionality for Service Point API payload when the LCCD and consent timeframe includes time FRMP with multiple retailers? We note that the Participant ID is mandatory in the header as a part of AEMO API guide but is not included in the DSB specs.

  2. Accredited Data Holders cannot open AEMO payloads. Does the DSB expect that that ADRs would be responsible for opening and updating the payload "oldest_date" with the LCCD from AEMO?

  3. If this Decision Proposal 361 takes effect as currently drafted, Data Holders will no longer be able to share closed account data because of an updated LCCD.

However, Part 3.2 Clause (7) of the Competition and Consumer (Consumer Data Right Rules) 2020 specifies that:
Despite subclause (2), for an account that is closed at a particular time, each of the following is not required consumer data:

(a) CDR data held by AEMO, other than metering data;
(b) CDR data held by a retailer, other than billing data; and
(c) CDR data that is not excluded by paragraph (a) or (b), but relates to a transaction or event that occurred more than 2 years before that time.

AGL is seeking clarification as to whether the DSB believes that the Decision Proposal 361 is compatible with the provisions above, or whether it anticipates that Treasury will seek changes to the legislation to facilitate these new changes.

@HemangCDR
Copy link
Contributor

To confirm, is the intent of the 2-year data-sharing rule to provide all available usage data from the smart meter for the past 2 years, regardless of the customer’s tenure with their current retailer?

For example, if a customer has had a smart meter for 2 years but has only been with the current retailer for one week, would they still receive the full 2 years of data upon providing consent?

@TaylorWoods1 yes this is the intent.

@AGL-ENERGY-CDR, I've tried to respond to your questions below:

  1. How the Participant ID will be used as a part of the Get Usage functionality for Service Point API payload when the LCCD and consent timeframe includes time FRMP with multiple retailers? We note that the Participant ID is mandatory in the header as a part of AEMO API guide but is not included in the DSB specs.

I don't fully understand the question. We don’t anticipate any changes to how participant ID will be used. Our understanding is it is used by AEMO for authenticating retailers and will continue to be so. Does this address your query?

2. Accredited Data Holders cannot open AEMO payloads. Does the DSB expect that that ADRs would be responsible for opening and updating the payload "oldest_date" with the LCCD from AEMO?

It is not clear if the question is for data holders or data recipients accessing AEMO payload? The ADR's would set the oldest-date to the value they want the usage data to go back to. The DH (Retailer) would (assuming its for an existing open account), would send it to AEMO as is. AEMO would use that date and apply various checks to determine how much historical data to share. One of the checks would be LCCD as being proposed by this paper.

3. If this Decision Proposal 361 takes effect as currently drafted, Data Holders will no longer be able to share closed account data because of an updated LCCD.

As noted in the DP, we do not anticipate any impacts on how data is shared for closed accounts. The data sharing for closed accounts would continue as it presently occurs – being limited to the duration the retailer was the FRMP for the NMI. This would require the retailer to ensure the request range for closed accounts is limited to how long the customer was their account holder for the NMI.

AGL is seeking clarification as to whether the DSB believes that the Decision Proposal 361 is compatible with the provisions above, or whether it anticipates that Treasury will seek changes to the legislation to facilitate these new changes.

We do not believe the proposed changes have any conflict with data sharing requirements for closed accounts. As noted above, sharing of usage data for closed accounts will continue as it presently occurs, limited to the duration the retailer was the FRMP for the account.

Hope the above clarifies.

@CDR-CX-Stream CDR-CX-Stream added Category: API A proposal for a decision to be made for the API Standards made Category: CX A proposal for a decision to be made for the User Experience Standards labels Jan 24, 2025
@johnAEMO
Copy link

johnAEMO commented Jan 30, 2025

AEMO thanks DSB for the opportunity to comment on the above Decision Proposal.

We take this opportunity to request an extension of the feedback period of at least one week to allow our development staff who are on leave, to return and confirm our ability to meet our proposed FDO of November or if it needs to be later.

Our initial feedback is enclosed in the following pdf and includes a tentative proposed FDO of November 2025 which is to be confirmed or modified and appended to this feedback.

AEMO feedback - DP 361.pdf

We welcome any further feedback or discussion.

@CDR-API-Stream
Copy link
Contributor Author

@johnAEMO , thank you for your feedback.

We will extend the consultation period till next Friday the 7th of February as per AEMO's request to confirm the FDO dates.

@SelenaLiuEA
Copy link

Hi,

On behalf of EnergyAustralia:

We note an issue with Decision Proposal 361 which does not appear to have been contemplated and would result in unintended outcomes.

The issue is a consequence of the following factors:

  • The DSB’s proposed implementation date being 8 September 2025
  • AEMO being obligated to share data back to the LCCD for up to the previous two years so potentially back to 8 September 2023. This two year cap is not clear from the CDR Rules but we believe this is the intent. If this is not the case, then this issue is broader and the proposed solution below will not resolve it.
  • All retailers only starting to populate the LCCD flag from 1 November 2023 in accordance with AEMO requirements. Note: AEMO’s decision for the initial population of the LCCD of “Do not initialise” means that until the LCCD is populated gradually as customers inform their retailers, there is no data in the field.
  • In combination the above means that AEMO would be sending historical usage data between 8 September 2023 and 1 November 2023 when the LCCD requirement was not in place, and it is blank, when, in reality there may have been a customer change that was not recorded by the retailer (because the requirement had not commenced yet). This would result in extra and incorrect data being shared which does not relate to the customer.

To resolve the issue, we recommend that the implementation date of the Decision Proposal be aligned to retain at least two years from the date the LCCD flag commenced i.e. 1 November 2025. This would mean the issue does not result as AEMO data sharing of historical usage data would be capped at two years, and not send data prior to 1 November 2023.

We’d welcome a discussion if more explanation is required.

@SelenaLiuEA
Copy link

Hi,

EnergyAustralia also has a comment in response to:

Getting back to LCCD. Adjacent to LCCD, being able to track the energy tariffs a customer had during their historical usage across multiple DHs would be a significant benefit to ADRs and the end consumer. This will allow for the usage to be translated into dollar values (costs), which by and far is the preferred unit of measure for consumers, rather than kWh or some other energy-based measure of units. Perhaps this is out of the scope of LCCD, but this would be a huge next step to build upon the work of LCCD for the benefit of the consumer.

At first glance historical energy tariffs (which match the period that historical usage data is provided under LCCD) does sound helpful and we generally agree that expressing unit of measure in dollar value is more meaningful to the customer.

However, when unpacking this proposal against use cases, we question the utility of this proposal in practice. We note that use cases will involve comparing a customer's current pricing/tariffs against new product pricing, and that this comparison will be forward looking rather than historical. To do this, the key information is the customer's current pricing/tariffs and the best prediction of their usage (which is historical usage data). That is, current tariff rather than historical tariff information is the most relevant input.

Further, we also note that the current retailer through which historical usage data will be provided (using the LCCD field), will not have historical tariff information to match this data. Rather this information will be held by the previous retailer. This would require significant changes to the CDR to access this information, with the simplest and most appropriate solution being that the data be provided by the previous retailer (to avoid sharing of tariff information between competitors). This then raises the question why the previous retailer is not also providing the historical usage data which the LCCD change was built to facilitate.

Very happy to unpack this issue more with the DSB and others.

Thank you

@perlboy
Copy link
Contributor

perlboy commented Feb 4, 2025

@SelenaLiuEA point of clarification here:

This then raises the question why the previous retailer is not also providing the historical usage data which the LCCD change was built to facilitate.

Because the retailer isn’t considered the Holder of usage data in ANY circumstance, this is AEMO as the Secondary Data Holder, the Retailers have an obligation to pass this through unaltered and it is non-compliant to do otherwise.

@JamesMBligh
Copy link
Contributor

My 2 cents on this DP and the related conversation.

On the proposal itself
This awesome and about time. It will make a huge difference for consumers that frequently switch (which should be a larger number than it currently is) and it has taken a long time to get here. Kudos to the DSB, AEMO, Treasury and the retailers.

On the lastConsumerChangeDate field

  • An optional designation is clearly correct based on the definition of optionality in the standards. Conditional would mean the presents is mandatory due to some external logic which is not the case here. Optional simply means there are known cases where the data is not held. If it is held it must be presented so there is no issue.
  • The lastConsumerChangeDate field should not default to two years ago if not present as this will confuse the use of the data. If I have owned my home for ten years and I switched retailer multiple times five years ago then the the LCCD probably won't be set because the current retailer can't possibly know how long I have lived there. In this case a blank lastConsumerChangeDate is meaningfully different from a date that is set. Also, in three years time if we use this default then you will have some NMIs with lastConsumerChangeDate set to some time in 2024 (a real date) and blank fields will be represented as occurring in 2026
  • Agree that it should be date and not datetime

Regarding AEMO's feedback

  • Strongly disagree that AEMO should start sending errors if the retailer screws up. This is a bad precedent as AEMO can't definitively know (as they don't really know the customer) and it blurs the lines of accountability. The existing obligation on the retailer to only request data for a period they know the customer owned the NMI still applies (as it does now).
  • Sending errors is also a bad idea as the ADR gets no data for the request at all and has no ability to fix this as the error can only be with the retailer. If AEMO encounters what they consider an error case they should supply the best data set possible to benefit the consumer who is ultimately making the request and then call the retailer and explain that they have a bug as a secondary process (ie. respond but escalate)
  • Also, there is no issue here. The LCCD date is simply a more granular substitute for the FRMP boundary which is already in place. All of the cases where a retailer could screw up and request data for a NMI they own for a long time still apply if the LCCD field is set. Also, if the LCCD field is set mid FRMP ownership period (and not on a FRMP boundary) then the field must have been set by that retailer so they should know to truncate requests to that date anyway
  • Keep it simple. All this should do is introduce a new boundary condition alongside the two year rule and FRMP boundaries

On the FDO date

  • I don't really care as I don't have a dog in the hunt
  • The date should be set for when AEMO can build it exclusively but because it is grey as to whether the regulator can even hold another government entity to account it is a bit moot. The FDO should really be an indicator to ADRs as to when they can expect the change
  • The retailers should not care about the FDO date as it has zero impact on them. The payload is pass through and their obligations don't change. Also, the argument that it should go from two years from when the LCCD field started isn't strong as that date is only when it started, not when it was universally populated. It isn't even the start data of the values in the field as it's possible for a retailer that knows the real date to set a value prior to the commencement date

On Closed accounts
AGL has a point regarding closed accounts due to the current language. For closed accounts the period requested should allows be entirely prior to the LCCD date by definition. This means that the language should be changed so that the LCCD truncation only occurs when the LCCD date falls within the requested period. If the LCCD date is wholly before the requested period then no issue. If wholly ahead of the requested period then its a closed account and normal FRMP boundary rules apply

Based on the above I recommend only one change which is to modify the standards language to facilitate closed accounts as follows:

If the request is for historical usage data, then:

  • If LCCD value is held for the requested NMI, and the LCCD falls within the period request, then:
    • AEMO MUST limit the period of historical usage data returned to the period
      dating back to (and inclusive of) the LCCD
  • If LCCD value is not held for the requested NMI:
    • AEMO MUST NOT respond with an error
    • AEMO MUST respond with the usage for the period that the Primary Data Holder can be definitively determined to be in control of the NMI
    • AEMO MUST NOT share data outside the period of control of the Primary
      Data Holder

@SelenaLiuEA
Copy link

Just to clarify

*>

  • Also, the argument that it should go from two years from when the LCCD field started isn't strong as that date is only when it started, not when it was universally populated. It isn't even the start data of the values in the field as it's possible for a retailer that knows the real date to set a value prior to the commencement date

Thanks James, it won't be universally populated, but the two year limit means at least the blanks and LCCD dates that are actually in MSATS will be correct (assuming AEMO only has to share historical usage data up to two years)

@SelenaLiuEA
Copy link

@SelenaLiuEA point of clarification here:

This then raises the question why the previous retailer is not also providing the historical usage data which the LCCD change was built to facilitate.

Because the retailer isn’t considered the Holder of usage data in ANY circumstance, this is AEMO as the Secondary Data Holder, the Retailers have an obligation to pass this through unaltered and it is non-compliant to do otherwise.

Agree, loose language. I mean it raises the question why the previous retailer isn't facilitating the provision of historical usage data (via the customer authenticating, authorising through them as opposed to the current retailer)

@johnAEMO
Copy link

johnAEMO commented Feb 4, 2025

Great discussion

Closed accounts and access to historical usage data.

Can I attempt to flesh out an underlying question that seems to have arisen and is perhaps unclear as to the intended position?

AEMO has (possibly incorrectly) defined a "closed account" from an energy perspective as any account relating to the NMI where the retailer is no longer the FRMP. Any closed account retailer will get only the usage data for the period they were the FRMP for that NMI - as now, so there is no change. This is independent of when any LCCD falls.

I get the impression that there are different expectations as to the ability of a past retailer to gain access to historical usage data, depending on if their period as FRMP was more recent than the LCCD or earlier than the LCCD. AEMO's preference is to treat these as the same - both as "closed accounts" and both only get the usage data for the period they were FRMP (as now). Our position is that only the current FRMP is the Primary Data Holder and has access to all historical usage data from now back to the LCCD (if present).

On LCCD not populated
Reiterating that AEMO's position is that, as the maximum request for usage data is 2 years, if the FDO commenced on or after the 2 year anniversary of LCCD being automatically populated in MSATS, any LCCD that is not populated with a date can be treated (during the determination of the usage response payload) as having a value older than 2 years. This does not mean any population of non-populated LCCDs. It is purely for the calculation of the extent of the usage data to be made available to the FRMP. A non-populated LCCD will remain non-populated until populated by the retailer under the normal MSATS provisions. In this way there would be no need for DSB to make allowance for the handling of non-populated LCCDs as they would be treated as being older than 2 years.

On errors for closed account requests spanning an LCCD
Based on the above definition of "closed accounts", there is a risk, where the retailer has remained the same for successive consumers and the retailer is not the current FRMP (ie there has been the same past retailer at a NMI but an LCCD within the date range requested), AEMO would be happy to concede not returning an error and leave this responsibility with the retailer - as now (and ignore the LCCD). Our concern in ignoring the LCCD is that we are unclear as to which consumer to protect - and would seek guidance to either ignore the LCCD or trim the dates to favour the current consumer.

AEMO would be interested in any further discussion on these points.

@JamesMBligh
Copy link
Contributor

Responding to @johnAEMO...

Great discussion

Closed accounts and access to historical usage data.
...
AEMO has (possibly incorrectly) defined a "closed account" from an energy perspective as any account relating to the NMI where the retailer is no longer the FRMP. Any closed account retailer will get only the usage data for the period they were the FRMP for that NMI - as now, so there is no change. This is independent of when any LCCD falls.

I get the impression that there are different expectations as to the ability of a past retailer to gain access to historical usage data, depending on if their period as FRMP was more recent than the LCCD or earlier than the LCCD. AEMO's preference is to treat these as the same - both as "closed accounts" and both only get the usage data for the period they were FRMP (as now). Our position is that only the current FRMP is the Primary Data Holder and has access to all historical usage data from now back to the LCCD (if present).

A closed account means a previous customer holding the NMI. It doesn't mean the FRMP has changed. If I sell my house and the new owner stays with the same retailer then the LCCD is set but the FRMP doesn't change. If I access my usage then I will only be able to get my data as a "closed account" request and only for a period prior to the LCCD. It will be the current FRMP though.

This is why I have suggested that any usage request for a range prior to the LCCD should be limited to the period where the requesting retailer was the owning FRMP (which is the current case). I think this aligns with what you're saying John but correct me if I'm wrong.

At the risk of being a pedant, the Primary Data Holder is an CDR definition (ie. they are the one receiving the call which they then pass on to the Secondary Data Holder), but they don't have to be the current FRMP. It is true that they are usually the same but this is not necessarily true in all instances (for example, the case of closed accounts). The term Primary Data Holder and the term "current FRMP" are therefore not equal.

On LCCD not populated Reiterating that AEMO's position is that, as the maximum request for usage data is 2 years, if the FDO commenced on or after the 2 year anniversary of LCCD being automatically populated in MSATS, any LCCD that is not populated with a date can be treated (during the determination of the usage response payload) as having a value older than 2 years. This does not mean any population of non-populated LCCDs. It is purely for the calculation of the extent of the usage data to be made available to the FRMP. A non-populated LCCD will remain non-populated until populated by the retailer under the normal MSATS provisions. In this way there would be no need for DSB to make allowance for the handling of non-populated LCCDs as they would be treated as being older than 2 years.

That makes sense and clarifies the AEMO feedback. That is essentially what I was saying. If the LCCD isn't populated it just means an indeterminate time in the past (probably before the field went live).

On errors for closed account requests spanning an LCCD Based on the above definition of "closed accounts", there is a risk, where the retailer has remained the same for successive consumers and the retailer is not the current FRMP (ie there has been the same past retailer at a NMI but an LCCD within the date range requested), AEMO would be happy to concede not returning an error and leave this responsibility with the retailer - as now (and ignore the LCCD). Our concern in ignoring the LCCD is that we are unclear as to which consumer to protect - and would seek guidance to either ignore the LCCD or trim the dates to favour the current consumer.

What I would suggest is that, if the request range covers an LCCD date then the request should be truncated to the LCCD date even if the range is entirely in the period covered by the current FRMP. That covers the risk you are describing and would be simpler to implement as it is the same logic for if the range covers multiple FRMPs. This scenario would only occur if the current retailer made a mistake - either on the API call or in setting the LCCD - so I believe this would be the conservative way to handle this edge case.

@johnAEMO
Copy link

johnAEMO commented Feb 5, 2025

Question from our developers who are sizing the effort for LCCD:
Do we expect this to require a new version of the APIs to be maintained - particularly the Service Point APIs (with the additional LCCD field), or is this change small enough not to warrant it?
Obviously our preference would be for no versioning and not keeping 2 versions running in parallel but happy to take advice.

@AGL-ENERGY-CDR
Copy link

@JamesMBligh

A closed account means a previous customer holding the NMI. It doesn't mean the FRMP has changed. If I sell my house and the new owner stays with the same retailer then the LCCD is set but the FRMP doesn't change. If I access my usage then I will only be able to get my data as a "closed account" request and only for a period prior to the LCCD. It will be the current FRMP though.

This is why I have suggested that any usage request for a range prior to the LCCD should be limited to the period where the requesting retailer was the owning FRMP (which is the current case). I think this aligns with what you're saying John but correct me if I'm wrong.

Agreed on the first point that if the retailer remains FRMP following a move out/move in the account will be a "closed account".

On the second point, if the requesting retailer is not currently FRMP would this still return an error from the Get Services call following the introduction of the LCCD? Meaning that "closed accounts" would only be returned if the requesting retailer remains FRMP following an LCCD update.

Based on this logic could we see the edge case where the LCCD is changed multiple times within a two-year period, the same retailer remains FRMP, and as a result shares usage data from the latest LCCD update, spanning the now multiple customer's usage periods and risking a privacy issue? Or is there an existing part of the process that would stop this?

@johnAEMO with the non-populated LCCD did all retailers go live with the LCCD process on November 1st 2023, could this be a risk making the assumption all retailers commenced populating the LCCD from November 1st?

@johnAEMO
Copy link

johnAEMO commented Feb 6, 2025

@AGL-ENERGY-CDR

  • Regarding any retailer (current FRMP or not) requesting usage data for a date range wholly prior to the LCCD then their call would be a closed account call and limited to their period as FRMP - so agree there would be no change and your comment does align with what we are saying.
  • Not certain the point you are trying to make on the "second point" comment. By get Services do you mean get Service Points? If so then the DP is silent on the current error treatment of Service Points and DER APIs - the current treatment being that any retailer who is not the current FRMP will receive an error for those 4 APIs. The logic being that the data for those 4 APIs is the current configuration of service point and DER, so only the current FRMP is entitled to share access to those. Usage data is different as it can go back 2 years and span multiple retailers. We are proposing the same handling logic, only the current FRMP gets access to current usage data back to the LCCD/2 years maximum. All other retailers receive the same as now - for the period they were the FRMP.
  • Your edge case where there are multiple move in/outs over 2 years but the retailer stays the same, we can only see the most recent LCCD so we can only trim any request to that LCCD. That is why the existing clause where the retailer remains responsible to" ensure that the data requested... is not outside the bounds of... the specific CDR Consumer" is retained. If a request is made by the FRMP for a time prior to the LCCD, the retailer must ensure this is for the specific CDR consumer (as now).
  • last point, yes, my understanding is that the MSATS automated population of the LCCD was in place for all retailers from 1 November 2023.

@AGL-ENERGY-CDR
Copy link

Thank you all for the opportunity to discuss LCCD and provide feedback on Decision Proposal 361.

AGL agrees with James’ recommendation above that the standards language will need to be in updated to account for “LCCD truncation which only occurs when the LCCD date falls within the requested period”. However, there are two other scenarios that will need to be explicitly catered for, specifically, in circumstances where the LCCD request falls wholly before of the period request and where it falls wholly ahead of the period request to provide clarity around expectations. For example:

Under Section 1. Update Shared Responsibility, replace the current text with:

  1. “If LCCD value is held for the requested NMI, and the LCCD falls within the period request, then:

AEMO MUST limit the period of historical usage data returned to the period dating back to (and inclusive of) the LCCD…,” and so forth.

  1. “If LCCD value is held for the requested NMI, and the LCCD falls wholly before the period request, then:…

(the DSB and AEMO may need to determine the appropriate actions for AEMO to take in this scenario, as they may be different to the requirements under 1, above).

  1. If the LCCD value is held for the requested NMI, and the LCCD falls wholly ahead of the period request, then:...

the DSB and AEMO may need to determine the appropriate actions for AEMO to take in this scenario, as they may be different to the requirements under 1 and 2, above).

Benefit analysis

AGL notes that a risk and benefits analysis was performed as a part of the consultation process. As of 21 January 2025, only 18% of NMIs had the LCCD field populated since inception in November 2023. This raises the question as to whether the DSB should pause these reforms until a higher penetration of LCCD data is available in the market, and a greater uptake of energy CDR.

Complaint Handling

AGL is also seeking further clarification as to how the complaints management process is expected to be handled in this context. For example, if another retailer was FRMP up until 12/12/2024 but the LCCD was not populated and customer churned to AGL on the 13/12/2024, they would only be able to access historical usage data from the 13/12/2024. If the customer or their representative complains about not seeing historical data before this date, AGL would have limited avenues to resolve the issue as we do not have visibility of the LCCD in MSATs prior to this.

@johnAEMO
Copy link

johnAEMO commented Feb 7, 2025

AEMO thanks DSB for the extension to the deadline for this consultation which has allowed our developers sufficient time to confirm at a high level, their ability to meet an FDO of no earlier than November 2025.

@CDR-API-Stream CDR-API-Stream added Status: Feedback Period Closed The feedback period is complete and a final decision is being formulated and removed Status: Open For Feedback Feedback has been requested for the decision labels Feb 7, 2025
@CDR-API-Stream
Copy link
Contributor Author

Feedback for this consultation has closed. Thank you to everyone who contributed. Feedback will be reviewed and responded to before a data standard is made by the Chair.

@ConsumerDataStandardsAustralia ConsumerDataStandardsAustralia locked and limited conversation to collaborators Feb 7, 2025
@HemangCDR
Copy link
Contributor

HemangCDR commented Feb 13, 2025

Thank you everyone for your contributions to this consultation. The DSB have reviewed the feedback and acknowledge broad support for the change proceeding. Below are responses to feedback on the consultation:

lastConsumerChangeDate field

  • The type of lastConsumerChangeDate field will be changed to DateString as per @johnAEMO's feedback

FDO

  • The FDO will be changed to November as per feedback from @SelenaLiuEA and @johnAEMO. It will specifically be set to Y25 # 5 2025-11-10 as per the Obligation Date Schedule
  • Do we expect this to require a new version of the APIs to be maintained - particularly the Service Point APIs (with the additional LCCD field), or is this change small enough not to warrant it?

    • A new version of the impacted APIs will be needed as this will be a breaking change. We recommend setting the retirement date of older versions to the next immediate available FDO in the obligation date schedule which is Y26 # 1 2026-03-16

Sharing of historical usage data, error code and closed accounts

  • As mentioned by @JamesMBligh, LCCD is another granular option to FRMP boundary for usage data sharing. It provides a means to truncate or limit data sharing if the LCCD falls within a requested duration. Any request that falls fully prior or after the LCCD can be dealt as per present requirements
  • We agree that AEMO should provide as much usage data as possible, which is how it is currently done. There are existing obligation on the retailer to ensure data requests are for a period they know the customer owned the NMI which still apply
  • As noted in the proposal, the sharing of usage data for closed accounts is to be limited to retailer-NMI relationship. This can be achieved as requests for closed accounts would have periods prior to LCCD. The retailers of closed account have the obligation to ensure the period of data requested is within the bounds of the customer
  • We are minded to adopt the minor amendment proposed by @JamesMBligh's on updated standards language which addresses all of the above and caters for scenarios noted by @AGL-ENERGY-CDR. However, we may refine it pending further internal review. We will update this comment with the final proposed language.
  • The final proposed language that will replace existing text noted in the proposal in the Shared Responsibility->Additional Requirements section is noted below:

If the request is for electricity usage data, then:

  • If LCCD value is held for the requested NMI, and the LCCD falls within the period request, then:
    • AEMO MUST limit the usage data returned to the period dating back to (and inclusive of) the LCCD
  • If LCCD value is not held for the requested NMI, or the period requested is prior to LCCD, then:
    • AEMO MUST NOT respond with an error
    • AEMO MUST respond with the usage for the period that the Primary Data Holder can be definitively determined to be in control of the NMI
  • AEMO MUST NOT share data outside the period of control of the Primary Data Holder
    Note: the maximum historical period for usage data would be provided in accordance with rule 3.2 in Schedule 4, Part 3 of the CDR rules.
  • The primary Data Holder MUST ensure that the data requested and then shared with the Data Recipient is not outside the bounds of control of the specific CDR Consumer.

It has been amended based on the feedback to:

  • Clarify that AEMO will apply LCCD when LCCD falls within request period
  • Sharing will be limited to retailer-NMI relationship when LCCD is not present or the request is prior to LCCD date
  • Retain existing obligation on primary data holders to ensure they request data for a period they know the customer owned the NMI.

The DSB will also publish guidance article further clarifying the conditions and obligations on retailers and AEMO for usage data requests with LCCD.

Benefits

  • AGL notes that a risk and benefits analysis was performed as a part of the consultation process. As of 21 January 2025, only 18% of NMIs had the LCCD field populated since inception in November 2023. This raises the question as to whether the DSB should pause these reforms until a higher penetration of LCCD data is available in the market, and a greater uptake of energy CDR.

    • Our analysis suggests that 18% of NMIs equates to approx 2 million connections (and potential consumers) and is expected to grow each year. As such, delaying these changes would only delay the benefits to the significant and growing number of consumers who have the LCCD value set. Also noting that the proposed changes allow for data sharing to continue where LCCD is not available and the gradually population of LCCD will continue in parallel

Complaints management

  • AGL is also seeking further clarification as to how the complaints management process is expected to be handled in this context. For example, if another retailer was FRMP up until 12/12/2024 but the LCCD was not populated and customer churned to AGL on the 13/12/2024, they would only be able to access historical usage data from the 13/12/2024. If the customer or their representative complains about not seeing historical data before this date, AGL would have limited avenues to resolve the issue as we do not have visibility of the LCCD in MSATs prior to this.

    • Complaints handling would be managed as per existing processes and channels where consumer would notify their current retailer/DH who would assist them in resolving the issue, which may include pointing them to the appropriate complaints channel (which sit outside of CDR). In the example above, the retailer could help the customer set the LCCD value in collaboration with AEMO

The proposal with the above noted amendments will be put to the Chair for review and approval.

@HemangCDR
Copy link
Contributor

The comment has been updated with the final proposed standards language to be used in the Shared Responsibility->Additional Requirements section.

@CDR-API-Stream
Copy link
Contributor Author

Decision 361 has been approved by the Data Standards Chair:
Decision-361-LCCD-Change-Phase2.pdf

These changes will be incorporated into v1.34.0 of the Standards.

@CDR-API-Stream CDR-API-Stream added Status: Decision Made A determination on this decision has been made and removed Status: Feedback Period Closed The feedback period is complete and a final decision is being formulated labels Feb 26, 2025
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
Category: API A proposal for a decision to be made for the API Standards made Category: CX A proposal for a decision to be made for the User Experience Standards Industry: Electricity This proposal impacts the electricity industry sector Status: Decision Made A determination on this decision has been made
Projects
None yet
Development

No branches or pull requests