Track technology and upgrade tallies over time. Expose on website.

[lyricnz]

There's already a function to generate this - get_tech_and_upgrade_breakdown() - but it just emits output to the console. It does take several minutes to run though, so shouldn't be included in every cycle.

All tech breakdown: 14652833
Counter({'FTTN': 4611720,
         'HFC': 2886497,
         'FTTP': 2793945,
         'FTTC': 1696950,
         'WIRELESS': 969169,
         'SATELLITE': 654873,
         'FTTB': 650671,
         'NULL': 389008})
All upgrade breakdown (excluding tech=FTTP): 11858888
Counter({'NULL_NA': 8339191,
         'FTTP_SA': 2030854,
         'UNKNOWN': 1351746,
         'FTTP_NA': 130383,
         'FW_NA': 4215,
         'FW_CT': 1767,
         'FTTN_CT': 552,
         'SAT_CT': 151,
         'FTTC_NA': 16,
         'FTTN_NA': 12,
         'FTTC_CT': 1})

[LukePrior]

Yeah this would be a very interesting statistic to watch over time, we could also partially backfill using the git history.

[lyricnz]

I'm experimenting with running this code on old checkouts of /results to generate historical data.

Roughly:

    while co_date > datetime(2021, 7, 1):
        co_date -= timedelta(days=7)
        cmd = f"git checkout `git rev-list -n 1 --before=\"{co_date.strftime('%Y-%m-%d %H:%M')}\" main`"
        get_tech_and_upgrade_breakdown()
        os.rename("results/breakdown.json", f"results/breakdown-{co_date.strftime('%Y-%m-%d')}.json")

[lyricnz]

That's being somewhat stymied by the API changes a couple of months back (combined-suburbs.json etc). Will change to a pure geojson verison...

[lyricnz]

Running against git history - takes about 2 minutes per cycle (currently running every 7 days, starting now moving backwards)

[lyricnz]

[lyricnz]

Pushed PR and output https://github.com/LukePrior/nbn-upgrade-map/pull/266/files#diff-bbd7c3dbacf5aed3a6ba4d4847412e11a7b8098f831e6e660560a2823365d55f

[lyricnz]

FWIW project got to 100% address coverage on 31 July

[LukePrior]

Oh woah this looks great, might be a good idea to have an option to only include data from July 31st onwards.

Also maybe fixing the date so it will always be one week apart and new data can be added by just generating the missing weeks and appending to the file.

[lyricnz]

Changed the processing to go old-to-new, starting on 23/5/23, and using 00:00 timestamp.

[LukePrior]

In terms of getting it onto the site, I might just quickly make a new page /stats or something and generate a few charts using Chart.js

[LukePrior]

Mockup I made with ChatGPT: https://lukeprior.github.io/nbn-upgrade-map/stats

[lyricnz]

I'll fix the code to make it update the same file, so it's rerunnable, without lots of little files. Give me 30 mins

[LukePrior]

The breakdown.json file seems to have some issues at the start with dates and values.

[lyricnz]

Yup, I think I broke it while reformatting/merging it. I'll just nuke it and rebuild it.
Edit: Done

[LukePrior]

Merged, will update site then need to make GHA for automation each week.

[lyricnz]

I don't know if the git commands (in particular the checkout) will work within GHA. We may need a different version of the code which just appends progress.json with the current checkout values.

[lyricnz]

Maybe a table on the stats page, for easier reading without lots of mouseover?