Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reduce time of testing #378

Merged
merged 14 commits into from
Apr 30, 2024
Merged

Reduce time of testing #378

Show file tree
Hide file tree
Changes from 9 commits
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
474145d
update Minimize Unserved Load Tests for speed
zolanaj Apr 24, 2024
3b7dd49
Update outage.json
zolanaj Apr 24, 2024
ed2e34a
update additional minimize unserved load test
zolanaj Apr 24, 2024
f685523
Merge branch 'develop' into simplify-tests
zolanaj Apr 24, 2024
1d6c088
Temporarily disable ubuntu tests because always timing out
Bill-Becker Apr 23, 2024
b9f87e9
simplify emissions + RE testset
zolanaj Apr 24, 2024
b703384
rm "Imported Xpress Test Suite" testset marker (but keep tests)
zolanaj Apr 24, 2024
1b70783
Revert "rm "Imported Xpress Test Suite" testset marker (but keep tests)"
zolanaj Apr 24, 2024
6d08397
Update CHANGELOG.md
zolanaj Apr 25, 2024
50c8d4b
rm minimize unserved load comment
zolanaj Apr 30, 2024
e68ac12
move test edits to .json files
zolanaj Apr 30, 2024
6747804
simplify run_reopt() calls in minimize unserved load tests
zolanaj Apr 30, 2024
296bbd6
more specific test value for outage test
zolanaj Apr 30, 2024
ffcc062
Update CHANGELOG.md
zolanaj Apr 30, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion .github/workflows/CI.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,8 @@ jobs:
matrix:
julia-version: ['1.8']
julia-arch: [x64]
os: [ubuntu-latest, windows-latest, macOS-11]
# os: [ubuntu-latest, windows-latest, macOS-11]
os: [windows-latest, macOS-11]
zolanaj marked this conversation as resolved.
Show resolved Hide resolved

steps:
- uses: actions/checkout@v2
Expand Down
4 changes: 4 additions & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,10 @@ Classify the change according to the following categories:
### Deprecated
### Removed

## Develop 2024-04-24
### Changed
- Updated test sets "Emissions and Renewable Energy Percent" and "Minimize Unserved Load" to decrease computing time.

## v0.45.0
### Fixed
- Added `export_rate_beyond_net_metering_limit` to list of inputs to be converted to type Real, to avoid MethodError if type is vector of Any.
Expand Down
102 changes: 75 additions & 27 deletions test/runtests.jl
View file
Open in desktop
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@zolanaj can we remove the following uncommented line?

"# @testset "Minimize Unserved Load" is too slow with Cbc (killed after 8 hours)" (line 220)

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done in 50c8d4b. Thanks for catching!

Original file line number Diff line number Diff line change
Expand Up @@ -909,18 +909,47 @@ else # run HiGHS tests
end

@testset "Minimize Unserved Load" begin

d = JSON.parsefile("./scenarios/outage.json")
d["ElectricLoad"]["loads_kw"] = ones(8760)*200.0
d["ElectricLoad"]["loads_kw"][5100:5109] .= 410.0
d["ElectricLoad"]["loads_kw"][5200:5209] .= 410.0
d["ElectricLoad"]["critical_load_fraction"] = 0.5
d["PV"]["existing_kw"] = 0.0
d["PV"]["min_kw"] = 100.0
d["PV"]["max_kw"] = 100.0
d["CHP"]["min_kw"] = 100.0
d["CHP"]["max_kw"] = 100.0
d["Generator"]["existing_kw"] = 0.0
d["Generator"]["min_kw"] = 100.0
d["Generator"]["max_kw"] = 100.0
d["ElectricStorage"]["min_kw"] = 20
d["ElectricStorage"]["max_kw"] = 20
d["ElectricStorage"]["min_kwh"] = 50
d["ElectricStorage"]["max_kwh"] = 50
d["Financial"]["microgrid_upgrade_cost_fraction"] = 0.0
s = Scenario(d)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@zolanaj just curious, was there a compute time advantage to creating the Scenario and REoptInputs first, rather than just running run_reopt(m,d) ?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think there's any difference in computing - run_reopt(m, d) does the same sequence of commands. I've always used the Scenario and REoptInputs creation upfront for debugging purposes; it's fewer lines of code in the testing though so I can convert it.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done in 6747804 for the updated tests only; we can probably do this more widely but I don't see any reason for a computational difference so that's low priority.

p = REoptInputs(s)
m = Model(optimizer_with_attributes(HiGHS.Optimizer, "output_flag" => false, "log_to_console" => false, "mip_rel_gap" => 0.01, "presolve" => "on"))
results = run_reopt(m, "./scenarios/outage.json")

@test results["Outages"]["expected_outage_cost"] ≈ 0
@test sum(results["Outages"]["unserved_load_per_outage_kwh"]) ≈ 0
results = run_reopt(m, p)
@test results["Outages"]["expected_outage_cost"] ≈ 0 atol=0.1
@test sum(results["Outages"]["unserved_load_per_outage_kwh"]) ≈ 0 atol=0.1
@test value(m[:binMGTechUsed]["Generator"]) ≈ 1
@test value(m[:binMGTechUsed]["CHP"]) ≈ 1
@test value(m[:binMGTechUsed]["PV"]) ≈ 1
@test value(m[:binMGStorageUsed]) ≈ 1
@test results["Financial"]["lcc"] ≈ 6.83633907986e7 rtol=0.01


# Increase cost of microgrid upgrade and PV Size, PV not used and some load not met
d["Financial"]["microgrid_upgrade_cost_fraction"] = 0.3
d["PV"]["min_kw"] = 200.0
d["PV"]["max_kw"] = 200.0
s = Scenario(d)
p = REoptInputs(s)
m = Model(optimizer_with_attributes(HiGHS.Optimizer, "output_flag" => false, "log_to_console" => false, "mip_rel_gap" => 0.01, "presolve" => "on"))
results = run_reopt(m, p)
@test value(m[:binMGTechUsed]["PV"]) ≈ 0
@test sum(results["Outages"]["unserved_load_per_outage_kwh"]) > 0
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@zolanaj any reason to not set this to the value of 24.16 instead of just > 0 ?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this one is stable enough to make 24.16. I usually stay away from fixed values when a 1% mip tolerance is involved, but I think this one should be ok fo now. Good catch!

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done in 296bbd6


#=
Scenario with $0.001/kWh value_of_lost_load_per_kwh, 12x169 hour outages, 1kW load/hour, and min_resil_time_steps = 168
- should meet 168 kWh in each outage such that the total unserved load is 12 kWh
Expand All @@ -943,13 +972,32 @@ else # run HiGHS tests
@test results["Financial"]["lcc"] ≈ 8.6413594727e7 rtol=0.001

# Scenario with generator, PV, wind, electric storage
d = JSON.parsefile("./scenarios/outages_gen_pv_wind_stor.json")
d["ElectricLoad"]["loads_kw"] = ones(8760)*200.0
d["ElectricLoad"]["loads_kw"][5100:5119] .= 425.0
d["ElectricLoad"]["critical_load_fraction"] = 0.5
d["PV"]["existing_kw"] = 0.0
d["PV"]["min_kw"] = 100.0
d["PV"]["max_kw"] = 100.0
d["Wind"]["min_kw"] = 100.0
d["Wind"]["max_kw"] = 100.0
d["Generator"]["min_kw"] = 100.0
d["Generator"]["max_kw"] = 100.0
d["ElectricStorage"]["min_kw"] = 100
d["ElectricStorage"]["max_kw"] = 100
d["ElectricStorage"]["min_kwh"] = 100
d["ElectricStorage"]["max_kwh"] = 100
d["Site"]["min_resil_time_steps"] = 1
s = Scenario(d)
p = REoptInputs(s)
m = Model(optimizer_with_attributes(HiGHS.Optimizer, "output_flag" => false, "log_to_console" => false, "presolve" => "on"))
results = run_reopt(m, "./scenarios/outages_gen_pv_wind_stor.json")
results = run_reopt(m, p)
@test value(m[:binMGTechUsed]["Generator"]) ≈ 1
@test value(m[:binMGTechUsed]["PV"]) ≈ 1
@test value(m[:binMGTechUsed]["Wind"]) ≈ 1
@test results["Outages"]["expected_outage_cost"] ≈ 446899.75 atol=1.0
@test results["Financial"]["lcc"] ≈ 6.71661825335e7 rtol=0.001
@test results["Outages"]["expected_outage_cost"] ≈ 1.296319791276051e6 atol=1.0
@test results["Financial"]["lcc"] ≈ 4.8046446434e6 rtol=0.001

end

@testset "Outages with Wind and supply-to-load no greater than critical load" begin
Expand Down Expand Up @@ -1975,41 +2023,41 @@ else # run HiGHS tests
end

if i == 1
@test results["PV"]["size_kw"] ≈ 60.12 atol=1e-1
@test results["PV"]["size_kw"] ≈ 59.7222 atol=1e-1
@test results["ElectricStorage"]["size_kw"] ≈ 0.0 atol=1e-1
@test results["ElectricStorage"]["size_kwh"] ≈ 0.0 atol=1e-1
@test results["Generator"]["size_kw"] ≈ 21.52 atol=1e-1
@test results["Generator"]["size_kw"] ≈ 9.13 atol=1e-1
@test results["Site"]["total_renewable_energy_fraction"] ≈ 0.8
@test results["Site"]["total_renewable_energy_fraction_bau"] ≈ 0.147576 atol=1e-4
@test results["Site"]["lifecycle_emissions_reduction_CO2_fraction"] ≈ 0.58694032 atol=1e-4
@test results["Financial"]["breakeven_cost_of_emissions_reduction_per_tonne_CO2"] ≈ 355.8 atol=1
@test results["Site"]["annual_emissions_tonnes_CO2"] ≈ 11.64 atol=1e-2
@test results["Site"]["annual_emissions_from_fuelburn_tonnes_CO2"] ≈ 7.0605
@test results["Site"]["total_renewable_energy_fraction_bau"] ≈ 0.148375 atol=1e-4
@test results["Site"]["lifecycle_emissions_reduction_CO2_fraction"] ≈ 0.57403012 atol=1e-4
@test results["Financial"]["breakeven_cost_of_emissions_reduction_per_tonne_CO2"] ≈ 332.4 atol=1
@test results["Site"]["annual_emissions_tonnes_CO2"] ≈ 11.85 atol=1e-2
@test results["Site"]["annual_emissions_from_fuelburn_tonnes_CO2"] ≈ 7.427
@test results["Site"]["annual_emissions_from_fuelburn_tonnes_CO2_bau"] ≈ 0.0
@test results["Financial"]["lifecycle_emissions_cost_climate"] ≈ 8315.69 atol=1
@test results["Site"]["lifecycle_emissions_tonnes_CO2"] ≈ 232.85
@test results["Site"]["lifecycle_emissions_from_fuelburn_tonnes_CO2"] ≈ 141.21
@test results["Financial"]["lifecycle_emissions_cost_climate"] ≈ 8459.45 atol=1
@test results["Site"]["lifecycle_emissions_tonnes_CO2"] ≈ 236.95
@test results["Site"]["lifecycle_emissions_from_fuelburn_tonnes_CO2"] ≈ 148.54
@test results["Site"]["lifecycle_emissions_from_fuelburn_tonnes_CO2_bau"] ≈ 0.0
@test results["ElectricUtility"]["annual_emissions_tonnes_CO2_bau"] ≈ 28.186 atol=1e-1
@test results["ElectricUtility"]["lifecycle_emissions_tonnes_CO2_bau"] ≈ 563.72
@test results["ElectricUtility"]["annual_emissions_tonnes_CO2_bau"] ≈ 27.813 atol=1e-1
@test results["ElectricUtility"]["lifecycle_emissions_tonnes_CO2_bau"] ≈ 556.26
elseif i == 2
#commented out values are results using same levelization factor as API
@test results["PV"]["size_kw"] ≈ 106.13 atol=1
@test results["ElectricStorage"]["size_kw"] ≈ 21.58 atol=1 # 20.29
@test results["ElectricStorage"]["size_kwh"] ≈ 166.29 atol=1
@test results["ElectricStorage"]["size_kw"] ≈ 20.09 atol=1 # 20.29
@test results["ElectricStorage"]["size_kwh"] ≈ 170.94 atol=1
@test !haskey(results, "Generator")
# Renewable energy
@test results["Site"]["renewable_electricity_fraction"] ≈ 0.78586 atol=1e-3
@test results["Site"]["renewable_electricity_fraction_bau"] ≈ 0.132118 atol=1e-3 #0.1354 atol=1e-3
@test results["Site"]["annual_renewable_electricity_kwh_bau"] ≈ 13211.78 atol=10 # 13542.62 atol=10
@test results["Site"]["annual_renewable_electricity_kwh_bau"] ≈ 13308.5 atol=10 # 13542.62 atol=10
@test results["Site"]["total_renewable_energy_fraction_bau"] ≈ 0.132118 atol=1e-3 # 0.1354 atol=1e-3
# CO2 emissions - totals ≈ from grid, from fuelburn, ER, $/tCO2 breakeven
@test results["Site"]["lifecycle_emissions_reduction_CO2_fraction"] ≈ 0.8 atol=1e-3 # 0.8
@test results["Financial"]["breakeven_cost_of_emissions_reduction_per_tonne_CO2"] ≈ 460.7 atol=1e-1
@test results["Financial"]["breakeven_cost_of_emissions_reduction_per_tonne_CO2"] ≈ 491.5 atol=1e-1
@test results["Site"]["annual_emissions_tonnes_CO2"] ≈ 11.662 atol=1
@test results["Site"]["annual_emissions_tonnes_CO2_bau"] ≈ 58.3095 atol=1
@test results["Site"]["annual_emissions_from_fuelburn_tonnes_CO2"] ≈ 0.0 atol=1 # 0.0
@test results["Financial"]["lifecycle_emissions_cost_climate"] ≈ 8401.1 atol=1
@test results["Financial"]["lifecycle_emissions_cost_climate"] ≈ 8397.85 atol=1
@test results["Site"]["lifecycle_emissions_tonnes_CO2_bau"] ≈ 1166.19 atol=1
@test results["Site"]["lifecycle_emissions_from_fuelburn_tonnes_CO2"] ≈ 0.0 atol=1 # 0.0
@test results["Site"]["lifecycle_emissions_from_fuelburn_tonnes_CO2_bau"] ≈ 0.0 atol=1 # 0.0
Expand Down
Loading
Loading