Add commodity balance epsilon to force candidate assets to activate

[tsmbland]

Description

As discussed in #1386, adding an epsilon to the commodity balance constraints to force candidate assets to dispatch. You can see in debug_commodity_balance_duals.csv for the "final with candidates" runs (where shadow prices come from), that a lot of the zeros are gone.

Doesn't eliminate all problems with zero shadow prices, as discussed below. But a step in the right direction.

Have based on #1389 as this also has a small affect on flows in the dispatch run with candidates, which would impact the market price calculation before #1389

I've played around with the value of COMMODITY_BALANCE_EPSILON_FOR_CANDIDATES, and it seems pretty sensitive to the value, as explained here. Adam seems ok to go ahead with this and keep an eye on it. There's also an alternative suggestion with a different approach for getting shadow prices, which we may get to. If we're happy to merge this then I'll open up a separate issue about those points.

This also seems to impact the circularity models. I've noticed that in the electricity-hydrogen loop, hydrogen has nonzero shadow prices even in seasons where it's not produced/consumed at all. Not sure why, or what these values really mean. With this PR, the shadow prices for hydrogen change in the seasons where it's not consumed/produced, and if anything look more sensible to my eyes (more consistent with the seasons in which it is produced). I guess this is a positive change then (a conversation with copilot suggests that the previous values were artifacts), but something to keep an eye on.

Fixes #1386

Type of change

• Bug fix (non-breaking change to fix an issue)
• New feature (non-breaking change to add functionality)
• Refactoring (non-breaking, non-functional change to improve maintainability)
• Optimization (non-breaking change to speed up the code)
• Breaking change (whatever its nature)
• Documentation (improve or add documentation)

Key checklist

• All tests pass: $ cargo test
• The documentation builds and looks OK: $ cargo doc
• Update release notes for the latest release if this PR adds a new feature or fixes a bug
  present in the previous release

Further checks

• Code is commented, particularly in hard-to-understand areas
• Tests added that prove fix is effective or that feature works

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 89.42%. Comparing base (6493671) to head (932fbdc).

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1387      +/-   ##
==========================================
+ Coverage   89.39%   89.42%   +0.03%     
==========================================
  Files          58       58              
  Lines        8352     8367      +15     
  Branches     8352     8367      +15     
==========================================
+ Hits         7466     7482      +16     
  Misses        564      564              
+ Partials      322      321       -1     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[tsmbland]

Previously, the following models finished with the warning introduced in #1365:

• missing_commodity
• muse1_default
• two_outputs
• two_regions

Now, it's the following:

• circularity
• circularity_npv
• muse1_default
• two_outputs
• two_regions

So, fixed missing_commodity (which makes sense), haven't fixed up muse1_default, two_regions and two_outputs, and have now "broken" circularity and circularity_npv (although all models still run to completion)

I think for muse1_default and two_regions, existing assets may be genuinely unattractive because of the increase in CO2 levy, which makes sense. For two_outputs, circularity and circularity_npv, we have processes with multiple output commodities, which we know can cause zero shadow prices, in a way which isn't addressed by this PR.

Update: after rebasing, muse1_default and two_regions now run without the warning

[Copilot]

Pull request overview

This PR introduces a small epsilon adjustment to commodity balance constraints in dispatch optimisation to encourage candidate assets to become active, with the aim of reducing zero shadow prices in cases where a commodity would otherwise be unused in a given year.

Changes:

• Add an epsilon term to commodity balance constraint lower bounds when candidate assets can produce the commodity, to force non-zero activity and improve dual (shadow price) signals.
• Thread candidate asset information into constraint construction so the epsilon can be applied conditionally.
• Update regression test fixtures (prices/flows/duals/objectives/assets/capacities) to reflect the new optimisation behaviour.

Reviewed changes

Copilot reviewed 18 out of 18 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
src/simulation/optimisation/constraints.rs	Adds candidate-aware epsilon logic to commodity balance constraint lower bounds.
src/simulation/optimisation.rs	Passes candidate_assets into constraint creation.
tests/data/simple/debug_solver.csv	Updates expected objective values under the new constraints.
tests/data/simple/debug_commodity_balance_duals.csv	Updates expected commodity balance duals (fewer zero duals).
tests/data/simple/debug_appraisal_results_time_slices.csv	Updates expected appraisal results time-slice outputs affected by dispatch changes.
tests/data/missing_commodity/commodity_prices.csv	Updates expected commodity prices under the new dual behaviour.
tests/data/missing_commodity/commodity_flows.csv	Updates expected commodity flows under the new dispatch solution.
tests/data/missing_commodity/assets.csv	Updates expected asset set ordering/IDs for the scenario outputs.
tests/data/missing_commodity/asset_capacities.csv	Updates expected capacities consistent with the new solution.
tests/data/circularity/commodity_prices.csv	Updates expected prices affected by the epsilon-driven activation.
tests/data/circularity/assets.csv	Updates expected asset listing/IDs for circularity scenario outputs.
tests/data/circularity/asset_capacities.csv	Updates expected capacities consistent with circularity scenario outputs.
tests/data/circularity_npv/commodity_prices.csv	Updates expected prices for the NPV variant scenario.
tests/data/circularity_npv/commodity_flows.csv	Updates expected flows for the NPV variant scenario.
tests/data/circularity_npv/assets.csv	Updates expected asset listing/IDs for the NPV variant scenario.
tests/data/circularity_npv/asset_capacities.csv	Updates expected capacities for the NPV variant scenario.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[tsmbland]

Slightly more controversial, and @alexdewar would be good to get your take when you're back. This seems to work though, without any obvious issues, so long as COMMODITY_BALANCE_EPSILON_FOR_CANDIDATES is set to a reasonable value. I imagine the relationship with candidate_asset_capacity is important (currently they're two orders of magnitude apart, which may be a sweet spot), so me might want to think about hardcoding that parameter, or specifying this parameter relative to that one.

I need to merge this now without review, but post-hoc reviews welcome