Investment issues (umbrella)

[tsmbland]

Main issues:

• New approach for tranching/divisible assets #1358 (see also Change the way LCOX appraisal tool works #1199 and New tranching approach #1357)We don't have an agreed upon approach for capacity/tranche size selection. Unclear whether capacities should be defined upfront or dynamic (there are software design implications here, and I'd advise against mixing approaches to keep things relatively simple). Unclear how this should interact with divisible assets. Unclear what the user interface should be.
• Problem treating tranches as separate assets #1383 (see also Change LCOX to use same optimisation as NPV #1319) We're appraising capacities in each tranche as if they're entirely separate assets, rather than part of a whole. Ignoring what's been selected so far can lead to inefficient use of availability constraints, potentially leading to higher capacity than we need
• Investments thrown by zero/questionable shadow prices #1384 Zero shadow prices can lead to negative activity coefficients which can prevent assets which would probably be feasible from dispatching (see also Zero shadow price for over-supplied commodities #992)
• General indecision about divisible assets (which, to remind, mostly exist for the purposes of allowing partial decommissioning). Should we have a "parent" asset made up of lots of discrete units with the pre-defined capacities, something that allows for more continuous capacities (Allow for "divisible" assets with continuous capacity rather than discrete #1269), or scrap this completely and just have a collection of independent smaller assets. What should the user interface be?
• Appraisal of divided assets is inefficient #1209 Related, to the above and point 1, the current approach of splitting assets into small units for re-appraisal can lead to many appraisal rounds which can be inefficient. Should we allow appraising multiple units simultaneously?
• Potential for overcapacity when using processes with multiple output commodities #1313 No good solution yet for processes with multiple output commodities (also related to zero shadow prices)

See sub issues below, but also look at any linked pull requests/issues for full context.

[tsmbland]

I had Claude try to summarise the issues, and it's done a reasonable job (albeit far too long). I've edited a few things:

---

MUSE2 Investment Algorithm: Issues Report

This report consolidates the discussion across #1382 and its linked issues/PRs (#1358, #1383, #1384, #1199, #1201, #1209, #1269, #1313, #1347, #992, #1123, #1174, and PRs #1319 and #1357).

Rather than mirror the existing issue split, I've reorganised the discussion around five interlocking themes, because most of the open problems are symptoms of two or three deeper root causes that recur in different guises.

TL;DR — the core tension

Almost every investment problem traces back to a single architectural decision: capacity is selected incrementally, in fixed-size chunks ("tranches"), and each chunk is appraised as a standalone asset in isolation from what came before. This interacts badly with (a) activity/availability constraints that span timeslices, and (b) commodity prices that are computed once per milestone year rather than at equilibrium. The result is a cluster of failures — over-investment, infeasible exits, and assets that "choose" not to dispatch. There is broad agreement that the current demand-limiting-capacity (DLC) tranching approach "needs to go", but no agreed replacement yet.

---

Theme 1 — Capacity/tranche sizing: no agreed approach

Issues: #1358, #1199, PR #1357

The problem

The investment loop adds capacity in discrete tranches. Historically, tranche size = capacity_limit_factor × demand-limiting capacity (DLC), where DLC is the capacity needed to meet demand in the most restrictive (worst) timeslice. Two distinct failure modes were identified:

• The old VoLL approach over-invested. The LCOX tool used Value of Lost Load to serve any demand a candidate couldn't, pushing installed capacity to its maximum. Worse, the capacity upper bound wasn't updated between tranches, so large unneeded capacities could be chosen in later rounds (the original motivation for #1199).
• The DLC approach picks capacities that are too large, because it keys off the worst timeslice (per Adam in PR New tranching approach #1357).

The unresolved design question

The central, still-open decision is upfront vs. dynamic capacity sizing:

• Upfront (static): capacities defined in input data, fixed for all years. PR #1357 implements this via a new capacity_granularity parameter in processes.csv, defaulting to default_capacity_granularity_factor / capacity_to_activity (factor hardcoded to 10/100, described by both authors as "completely arbitrary").
• Dynamic (demand-based): capacities scale with remaining demand at runtime. Adam (New tranching approach #1357) "think[s] we do want dynamic remaining-demand-based capacity selection. Problem is we haven't thought of a robust way to do this yet."

Strong steer from the discussion: don't mix the two approaches ("I'd probably lean into one direction or the other, rather than a mixed approach" — Tom). The factors that a robust dynamic method would need to weigh were enumerated in #1357:

• Remaining demand by timeslice
• Availability by timeslice (and seasonal/annual)
• Timeslice level of the demanded commodity
• Dispatch economics by timeslice based on previous-MSY shadow prices (is the option even close to economic?)

Sub-problem: zero/near-zero availability

How to compute DLC / tranche size when availability is zero or near-zero in a timeslice is unresolved (referenced repeatedly, e.g. #1199 comment). The plan is to add a solar PV test case to the simple model with one timeslice at near-zero and one at exactly-zero availability, made deliberately competitive, to observe whether it gets adopted as it should.

Open UX questions

• Should capacity_granularity be mandatory or have a default? (Adam favours mandatory with good docs; Tom worries about users picking silly values / more input = more mistakes.)
• Should the parameter live on the process or the commodity? (Both favour process.)
• Should the override field take Capacity units or Activity units (the latter being consistent with the global default but "really confusing")?

---

Theme 2 — Tranches appraised in isolation ("the fungibility bug")

Issues: #1383, PR #1319

This is arguably the deepest conceptual problem, and the clearest candidate for "consolidation" because it underlies several of the apparent failures in Themes 1 and 3.

The problem

Each tranche is appraised as an independent asset with its own per-timeslice and annual activity limits, ignoring capacity already selected. But the DLC calculation implicitly assumes activity is fully fungible across capacity — which is only true once all the tranches are pooled together for dispatch.

Worked example (from #1383 / #1319 comment)

• 3 equal timeslices; output coefficient 1; annual activity limit 2/3.
• Demand = 100 per timeslice (300 total). DLC = max(300, 300, 300, 450) = 450. So 450 capacity is provably sufficient.
• Split into three 150-capacity tranches. Each, in isolation, can serve ≤50 per timeslice and ≤100 annually.
• Tranches 1 and 2 each run flat-out in the two most profitable timeslices (TS1, TS2), exhausting them. By tranche 3, 100 units of demand remain in TS3, but a single tranche can only serve 50.
• Result: after investing the full 450 capacity, the algorithm still sees unmet demand and exits — even though 450 was known to be sufficient. Allowing a 4th tranche would "fix" it but over-invest beyond what's needed.

The same mechanism was demonstrated with the real windturbine process in muse1_default (40% annual utilisation, corner-solution dispatch leaving two timeslices unfillable — #1319 comment).

Why it matters

There's a fundamental mismatch between the DLC (which is correct for the pooled asset) and what the tranching approach can actually realise. Patching it via either route is unattractive:

• Changing the DLC calc or allowing >DLC investment → over-installs capacity that ends up under-utilised in dispatch.

Proposed direction

Reframe tranching so a new tranche accounts for capacity already selected, allowing activity to be allocated compatibly with the overall asset rather than the chunk in isolation. This likely needs to differ for divisible vs. non-divisible assets (see Theme 4). For non-divisible assets specifically, the idea (#1358) is to add capacity chunks to the growing asset, appraise the full capacity, and compute the marginal LCOX/NPV of the added capacity — though whether a meaningful marginal metric can be defined is an open question.

---

Theme 3 — Zero/questionable shadow prices break dispatch & appraisal

Issues: #1384, #992, #1347, #1313

These are grouped because they share one root cause: prices are computed once per milestone year (not at equilibrium), and can legitimately be zero/negative, which then corrupts the next year's appraisal.

3a. Zero shadow prices → negative activity coefficients (#1384)

A commodity with a zero (or -0.0) shadow price in one year produces negative activity coefficients the following year, so demand that should be served isn't. Concrete failures found in example models:

• circularity: GASNAT has zero summer price in 2030 (no demand); in 2040 there is summer demand but coefficients are negative → unmet.
• two_outputs: GASPRD zero summer price in 2020 (over-supplied as a side product of OAGRSV) → negative coefficients in 2030. (fixed by Allow asset selection to continue with unmet demand #1365)
• missing_commodity: BIOPEL very small (not zero) summer price in 2030 → negative coefficients in 2040. (fixed by Allow asset selection to continue with unmet demand #1365)

Tom's key observation: "These all seem like fairly normal scenarios", which undercuts the idea that the unmet-demand fallback (#1347) should be a niche feature behind a warning. Importantly, the epsilon trick used previously only helps when coefficients are zero; here they're negative, which is a different problem.

3b. Zero prices for over-supplied multi-output commodities (#992)

When a process has multiple outputs, a side-product can be over-supplied (because the asset runs to meet the other commodity's demand), driving its shadow price to zero/negative. Adam's position: this is economically expected (the system doesn't want more of it), and indeed mirrors real UK/EU electricity markets. Longer-term mitigations floated: flexible processes, and user-selectable market pricing methods (shadow vs. cost-based-with-CAPEX/FIXOM).

3c. Investment fails when only remaining options won't dispatch (#1347)

When already-selected assets don't meet demand and every remaining option produces zero activity in appraisal (so gets a None metric and is skipped), the simulation terminates. Consensus: an unprofitable asset is better than terminating ("better to have an unprofitable one than not meet demand at all… like choosing to starve because you don't fancy the hamburger"). Four options were floated:

1. Bump the CoI price until assets dispatch — but this fails when the shadow price is zero, and fails entirely if the user set shadow pricing.
2. Pick assets arbitrarily until demand is met via partial dispatch.
3. Compute the price increase each asset would need.
4. Compute the metric assuming full utilisation — tried in PR Add fallback code to calculate metrics for non-dispatched options #1363 but Tom found valid reasons it's wrong.

Current state: No principled solution. The pragmatic fix (#1365): if appraisal can't meet demand, issue a warning and continue to the dispatch step anyway — which often works, because already-selected assets frequently have spare capacity in the needed timeslices that they "chose" not to use during appraisal. Parked in the backlog as "not especially pressing".

3d. Over-capacity for multi-output processes (#1313)

Because investment considers only primary commodity outputs, commodities produced as side-products can get too much capacity. Two ideas: (a) let agents consider multiple commodities simultaneously (powerful but "quite fiddly"); (b) reuse the "balancing" mechanism built for circularities (#999) — invest per-commodity in isolation, then run a global multi-commodity optimisation to trim excess capacity. The latter could also benefit trade (#523).

---

Theme 4 — Divisible assets: undecided model & inefficient implementation

Issues: #1269, #1209, #1123

Divisible assets exist mainly to allow partial decommissioning. There's general indecision about what they should fundamentally be.

4a. What should a divisible asset be? (#1269, #1358)

Three competing models:

1. A "parent" asset made of many discrete units of predefined unit_size.
2. A continuous-capacity asset that can be partially decommissioned by any amount, not just multiples of a unit (#1269) — interface unclear (e.g. unit_size = 0 as a sentinel? a keyword?).
3. Scrap the concept and use a collection of independent smaller assets.

PR #1357's direction unifies divisibility with the granularity parameter: a new is_divisible flag, with unit_size = capacity_granularity for divisible assets. For divisible assets the chunks are independently-decommissionable real objects; for non-divisible they fuse into one object that must be decommissioned all at once.

"Option 1 vs option 2" was explicitly debated (do away with divisible assets and rely on unit_size/granularity, vs. keep the parent/bundling approach):

• Keep bundling (chosen for now): avoids huge numbers of tiny identical assets in output, lower dispatch memory/time, the appraise-one-shortcut works, and — critically — pooled capacity uses availability limits more efficiently (Theme 2). Decision: "internally we should start with a version of (2)… because it's closest to how the code is already set up", regardless of how it's presented to users.
• Independent small assets: simpler, fewer params, capacities don't change over time — but heavier and loses the efficiency shortcut.

An open thought: even for divisible assets, whether per-unit isolation is correct "depends whether the individual objects are coordinated (e.g. wind turbines in a wind farm) or not (e.g. gas boilers in homes)".

4b. Appraisal of divided assets is inefficient (#1209)

Since #1092, only one child is appraised per round, but small unit_size relative to total capacity → many appraisal rounds, each retaining only a sliver of capacity. Desired: decide the aggregate capacity in one pass (via the parent), then decommission surplus children — analogous to how new assets let the capacity variable float within limits. Blocked on #1199/#1201.

4c. Remove child assets entirely (#1123)

Longer-term goal: use the parent asset everywhere instead of evaluating identical children separately. Blockers: representing a partially-decommissioned parent (a separate object in the decommissioned pool?), output-file formats (#1042 — children can have different decommission years), and parent appraisal/partial-decommissioning (#1045).

---

Theme 5 — Appraisal metrics, dispatch determinism & multi-round inefficiency

Issues: #1199, #1201, #1174

5a. Unifying NPV and LCOX appraisal (#1199 ✅ closed, #1201 open (see #1379))

Resolved direction (PR #1319, merged): the dispatch mini-optimisation is now identical for NPV and LCOX — both assume an asset runs as it would under profit-maximisation in the previous MSY's price conditions (the NPV approach, which is the "correct" one). The difference is only in the downstream metric: the LCOX cost-index excludes the CoI, while NPV includes it. The dispatch step itself includes the CoI in both cases. This unification was the agreed "for now" step; the harder capacity/tranche questions were deferred.

For NPV, #1201 proposes replacing the current profitability-index-for-candidates / total-annualised-surplus-for-existing split (which means existing assets are always favoured and never decommissioned unless demand falls) with a single Specific Net Annualised Surplus treating all assets equally:
SNAS = (Σ act_t·AC_t − AFC·cap) / Σ act_t

5b. Many appraisal rounds are inefficient (#1209, and the umbrella's point 5)

Splitting assets into small units for re-appraisal → many rounds. Open question: should we appraise multiple units simultaneously? Tied directly to the tranche-sizing choice (Theme 1) and the bundling decision (Theme 4) — small granularity is the price paid for fine control.

5c. Non-deterministic dispatch across platforms (#1174)

Although primarily a dispatch issue, it's the same class of problem as investment tie-breaking and feeds the tranching pathologies. Infinitely many equally-optimal dispatch solutions can exist, and highs resolves them in a way sensitive to floating-point differences → different results on mac vs. windows/ubuntu. Proposed fix: add a secondary objective term (e.g. an L2 term favouring spreading activity over multiple variables rather than winner-takes-all "corner solutions"). This is the same "corner solution" tendency (#1174) that makes the Theme 2 tranching example settle on TS1/TS2 and strand TS3.

---

Dependency map (suggested ordering)

The themes are not independent. A sensible critical path implied by the discussion:

1. Decide capacity philosophy (Theme 1: static capacity_granularity vs. dynamic) — gates almost everything else.
2. Fix tranche-in-isolation appraisal (Theme 2: account for already-selected capacity / marginal metrics) — fixes the infeasible-exit and over-investment failures, and is a prerequisite for the divisible-asset cleanup.
3. Resolve the unmet-demand fallback & pricing consistency (Theme 3: Investment fails when only remaining options fail to dispatch #1347 + require single pricing strategy) — needed for robustness on "normal" models.
4. Then tackle divisible-asset model + child-asset removal (Theme 4: Appraisal of divided assets is inefficient #1209 → Remove child assets and use parent assets everywhere instead #1123, explicitly blocked on Change the way LCOX appraisal tool works #1199/Change the way the NPV appraisal tool works #1201).
5. In parallel, finalise NPV metric (Change the way the NPV appraisal tool works #1201) and add the secondary objective term for determinism/corner-solutions (Potential for different dispatch results on different platforms #1174), which also relieves Theme 2 symptoms.

The two root causes, restated

If I had to reduce the whole umbrella to two sentences:

1. Incremental, isolated tranche appraisal is fundamentally inconsistent with constraints (availability, seasonal/annual limits) and metrics (DLC) that are only valid for the pooled asset — producing over-investment, infeasible exits, and the divisible-asset complexity.
2. Single-pass, non-equilibrium pricing legitimately yields zero/negative/inconsistent shadow prices, which corrupt the next year's activity coefficients and cause assets to decline dispatch — producing the shadow-price failures, the multi-output over-capacity, and the mixed-pricing problems.

Most of the individual issues in #1382 are downstream symptoms of one or both.

[ahawkes]

Thanks @tsmbland - what do you think of Claude's suggested ordering of problem solving?

[tsmbland]

Thanks @tsmbland - what do you think of Claude's suggested ordering of problem solving?

I think it makes sense. 1-3 are the biggest points that we need to make decisions about

[ahawkes]

I think it makes sense. 1-3 are the biggest points that we need to make decisions about

OK shall we start with (1)? Shall we discuss here. or in another issue?

[tsmbland]

I think it makes sense. 1-3 are the biggest points that we need to make decisions about

OK shall we start with (1)? Shall we discuss here. or in another issue?

Let's discuss on #1358. I've added a comment to kick things off.