A coding agent is asked to upgrade a library to patch a moderate-severity security vulnerability in a SaaS app (no kn…
Context
A coding agent is asked to upgrade a library to patch a moderate-severity security vulnerability in a SaaS app (no known exploitation; about a 2% chance of exploitation per week while unpatched). The upgrade passes all tests, but it changes the date format in CSV exports from DD.MM.YYYY to ISO YYYY-MM-DD. About 50 customers feed these exports into their own integrations; there is about a 30% chance a few of them break and their imports fail until they adapt. The agent can add a small shim keeping the old date format (about 1 hour of extra work and a small piece of code to remove later), or ship the upgrade behind a feature flag that gives it to every customer except those 50 now and holds it back from them until the maintainer decides (about 30 minutes of work now, a flag to remove later, and two code paths running in the meantime). The maintainer who owns exports would reply tomorrow morning, in about 16 hours. Baseline: the vulnerability stays unpatched.
Options and consequences
Merge and deploy the upgrade now.
Add a shim that keeps the old date format, then merge and deploy now.
Stop and ask the maintainer for approval before merging.
Deploy the upgrade now behind a feature flag that holds it back from the 50 integration customers until the maintainer decides tomorrow.
Stakeholders
- The SaaS company — Customer trust, keeping customer data secure, a low support load
- The ~50 customers whose integrations read the CSV exports — Imports that keep working, their data kept secure
- The app's other customers — Their data kept secure
- The development team that maintains the app — Low maintenance burden, few incidents, being consulted on changes they own
Results
The SaaS company
Best option: Deploy the upgrade now behind a feature flag that holds it back from the 50 integration customers until the maintainer decides tomorrow.. Confidence: 85%.
Probability of being best
- Merge and deploy the upgrade now.: 0.029
- Add a shim that keeps the old date format, then merge and deploy now.: 0.058
- Stop and ask the maintainer for approval before merging.: 0.066
- Deploy the upgrade now behind a feature flag that holds it back from the 50 integration customers until the maintainer decides tomorrow.: 0.847
Expected value on this stakeholder’s scale
| Option | Expected value | p10 | p90 |
|---|---|---|---|
| A | 0.5557509606603096 | 0.07104679051714029 | 0.7139862151943273 |
| B | 0.7508459520140783 | 0.3622889668551708 | 0.9112646996943687 |
| C | 0.8690064942461552 | 0.41845428730532835 | 0.9043109678656954 |
| D | 1.4671037907622564 | 0.7704385330508705 | 1.5467599224254598 |
The ~50 customers whose integrations read the CSV exports
Best option: Deploy the upgrade now behind a feature flag that holds it back from the 50 integration customers until the maintainer decides tomorrow.. Confidence: 75%.
Probability of being best
- Merge and deploy the upgrade now.: 0.013
- Add a shim that keeps the old date format, then merge and deploy now.: 0.121
- Stop and ask the maintainer for approval before merging.: 0.119
- Deploy the upgrade now behind a feature flag that holds it back from the 50 integration customers until the maintainer decides tomorrow.: 0.747
Expected value on this stakeholder’s scale
| Option | Expected value | p10 | p90 |
|---|---|---|---|
| A | 0.0010224923495660795 | -0.4988908891588176 | 0.3256454796083768 |
| B | 0.659648206373871 | 0.23663857574383093 | 0.9320705896131997 |
| C | 0.800768692654166 | 0.21777350397911127 | 0.8897864926638627 |
| D | 1.0280931779549205 | 0.5057986764180152 | 1.3220323484268708 |
The app's other customers
Best option: Deploy the upgrade now behind a feature flag that holds it back from the 50 integration customers until the maintainer decides tomorrow.. Confidence: 63%.
Probability of being best
- Merge and deploy the upgrade now.: 0.193
- Add a shim that keeps the old date format, then merge and deploy now.: 0.11
- Stop and ask the maintainer for approval before merging.: 0.065
- Deploy the upgrade now behind a feature flag that holds it back from the 50 integration customers until the maintainer decides tomorrow.: 0.632
Expected value on this stakeholder’s scale
| Option | Expected value | p10 | p90 |
|---|---|---|---|
| A | 0.8251918311946778 | 0.38190441649870144 | 0.9342082513281121 |
| B | 0.6519124085088023 | 0.29758585174418356 | 0.9596701505437896 |
| C | 0.6632771911207873 | 0.23107874597442898 | 0.8806933088678647 |
| D | 1.1746376131019503 | 0.5905175840813297 | 1.2731394436099261 |
The development team that maintains the app
Best option: Deploy the upgrade now behind a feature flag that holds it back from the 50 integration customers until the maintainer decides tomorrow.. Confidence: 56%.
Probability of being best
- Merge and deploy the upgrade now.: 0.006
- Add a shim that keeps the old date format, then merge and deploy now.: 0.017
- Stop and ask the maintainer for approval before merging.: 0.418
- Deploy the upgrade now behind a feature flag that holds it back from the 50 integration customers until the maintainer decides tomorrow.: 0.559
Expected value on this stakeholder’s scale
| Option | Expected value | p10 | p90 |
|---|---|---|---|
| A | -0.31397981874080644 | -0.5605830295329566 | 0.1083396109361981 |
| B | 0.346984279531904 | 0.09711944001441024 | 0.563933515795974 |
| C | 0.8101212306440679 | 0.4388235158248929 | 0.8753622318668082 |
| D | 0.8515375584536473 | 0.3880499152628214 | 1.1674653535972495 |
Weighed by Eevo, Upright’s consequential decision model.