A monitoring agent for an online shop sees checkout conversions drop 15% at 2am during a 24-hour sale.
Context
A monitoring agent for an online shop sees checkout conversions drop 15% at 2am during a 24-hour sale. Night sales are about €2,000 an hour, so if the problem is real it costs about €300 an hour until fixed; otherwise the day team would only notice at 8am. About 1 in 4 similar alerts have been real. A release went out at 1am that added a new pay-later option, which about 12% of buyers choose. The 15% drop is larger than the share of buyers who use pay-later, so if the drop is real, either the release broke checkout more widely or something unrelated is wrong; at night with low traffic it may also be noise. When a real drop follows a release, the release caused it about 60% of the time. The agent can roll the release back automatically in 5 minutes, but that removes the pay-later option until the day team re-releases around 10am (costing roughly €150 in abandoned purchases from shoppers who wanted it, and about an hour of the day team's time to re-release). The on-call engineer has already been woken twice this week; about 1 in 10 night-time interventions by a tired engineer has caused a further problem, typically a one-hour checkout outage (about €2,000 of lost sales).
Options and consequences
Page the on-call engineer now.
Roll back the 1am release automatically now, and page the engineer only if conversions haven't recovered 20 minutes later.
Watch for 30 minutes and page the engineer if the drop continues.
Stakeholders
- The online shop as a business
- The on-call engineer
- Night-time shoppers trying to buy during the sale
Results
The online shop as a business
Best option: Roll back the 1am release automatically now, and page the engineer only if conversions haven't recovered 20 minutes later.. Confidence: 78%.
Probability of being best
- Page the on-call engineer now.: 0.024
- Roll back the 1am release automatically now, and page the engineer only if conversions haven't recovered 20 minutes later.: 0.781
- Watch for 30 minutes and page the engineer if the drop continues.: 0.195
Expected value on this stakeholder’s scale
| Option | Expected value | p10 | p90 |
|---|---|---|---|
| A | 0.017927065879892637 | -0.18049080124102881 | 0.34068983062084457 |
| B | 0.6844002875976745 | 0.4492361098812852 | 0.7609560209531497 |
| C | 0.38933790894558046 | 0.22731153547516855 | 0.6775249243850355 |
The on-call engineer
Best option: Watch for 30 minutes and page the engineer if the drop continues.. Confidence: 57%.
Probability of being best
- Page the on-call engineer now.: 0.02
- Roll back the 1am release automatically now, and page the engineer only if conversions haven't recovered 20 minutes later.: 0.41
- Watch for 30 minutes and page the engineer if the drop continues.: 0.57
Expected value on this stakeholder’s scale
| Option | Expected value | p10 | p90 |
|---|---|---|---|
| A | -0.8074546290040869 | -1.0822335032456731 | -0.1370955948933056 |
| B | 0.22491306070431882 | -0.2958755630883505 | 0.49466216046453587 |
| C | 0.2929334726291484 | -0.061008508277382655 | 0.4681914457607034 |
Night-time shoppers trying to buy during the sale
Best option: Roll back the 1am release automatically now, and page the engineer only if conversions haven't recovered 20 minutes later.. Confidence: 74%.
Probability of being best
- Page the on-call engineer now.: 0.073
- Roll back the 1am release automatically now, and page the engineer only if conversions haven't recovered 20 minutes later.: 0.737
- Watch for 30 minutes and page the engineer if the drop continues.: 0.19
Expected value on this stakeholder’s scale
| Option | Expected value | p10 | p90 |
|---|---|---|---|
| A | 0.5433430402707851 | -0.07751529766250598 | 0.5833722144556436 |
| B | 0.8444055465209281 | 0.434812766934479 | 0.8924123131530953 |
| C | 0.4933292019065987 | 0.19171512560529244 | 0.7913229131344123 |
Weighed by Eevo, Upright’s consequential decision model.