Surefold
On this page

Benchmark study · supporting material

The benchmark models

The seven workloads in the Watershed benchmark, what each scenario changes and where the formulas come from. The measurements page has timings and correctness checks.

The models

All populations, portfolios and economic scenarios are synthetic. Nothing here is a result about a real institution or a real patient.

What a node, a scenario and a shared input are in each model
ModelThe graphA scenarioShared inputs
Insurance guaranteesEvery model point, month by month: fund, deaths, lapses, present valuesA market path of 240 monthly returnsDiscount curve, fees, shifts to mortality, lapses, expenses and fund values
Loan bookEvery mortgage, quarter by quarter: balance, default probability, expected lossAn economic path: unemployment, house prices and interest rates for 100 quartersThe credit model’s sensitivities
Pension schemeEvery member group, year by year: pension with capped indexation, survival; then scheme assetsAn economic path of 75 yearsAsset allocation, contributions, caps, mortality
Two-compartment dosing modelOne simulated subject for 72 hours: renal function, a two-compartment pharmacokinetic model stepped exactly hour by hour, a dosing protocolA simulated subjectThe protocol
Tax and benefit rulesOne household: income tax, social insurance, benefits with tapers and capsA householdRates and thresholds
Motor insurance pricingOne motor policy: rating factors, discounts, renewal limits, demandA policyThe rating factors
Excel valuation workbookThe cells between chosen inputs and outputs of an unmodified .xlsx fileA draw of the uncertain inputsNone

In the first three the population is in the graph and the scenario is a market or economic path. In the rest the graph is one entity and the population is the scenarios. The model author chooses which inputs vary by scenario. That affects how much work each scenario does and how much data the plan needs to move.

The spreadsheet reader translates only the formulas between the chosen cells; a cell that does not depend on a varied input is taken at the value stored in the file. It has been tried on one workbook, reads .xlsx only, and handles arithmetic, comparisons, IF, AND, OR, SUM, AVERAGE, MIN, MAX, EXP, LN, SQRT and VLOOKUP into a table of constants.

What the models found

Apart from the lifelib model, its tables and the workbook, the books, the economies and the rules are made up. These show what such a run can reveal; they are not facts about any insurer, bank or scheme.

  • A single steady path hides the cost of a guarantee. It prices the insurer’s end-of-term guarantee at 0.8 billion; 10,000 paths price it at 8.0 billion. The guarantee costs something only when markets fall.
  • The formula for capital overshoots. The worst year in two hundred costs 6.1 billion by nested Monte Carlo simulation, and 9.8 billion by two prescribed stresses added by formula.
  • A few scenarios understate loan losses. Three weighted scenarios set aside 7% less than the average of 10,000 paths.
  • One assumption dominates the loan book. Whether a repossessed house fetches 65% or 75% of its value changes the provision by 76%; a quarter more sensitivity to unemployment changes it by 6%.
  • More shares stop helping the pension scheme. With nothing paid in, moving from no shares to 40% cuts the chance of running out of money from 98% to 34%; 80% does no better.

The dosing model has no entry in this list. Its protocol and population are invented, so its outputs are not findings about any drug or patient. It is used as a compute workload only.

What each example rests on

What is taken from somewhere, with where, and what is made up.

  • The insurer’s guarantees and its capital. The model, its mortality table and its charges are lifelib’s savings.CashValue_ME_EX2, and the graph is checked against it. The policies are made up, as are the swings in the funds and how far markets and rates move in a year. The worst year in two hundred is Solvency II’s measure (Directive 2009/138/EC, Article 101); the 39% fall in funds is the size of its standard formula’s shock for shares, while the one-point move in rates and the way the two are added up here are simplifications, not that formula. The direct calculation is a nested Monte Carlo simulation, of which a Milliman paper says: “This brute force approach is too time-consuming to be used in a real insurance business case” (Boumezoued, Cherchali and Truc, 2022). The run here applies each outer scenario as an instantaneous shock and does not model a year of ageing and cash flows, so it is not a capital calculation.
  • The loan book. Banks set money aside for expected losses under IFRS 9, weighting several economic scenarios; the European Banking Authority’s 2023 monitoring report says “the most common approach is to envisage three scenarios”. Everything else is made up: the loans, how defaults respond to unemployment and house prices, and the economy. It follows no published model.
  • The pension scheme. Capped increases are how United Kingdom schemes work: the law requires pensions earned since 1997 to rise with prices up to 5% a year, and 2.5% for service since 2005 (limited price indexation; Pensions Act 2004, section 278). The scheme, its members, the death rates and the economy are made up.
  • The dosing model. A synthetic workload with a published model’s structure, not a clinical study. The two-compartment structure and its parameters are those of the vancomycin model of Goti and others (Therapeutic Drug Monitoring 2018;40:212) for adults not on dialysis: clearance 4.5 L/h scaled by (creatinine clearance ÷ 120)0.8 with creatinine clearance capped at 150 mL/min, central volume 58.4 L per 70 kg, peripheral volume 38.4 L, intercompartmental clearance 6.5 L/h, and between-subject variability of 39.8%, 81.6% and 57.1%. These values were checked against the transcription of the paper’s Table 2 in the open-source nlmixr2lib model library and match it. The paper itself is behind a paywall and was not read. The example differs from that transcription in two ways: it does not scale the peripheral volume by weight, and it does not apply the paper’s floor on serum creatinine for subjects over 60. Creatinine clearance is estimated by the Cockcroft and Gault (1976) formula. The exposure range of 400 to 600 mg·h/L is the target in the 2020 consensus guideline (Rybak and others, American Journal of Health-System Pharmacy 77:835). The dosing protocol and the population are invented. Nobody with medical training reviewed any of it, and nothing it outputs is a clinical result or a basis for dosing anyone.
  • The spreadsheet. fcffsimpleginzu.xlsx is from Aswath Damodaran’s page of valuation spreadsheets, as downloaded. The ranges the inputs were varied over are made up.
  • Tax and benefits, and motor insurance pricing. Made up throughout: the rules, the rates, the households and the drivers.