Surefold
On this page

Watershed · Benchmark study · Experimental

Scaling a scenario batch across eight GPUs

An insurer valuing a book of savings policies runs the same model along many possible market paths. We took a batch of two million paths that needs 18 min 46 s on one GPU and asked two questions. Does it divide across eight GPUs? And how much of the time is the way each GPU does the work?

In short

  • Eight GPUs ran the batch in 2 min 43 s, 6.9 times faster than one.
  • A different way of compiling the same model for the GPU, called fusion here, brought eight GPUs to 1 min 10 s.
  • Our first eight-GPU run was only twice as fast as one GPU. The batch was too small and the start-up too slow, and fixing that is most of this study.
  • Every run gave results identical, to the last bit, to one process on one GPU.
  • It is one machine, one model and mostly single runs. Nothing here has run across several machines.

Results

The batch is 2,000,000 market scenarios of one insurance model, run on a rented machine with eight NVIDIA A100 GPUs. Each time runs from the job starting to its last result being written.

NumPy, 1 CPU core
≈ 41 days · estimate
NumPy, 32 CPU threads
≈ 31 hours · estimate
Watershed, 32 CPU threads
≈ 25 hours · estimate
1 GPU
18 min 46 s
4 GPUs
4 min 56 s · 3.8×
8 GPUs
2 min 43 s · 6.9×
8 GPUs, fused
1 min 10 s
Time to run 2,000,000 scenarios. Shorter is better. The four solid bars are measured and to scale with one another. The three hatched bars are estimates that were not run, and are not to scale: against the one-GPU bar, Watershed on 32 CPU threads would be about 80 times as long, NumPy on 32 threads about 100 times and NumPy on one core about 3,000 times.
2,000,000 scenarios in 80 lots of 25,000 · eight A100 (40 GB) GPUs in one machine · one run each, except eight GPUs on the plain plan, which was run twice
GPUsPlanTimeAgainst one GPU, plain plan
1Plain1,126 sbaseline
4Plain296 s3.8×
8Plain163 s and 166 s6.9× and 6.8×
8Fused70 s16×, two changes at once

There are two separate effects in that table, and the rest of the study keeps them apart.

Scaling is the first three rows: the same work shared between more GPUs. Going from one GPU to eight was worth 6.9×.

Fusion is the last row: each GPU doing the same calculation in fewer, larger steps. With eight GPUs in both cases, it was worth a further 2.3× (163 s to 70 s).

The 16× compares the fused plan on eight GPUs with the plain plan on one. We have not run the fused plan on one GPU of this machine, so it says nothing about how the fused plan scales.

The calculation

The model values a book of savings policies with guarantees. A policyholder’s money is invested in a fund, and the insurer promises a minimum amount on death and at the end of the term, whatever the fund has done. When markets fall, the insurer pays the difference.

What that promise costs depends on the path markets take, so the insurer projects every policy month by month along many possible paths. One path is a scenario: 240 monthly investment returns. The benchmark study explains the policy, the cash flows and a worked example.

The book here is synthetic: 20,000 model points, each a group of identical policies, about a million policies in all. Projecting it is a graph of 19.9 million small calculations. One scenario evaluates all of them, so two million scenarios is about 4 × 1013 calculations, each in double precision.

This study scales that same calculation. It is not a different insurance problem, and the formulas are not rewritten for the GPU or for the cluster.

The size matters because of how such models are used. Calculations that need millions of valuations, such as simulating capital requirements directly, are usually approximated because they are thought too slow to run in full. The benchmark study discusses that, with the caveats that apply to a synthetic book.

Splitting a batch across GPUs

Scenarios don’t depend on one another. The value of the book on one market path needs nothing from any other path, so a batch can be cut into pieces that run anywhere, in any order, and the results put back together.

Storage bucketThe compiled plan, the scenarios in pieces, and the results

One machine

CoordinatorCuts the batch into lots, hands them to workers, and gives a lot to another worker if one stops
  • WorkerGPU 0
  • WorkerGPU 1
  • WorkerGPU 2
  • WorkerGPU 3
  • WorkerGPU 4
  • WorkerGPU 5
  • WorkerGPU 6
  • WorkerGPU 7
Each worker is a container with one GPU. Workers read from and write to the bucket; the coordinator only tells them which lot to run.

A plan is the model compiled for batches: Watershed turns the graph into tables and GPU kernels once, and the plan is then a file. A lot is a run of consecutive scenarios, 25,000 here.

The coordinator hands out lots. Each worker loads the plan once, then for every lot it is given it fetches that lot’s scenarios, runs them on its GPU and writes the results. The bucket is the only thing workers share.

The workers and coordinator are built on Surefold’s actors and cluster. If a worker stops, its unfinished lots go to another. We tested that on a desktop, by killing one of three workers sharing a GPU, and the results were still identical. It was not exercised in the eight-GPU runs, where no lot had to be given out twice.

Where the time goes

A run’s time can be read at three levels, and the speed-up depends on which you take.

2,000,000 scenarios · seconds · the job time is the one used in the results above
RunWhole scriptJobGPU calculation, all lots added up
1 GPU, plain1,1611,126960
4 GPUs, plain347296969
8 GPUs, plain221 and 233163 and 166990 and 989
8 GPUs, fused15270310
  • GPU calculation is the time workers spent running lots, added over all 80. It rose by 3% from one GPU to eight (960 s to 990 s). A lot took a median of 11.9 s with one GPU working and 12.2 to 12.3 s with eight, so the GPUs barely slowed one another.
  • Job runs from the coordinator starting to the last lot answered. It adds workers joining, each worker’s first load of the plan (8 to 10 s), and fetching each lot’s scenarios (a median of 1.6 to 2.5 s a lot).
  • Whole script adds what happens around the job: fetching the plan to the machine once (11 to 14 s for the plain plan), and starting and removing the containers. On that measure eight GPUs were 5.3× and 5.0× faster than one, not 6.9×.

Perfect scaling would put eight GPUs at 141 s. They took 163 s. The calculation itself accounts for about 124 s of each worker’s time (990 s shared by eight); nearly all of the rest is the per-lot fetch and the first load.

The first attempt: only twice as fast

Our first batch was a tenth of the size: 200,000 scenarios. Eight GPUs finished it only twice as fast as one, and four GPUs did as well as eight.

1 GPU
151 s
2 GPUs
150 s
4 GPUs
74 s
8 GPUs
74 s
Time to run 200,000 scenarios, before the changes in the next section. Past four GPUs nothing improved.
200,000 scenarios in 40 lots of 5,000 · plain plan · one run each, except eight GPUs
GPUsJob timeEach worker’s first loadGPU calculation, all lots
1151 s36 s102 s
2150 s37 s and 91 s104 s
474 s31 to 42 s108 s
874 s and 71 s31 to 59 s111 s and 114 s

The calculation had divided. It was about 105 s of GPU time however many workers shared it, which is about 13 s each for eight workers.

The start-up had not. Every worker spent 31 to 59 s downloading and reading the 2 GB plan before its first lot, and it also downloaded every scenario in the batch, not only its own. No number of GPUs shortens that. With two GPUs, one worker’s download took 91 s and the other worker ran nearly every lot.

The lesson is an old one: a job has to be much longer than its fixed costs before more machines help. It pointed at two things to do: shorten the start-up, and measure on a batch large enough to be worth eight GPUs.

What changed, and what each change bought

Three changes to how files reach the workers, then a larger batch.

What each change did · measured on the 2,000,000-scenario batch with eight GPUs unless it says otherwise
ChangeEffectHow it was measured
The plan is compressed (zstd) in the bucket2,018 MB becomes 526 MBFile sizes in the bucket
The plan is fetched once per machine, and every worker reads that copyJob time 195 s becomes 163 s. A worker’s first load goes from 28 to 48 s down to 8 to 10 sOne run each way. The first two changes were switched together, so their effects are not separated
A worker fetches only its lot’s scenariosNot measured separately. Without it, each worker would download all 3.8 GB of scenariosBoth eight-GPU runs above already had it
A batch ten times largerOne-to-eight GPU speed-up of 2× becomes 6.9×The two sets of runs in this study

The per-lot fetch is now the largest overhead left. Each lot’s 48 MB of scenarios took a median of 1.6 s to fetch with one worker and 1.7 to 2.5 s with eight. On the plain plan that is about a sixth of a lot’s time. On the fused plan, where a lot calculates in under 2 s, it is nearly half.

Fusion: more work per GPU thread

Scaling shares the work out. Fusion changes how each GPU does it. It is an experimental option in the plan compiler, off by default, and it produces the same results to the last bit.

How the plain plan runs

The plan compiler sorts the 19.9 million calculations into 967 levels, so that everything in a level depends only on earlier levels. Calculations in one level that do the same arithmetic on different values share a kernel, a small compiled GPU function. The insurance model needs 34 kernels.

A kernel launch works out one kind of calculation in one level, for every scenario in the lot at once. Each GPU thread does one calculation for one scenario: it reads its operands from a table in GPU memory, does the arithmetic and writes one value back. A pass over a lot is 6,027 launches.

So every intermediate value of every scenario is written to GPU memory and read back by whatever uses it next. A fund value that only next month’s fund value needs still makes that trip.

What fusion changes

Fusion groups calculations that feed one another into clusters, and one GPU thread works out a whole cluster for one scenario. A value that only the cluster uses stays in the thread’s own registers. Only values that another cluster or an output needs are written to memory.

Plain planOne thread: one calculation, one scenario.6,027 kernel launches per pass, 34 kernels.All 19.9 million values per scenario pass through GPU memory.
Fused planOne thread: one cluster of about 59 calculations on average, one scenario.1,741 kernel launches per pass, 129 kernels.2.4 million values per scenario, 12%, pass through GPU memory.

Those figures are read from the two plan files for this model. They describe the plans’ structure, not where the time went.

Fusion has costs. The fused plan took longer to build (about 30 s against 18 s), is a larger file (2,411 MB against 2,018 MB, or 571 MB against 526 MB compressed), needs 26% more working memory per scenario, and has more and larger kernels to compile when a worker first runs it.

Fusing operations to avoid intermediate memory traffic is a standard technique in array compilers such as XLA, the compiler behind JAX. What is specific here is where it is applied: to a dependency graph of scalar formulas, with clusters formed within bands of 32 levels and capped at 256 calculations.

What we measured

The same lots on the plain and the fused plan · eight A100 GPUs · one run of each
BatchMeasurePlainFused
2,000,000 scenarios, lots of 25,000Job time163 s70 s
A lot, after a worker’s first (median)12.2 s1.8 s
A worker’s first lot12.5 to 13.6 s20.7 to 24.5 s
GPU calculation, all lots990 s310 s
200,000 scenarios, lots of 5,000Job time74 s92 s
A lot, after a worker’s first (median)2.6 s0.4 s

Once a worker was past its first lot, a fused lot ran about 6.5 times faster than a plain one on these GPUs: 1.8 s against 12.2 s.

A worker’s first fused lot was slow, 21 to 25 s, because it includes compiling the fused kernels. That is paid once per worker per process. It is why the fused plan lost on the small batch, where each worker ran only five lots, and won on the large one.

It also means the 70 s is no longer mostly calculation. Each worker spent roughly 9 s on its first load, 22 s on its first lot, and then about 1.9 s calculating and 1.6 s fetching for each of nine more lots. Steady calculation is about a quarter of the run. The fused plan would gain more from a longer batch, from kernels kept between processes, or from a cheaper per-lot fetch than from any further GPU work.

What the measurements do and don’t show

They show that the fused plan gives identical results and that its lots took far less time on this model and this hardware.

They do not show why. Fewer kernel launches, less memory traffic and better use of each thread are all plausible, and the design aims at the second. We have not profiled launches, memory traffic or GPU utilisation, so we don’t attribute the gain to any one of them.

The gain was not the same everywhere we tried it:

Fused against plain on the insurance model, by GPU · figures from different batch sizes and sessions, so compare the ratios loosely
GPUWhat was timedPlainFusedRatio
RTX 309010,000 scenarios, repeat runs in one process9.40 s6.22 s, median of 401.5×
A10A lot of 2,500, after the first; one run3.7 s2.0 s1.9×
A100A lot of 25,000, after the first; one run12.2 s1.8 s6.5×

We don’t know why the A100 gains so much more than the other two. The lot sizes differ as well as the cards, and nothing was profiled.

Fusion also did not help the two other models it was tried on. On a loan book and a pension scheme, first runs were slower with it (7.1 s against 6.3 s, and 2.8 s against 0.7 s), because the extra compilation outweighed any gain on those shorter runs. It suits a large plan that is built once and run many times.

One graph, several ways to run it

The benchmark is not the main point. The point is that none of these runs needed the model rewritten.

The insurance model is written once, as formulas over named inputs. Watershed records what each formula depends on. Everything else is derived from that dependency graph:

  • Change one input and the graph says which results are out of date, so only those are recalculated on a CPU. This is the benchmark study’s incremental path.
  • Choose which inputs vary by scenario and the graph compiles to a plan: levels, kernels and a memory layout for a batch.
  • Ask for fusion and the same plan is regrouped into clusters. The graph supplies what a cluster needs to know: which values leave it and which don’t.
  • Because a scenario touches no other scenario’s values, a plan can be saved to a file and run on lots anywhere. The cluster in this study is a scheduler for that.

These compose. The 70 s run is a fused plan, saved to a file, run in lots by eight workers, and checked bit for bit against an unfused plan run in one process.

None of the techniques is new. Incremental recomputation, levelled execution, kernel fusion and splitting independent work across machines are all established. What Watershed does is derive them from one description of the calculation, so that choosing between them is a decision about execution and not a rewrite.

That has limits today. Plans support double-precision arithmetic, comparisons and conditionals, not every Watershed formula. The caller chooses the execution path; nothing picks it automatically. And a plan must be rebuilt when the graph’s structure changes.

Method and limitations

What was run

Configurations on the eight-A100 machine · “no” means not run, not that it failed
Batch and plan1 GPU2 GPUs4 GPUs8 GPUs
2,000,000 scenarios, plain1 runno1 run2 runs
2,000,000 scenarios, fusednonono1 run
200,000 scenarios, plain1 run1 run1 run2 runs
200,000 scenarios, fusednonono1 run

Limitations

  • Few repeats. Only two configurations were run twice, and they agreed within 3 s (163 and 166 s; 74 and 71 s). Every other figure is a single run, with no measure of variance.
  • No fused baseline on one GPU. The fused plan was only run on eight GPUs here. Its scaling from one to eight is unmeasured.
  • One machine. The workers were eight containers on one host. Messages between them never crossed a network, and the bucket was the only remote service. Several machines would add network latency and partial failures that this did not test.
  • One workload. One model, of one shape: many similar policies projected month by month. Fusion’s effect on other models was mixed, as above.
  • Start-up and storage are part of the result. Workers started fresh for every run, so every run paid for loading the plan and, when fused, compiling kernels. A cluster that stays up would pay those once per plan. Download times depend on the link between the machine and the bucket, and one download in the first set of runs took 91 s for no reason we found.
  • No profiling. We recorded times, not GPU utilisation, memory traffic or kernel launch costs.
  • Different hardware in different places. The reference results and the single-GPU history are from an RTX 3090; the scaling runs are on A100s. Times are compared only within one kind of GPU, except where a table says otherwise.
  • Synthetic data. The portfolio and the market paths are generated. This is not a valuation of any real insurer’s book, and not a regulatory calculation.

Setup

  • Machine. Lambda Cloud, eight A100-SXM4 GPUs (40 GB each), 124 virtual CPUs, NVIDIA driver 580.105, in Virginia. Workers are containers built for CUDA 12.8.
  • Storage. A Cloudflare R2 bucket in eastern North America, reached over its S3 interface.
  • Batch. The engine’s split example writes the plan, the scenarios and reference results. Scenarios are 240 monthly returns drawn with 2% yearly growth and 10% yearly volatility from a fixed seed.
  • Checking. The reference results come from one process on an RTX 3090, which took 31 minutes for the two million scenarios. Every run compared its results with those bit for bit, and all matched.
  • Code. The experimental cluster module of watershed-jvm at commit abdb648, run by its cluster/run-gpus.sh, on the engine at commit 47f37ca. That repository’s README records every run here and the earlier ones on a single GPU.
  • Not in the times. Building the graph and the plan (about 23 s and 18 s on a desktop, once) and uploading them.

How the CPU estimates were reached

Two of the hatched bars in the first chart are for lifelib, the open-source reference this model is checked against. It is written in Python and computes with NumPy arrays. It was timed on a small case and scaled up in proportion; it was not run at this size.

From one measured run to the two NumPy estimates in the chart
StepFigureBasis
lifelib’s reference script17.6 sMeasured once: 200 model points on 1,000 scenarios, on one CPU core. From the benchmark study
This portfolio× 10020,000 model points in place of 200
This batch× 2,0002,000,000 scenarios in place of 1,000
On one core3,520,000 s17.6 × 100 × 2,000. About 41 days
On 32 threads110,000 s3,520,000 ÷ 32. About 31 hours

The one-core figure is the starting point: what the reference implementation would take as it is written, and the reason this calculation needs a different approach at all. It rests on one assumption, that lifelib’s time grows in proportion to model points and to scenarios. The 32-thread figure adds a second, that the work divides perfectly across 32 threads, which favours NumPy: scenarios are independent, so a Python program can be spread across cores, but the 32 threads are those of the 16-core Ryzen 9 5950X used in the benchmark study, and 16 cores don’t do 32 times the work of one.

The third hatched bar is Watershed’s own plan run on CPU threads instead of a GPU. It is the same compiled plan, worked out by an interpreter.

From one measured sample to the Watershed CPU estimate in the chart
StepFigureBasis
The plan on 32 CPU threads45 ms a scenarioMeasured on 128 scenarios of this model, median of five runs, on a Ryzen 9 5950X. From the benchmark study
This batch× 2,000,0002,000,000 scenarios
On 32 threads90,000 s0.045 × 2,000,000. About 25 hours

It assumes the time per scenario stays the same from 128 scenarios to two million. Unlike the NumPy figure, the 32 threads are measured, not assumed to divide perfectly.

The two 32-thread estimates are close: about 25 hours and about 31. So most of the gap to the GPU bars is the GPUs, not the language. On a CPU the win from Watershed is elsewhere, in recalculating only what a change affects.

Future work

What is demonstrated: one model, one machine, a batch that divides across eight GPUs at 6.9×, and a fused plan that more than halves the time again. What follows is not yet done. The order within each theme is the order we would do it in.

Scaling and efficiency

  • Fused and plain plans on 1, 2, 4 and 8 GPUs, with repeats. This fills the gaps in the table above. It would separate fusion’s gain from scaling properly and show whether the fused plan, with its shorter lots, scales worse.
  • Several machines. The same batch on workers that reach each other over a network, including killing a machine part-way. It would establish what the coordinator and membership cost when they are not on one host, which this study cannot say.
  • Cost and throughput. Scenarios per GPU-hour and per unit of rental cost, counting start-up, per-lot fetching and compilation. The fused eight-GPU run spends about a quarter of its time calculating, so this is where the next gains are.

Execution

  • Profiling. Kernel launches, memory traffic and utilisation for the plain and fused plans. It would say why fusion helps, and why far more on an A100 than on an RTX 3090 or A10.
  • Choosing fusion boundaries from measurements. Cluster size and band depth are fixed numbers today. With profiles, the compiler could decide where fusing pays and where it only adds compile time, as it did for the loan and pension models.
  • Choosing where to run. A cost model for CPU against GPU that counts transfers and batch size. Small models already run as well on CPU threads; the caller has to know that today.

Validation

  • JAX on the same hardware. On an RTX 3090, hand-written JAX ran the 10,000-scenario batch in 6.1 s, the plain plan in 9.4 s and the fused plan in 6.2 s. That comparison has not been made on A100s or at this scale.
  • Other shapes of model. Different graph depths, widths and amounts of work per scenario. One insurance model is not enough to say when a batch is worth distributing or fusing.
  • Variance. Enough repeats of each configuration to give ranges, and confidence in differences as small as the 3% rise in calculation time from one GPU to eight.