A fraud score for a corporate purchase-card program: how well it holds up on months it never saw, and what it is worth when a confirmed alert blocks the card.
Jay Yu · UCLA Anderson, MS in Business Analytics
The ten largest card compromises in the out-of-time window, Nov–Dec 2010loststopped by blocking the cardcard blocked
Executive summary
1 · Where the fraud is
2 · Data and cleaning
3 · Point-in-time variables
4 · How much to trust the number
5 · Model selection and performance
6 · Card-stop value and alert briefs
7 · Cutoff, staffing and uncertainty
8 · Rollout and monitoring
9 · Limitations and next steps
A · Data quality report
B · Assumptions and reproducibility
Stopping the card · Model & decision reportExecutive summary
ToHead of fraud operations, purchase-card programFromJay YuReWhether to deploy a transaction fraud score, and how to run it
Recommendation: deploy the score with a 3% review budget, block a card the same day its alert is confirmed, and run a 60-day shadow pilot before switching on real-time declines.
It finds fraud on months it never saw. On Nov–Dec 2010, held out from every modeling choice, the top 3% of scores contain
of fraud transactions (Sep–Oct validation: ).
Optimized the same way, it is worth about twice the best single-variable rule:
against net over two months.
The value is in blocking the card, and speed decides it. Fraud arrives in runs on a compromised card;
of 2010 fraud dollars fell on a card's first fraud day.
With the same alerts, blocking the next day stops of out-of-time fraud dollars,
the same day , and declining in real time
.
It costs little to run. At this program's volume the score raises
– alerts a day,
under a tenth of one analyst. Review, card reissue and customer friction together cost
of the fraud dollars saved.
The headline number is honest. The usual random train/test split would have reported
detection for a model that delivers
on later months. Every choice here was made on earlier data only.
Value and uncertainty
Over the two out-of-time months the next-day policy nets at the agreed cost assumptions.
Resampling the compromised cards and every assumption gives a 90% range of
–, never below zero.
The range is wide because only cards were compromised in that window and ten cards hold
of all fraud dollars.
Decision requested
Approve a 60-day shadow pilot: score live purchases, work alerts the same day, and log what a block would have stopped. Go live if the top 3% of scores catch at least 45% of fraud, the lower edge of what a 60-day sample can confirm (§8).
What it rests on
One program's 2010 transactions, fraud labels assumed complete, and cost inputs that are placeholders for the program's own figures (Appendix B).
Stopping the card · Model & decision report1 · Where the fraud is
Fraud is rare by transaction but concentrated by card: cards carry all fraud records.
purchases on cards in 2010
of purchases are fraud
of fraud over the year
frauds on the median compromised card
Figure 1.1 · Compromised cards by number of fraud transactions on the card, 2010
Count of the cards with at least one fraud record.
What the shape means
A third of compromised cards () show a single fraud; the rest run for weeks, up to 69 transactions on one card. The ten costliest cards account for of fraud dollars, and of fraud dollars land on a card's first fraud day.
Three design choices follow
Describe the card's recent behavior, not the transaction alone: counts, spend and bursts over the last 0–30 days (§3).
Test on later months, because a random split puts the same card's burst on both sides (§4).
Value an alert by the run it ends, not the dollars it flags, and act fast (§6).
Stopping the card · Model & decision report2 · Data and cleaning
Two exclusions and three imputations make the file usable, and none of them looks at the fraud label.
The raw file has records and ten fields: card, date, merchant number, description, state and ZIP, transaction type, amount and the fraud label. The full profile is in Appendix A.
Exclusions
Rule
Records removed
Of which fraud
Only purchases (type P) are scored. The single $3.1M record is a data error, about 65 times the next largest amount ($47,900).
Imputation, most specific source first
Field
Step
Records
Why this order
A missing merchant number is usually recoverable from its description, which is the same merchant. State comes from the ZIP when present, otherwise from the merchant's other records. ZIP comes from the merchant first and the state's most populated ZIP last, the weakest guess. Merchants that never had a number get a new one built from their description so they still group correctly.
Why it is safe
Cleaning uses identity and location fields only. Nothing here reads the fraud label, so using the whole file to resolve merchants cannot leak the outcome into the model. Statistics that do use the label, such as variable ranking, are computed on training rows only (§3–§5).
Stopping the card · Model & decision report3 · Point-in-time variables
variables describe how each card, merchant and pairing behaved over the last 0–30 days, using only earlier records.
Entity
Count
Total $
Average $
Largest $
This $ ÷ average
Days since last
Burst ratio
Each statistic is computed for windows of 0, 1, 3, 7, 14 and 30 days. Burst ratio = transactions in the last 0–1 days ÷ the daily rate over 7, 14 or 30 days. Plus the amount itself and a round-dollar flag.
The rule that keeps them honest
A variable for a purchase may use that purchase and anything before it, never after. Same-day records count only if their record number is lower. A unit test rebuilds the table from a truncated file and checks that no earlier value changes.
Strongest variables in the final model
Variable
Meaning
KS
A w-day window covers the purchase's day plus the previous w days. KS = largest gap between the fraud and non-fraud distributions, on Jan 15 – Aug 31 training rows. Top 10 of the selected.
Stopping the card · Model & decision report4 · How much to trust the number
A random test split overstates detection by ; testing on later months lands within of reality.
The same model was trained three ways and scored on its own test set and on the shared out-of-time window. A split is trustworthy if its test number predicts the out-of-time one.
Figure 4.1 · Detection rate in the top 3% of scores (FDR@3%), test set vs out-of-time, mean of 5 seedstest set claimedout-of-time, Nov–Dec
Model
Split
Test
Out-of-time
Test − OOT
Test frauds on cards with fraud in training
Why the random split flatters
Under a random split, of test frauds sit on a card whose other frauds are in training. The model recognizes the burst it has already seen, which says nothing about the next compromised card.
What this report does instead
Hyperparameters are chosen on later months (§5), the cutoff on Sep–Oct (§7), and every headline number is out-of-time. A random forest on the same variables, shown for comparison, is also most overstated by the random split.
Stopping the card · Model & decision report5 · Model selection
Boosted trees rank best, but the test windows cannot separate close settings, and tighter regularization does not help.
Each setting was trained on Jan 15 – Jun 30 and tested on Jul – Aug, then trained on Jan 15 – Aug 31 and tested on Sep – Oct. The pick is the best mean test score. Nov–Dec was recorded but never used to choose.
Model family
Train
Test (mean of 2 folds)
± 1 SE
Out-of-time
Best setting per family, FDR@3%. SE from resampling compromised cards in the test windows.
Figure 5.1 · More variables helptestout-of-time
FDR@3% by number of variables kept.
Figure 5.2 · Bigger trees memorize, test stays flattraintest
FDR@3% by leaves per tree, other settings fixed.
Noise sets the limit
of settings are within one standard error of the best. Each test window has about 45–50 compromised cards, so a 2-point difference is not evidence. The report treats the choice as "one good boosted model", not "the best one".
Overfitting, read correctly
Training detection near 100% looks alarming, but cutting trees to 3 leaves lowers it to 81% without raising test detection, and test falls below 7 leaves. The model memorizes training bursts without hurting its ranking of new cards. What does lower the number is time: Sep–Oct 69%, Nov–Dec 64%.
Stopping the card · Model & decision report5 · Final model performance
On Nov–Dec the model finds of fraud in the top 3% of scores and in the top 5%.
Figure 5.3 · Share of fraud found by review budgettrainvalidationout-of-time
Window
Frauds
1%
2%
3%
5%
FDR at each review budget. Training rows are scored in-sample.
Final model
Against a rule an analyst could write
The strongest single variable, a card's spend over the last 3 days, used as a rule and tuned the same way, stops of out-of-time fraud and nets . The model stops and nets with the same next-day policy. The rule settles on a budget and alerts; the model works alerts and stops nearly twice the dollars, a trade worth making when an alert costs about $8 to review.
Stability
The score's distribution barely moved between training and Nov–Dec (population stability index ; below 0.1 is stable). The drop from validation to out-of-time comes from which cards were compromised, not from a shifted population.
Stopping the card · Model & decision report6 · Card-stop value
Blocking the card on its first confirmed alert stopped of out-of-time fraud dollars. Acting the same day would have stopped .
A confirmed alert is worth the rest of the run, not just the purchase it flags. Replay of Nov–Dec 2010 with the cutoff fixed on Sep–Oct; every alert worked, card blocked the next day.
Figure 6.1 · Every compromised card in the out-of-time window, Nov 1 – Dec 31loststoppedcard blocked
Bar height ∝ √(daily fraud $); cards renumbered by fraud dollars.
The first day decides most of the outcome
Card 01 was flagged on its first fraud day; blocking it the next morning stopped while was lost. Card 02 was also flagged on its first fraud day, but went through that same day, beyond the reach of a next-day block.
Policy, same cutoff
Stopped
Share
Sep–Oct
Alerts
Good declined
Good declined: legitimate purchases on a blocked card within the 5-day reissue lag, plus false real-time declines.
Stopping the card · Model & decision report6 · What an analyst sees
Every alert arrives with measured reasons and a calibrated fraud rate, so an analyst can confirm or clear it in minutes.
A score alone tells an analyst where to look, not what to check. Each alert gets a brief: the reasons it fired, how often alerts in its score band were fraud, the card's recent activity, and the action.
An actual out-of-time alert from the published sample. Card relabeled, dollar amounts rounded to $10.
How reasons are measured
Each variable is reset to its typical value, the median of non-fraud training purchases, and the drop in the score is that variable's contribution. Windows of the same statistic are grouped so a brief never lists "spend over 1 day" and "spend over 3 days" as two reasons. The method needs only the model's predictions, so it works for any model the program swaps in later.
Optional written summary
An LLM can rewrite a brief as two sentences (fraudscore briefs --llm, Claude Opus 5.5). It is kept only if every number in it appears in the brief; otherwise the template text is used. The guard is unit-tested with a stand-in client, so the pipeline never depends on the API.
Top reason by outcome, every Nov–Dec alert
Top reason
True alerts
False alerts
Stopping the card · Model & decision report7 · Cutoff and staffing
Net value stops rising at a 3% review budget, which at this volume is minutes of analyst time a day.
Net value = loss share × fraud dollars stopped − review time − card reissue − clearing false alerts − declined good purchases. The budget is chosen on Sep–Oct and applied unchanged to Nov–Dec.
Figure 7.1 · Net value by review budget, next-day blockingSep–Oct (chosen on)Nov–Dec (out-of-time)
Each window is two months. Steps appear where a large card gets caught earlier.
Why the curve is flat past 3%
Below 3% the score misses the early purchases of big runs, and value climbs in steps as each large card is caught earlier. Past 3% extra alerts are mostly false: going from 3% to 10% raises Nov–Dec alerts from to but fraud stopped only from to . Review costs are small ( of savings), so the budget is set by where value stops rising, not by analyst cost.
If analyst time is capped
Alerts worked per day
Nov–Dec net value
Unworked alerts
First come, first served; the budget re-chosen on Sep–Oct for each cap.
Scaled to a program of about 1M purchases a year, the same 3% budget means roughly analysts.
Stopping the card · Model & decision report7 · Uncertainty
Only one assumption moves the answer, how much of unstopped fraud the program really loses, and the result stays positive across the range.
Figure 7.2 · Nov–Dec net value when one assumption moves to the end of its range
Figure 7.3 · Net value across simulated outcomes
Assumptions drawn from triangular distributions over their ranges; compromised cards resampled with replacement.
Reading the range
The 90% range runs from to for two months, with a median of , and no draw below zero. Almost all of the width comes from which cards happen to be compromised: holding every cost input at its default and resampling cards alone gives –. With cards, one large compromise more or less moves the total by tens of thousands.
Why no annual number
Fraud dollars swing by month (Appendix A), so multiplying two months by six would mislead. The portable result is the share stopped: – of fraud dollars, depending on response speed.
Stopping the card · Model & decision report8 · Rollout and monitoring
A 60-day pilot can confirm the score works, not fine-tune it, so the go-live rule is set to what that sample can show.
Rollout in three steps
Shadow, 60 days. Score every purchase; analysts work the top 3% the same day but do not block. Log what a block would have stopped.
Same-day blocking. Switch on blocks for confirmed alerts. Keep a 10% random holdout of cards scored but not blocked as the comparison group.
Real-time declines for the highest scores. Only once the false-decline rate is known from steps 1–2.
How precise a pilot can be
Compromised cards seen
About
FDR@3% ± 95%
Card-bootstrap standard error at Nov–Dec, scaled by √cards; months at the Nov–Dec rate of 8 compromised cards a month.
A 60-day pilot pins detection only to about ±. The go-live rule is therefore "the top 3% catches at least 45%", which a working model clears and a broken one does not. Comparing two models needs a year of data, not a pilot.
Labels arrive late. Fraud is confirmed days to weeks after the purchase; measure FDR on a 30-day lag.
Retrain quarterly on the latest 9 months, re-run the time-fold check, and keep the old model as challenger until the new one matches it.
Every alert carries measured reasons and its band's fraud rate (§6), so analysts confirm in minutes and reviews are auditable.
Stopping the card · Model & decision report9 · Limitations and next steps
The direction of the result is robust; its size rests on one year, one program and sixteen compromised cards.
Limitations
One program, one year. Transactions are from 2010. Fraud tactics change; the time-fold design tests for drift over months, not years.
Small out-of-time sample. compromised cards in Nov–Dec. Shares are stable across validation and out-of-time in direction, not in size: next-day blocking stopped in Sep–Oct and in Nov–Dec.
Labels assumed complete. Undetected fraud would be counted as good, which understates both fraud and the model's catch.
Blocking is assumed to work. The replay counts every later fraud on a blocked card as stopped. A fraudster with several stolen cards would move on, so per-card value can be overstated.
Costs are placeholders. Review time, analyst cost and friction costs are agreed planning values, not the program's figures. Only the loss share materially changes the conclusion (§7).
Next steps, in order of value
Same-day operating model. The largest lever is process, not modeling: alerts worked within hours lift the share stopped from to .
Cardholder self-confirmation by text message for review-band alerts, to cut analyst time further.
Merchant-side detection. Compromises often share a merchant; a merchant-level score could block cards before their first fraud.
Real-time scoring service with the same point-in-time variables, for declines at the highest scores.
Longer history. Two to three years of data to measure year-over-year drift and tighten the value range.
What would change the recommendation
If the program recovers most fraud losses through chargebacks, the case weakens but holds: at a loss share of 0.3 the next-day policy still nets over Nov–Dec. If the pilot's FDR@3% comes in under 45%, stop and revisit the variables before going live.
Stopping the card · Model & decision reportA · Data quality report
Appendix A · Data quality report
Raw file before exclusions: records, 10 fields, one row per card transaction, Jan 1 – Dec 31 2010.
Field summary
Field
Type
Populated
Unique
Min
Median
Max
Most common share
Field notes
Recnum is a row identifier in date order; it breaks ties within a day.
Cardnum has no missing values; cards.
Merchnum is blank on of records and '0' on 237 more; recovered from descriptions where possible (§2).
Merch state includes numeric codes for foreign merchants, grouped as FOREIGN.
Merch zip is the least complete location field; imputed from merchant, then state.
Transtype is almost entirely P (purchase); the others carry no fraud and are excluded.
Amount runs from $0.01 to $3.1M; the $3.1M record is an error.
Fraud is the label: 1 for fraud.
Transaction types
Type
Records
Top merchant states
State
Records
Stopping the card · Model & decision reportA · Data quality report
Appendix A · Distributions
Figure A.1 · Purchase amount, share of records per bin (log scale)not fraudfraud
Bins evenly spaced on a log scale from $0.01 to $10M. Fraud skews to mid-sized purchases.
Figure A.2 · Records per monthFigure A.3 · Fraud rate per month
October volume is well below other months, and monthly fraud rates move between and . Both are reasons to judge the model on later months rather than a random sample.
Stopping the card · Model & decision reportB · Assumptions and reproducibility
Appendix B · Assumptions and reproducibility
Cost assumptions
Assumption
Used
Range tested
Planning values agreed for this analysis, not sourced benchmarks. Replace with the program's own figures; the decision tool recomputes every result.
Reproduce every number
fraudscore run
cleaning, variables, validation audit
fraudscore tune
time-fold search, final model
fraudscore decide
cutoff, staffing, response policy, uncertainty
fraudscore briefs
reason codes and analyst briefs
fraudscore export
aggregates for site, report, deck, README
fraudscore synth
synthetic data to run it all without the dataset
scripts/build.sh
PDFs and figures
fraudscore all runs everything in about eight minutes on a laptop. Unit tests cover cleaning order, point-in-time variables, split boundaries, FDR, PSI, replay policies, cost arithmetic, reason codes and the LLM grounding guard; CI also runs the whole pipeline on synthetic data.
Definitions
FDR@k%: share of fraud transactions found among the k% highest-scored transactions.
Out-of-time (OOT): Nov 1 – Dec 31 2010, never used for any choice.
Review budget: share of purchases raised as alerts, set as a score cutoff on Sep–Oct.
Prevented: fraud dollars declined because the card was already blocked (or, in real time, declined at the alert).
Data and authorship
About 98,000 labeled purchases from one purchase-card program in 2010. The transaction file is not redistributed; this report and the site publish only aggregates and a relabeled, rounded alert sample. Analysis, code and report: Jay Yu.