Executive Summary

Understand False Positives & the 5-Layer Architecture

Why teams over-block

• Missing context features: merchant-level priors, device trust history, and account tenure often go unused.
• Static rules drift: rulebooks accrete; few get retired. Drift amplifies hidden correlations and over-blocking.
• Label gaps: appeals, partial refunds, and winbacks aren’t stitched back into labels, so models learn the wrong lesson.
• No counterfactuals: teams don’t measure “what would have happened if approved?”—so gains remain invisible.

Scope & assumptions

This guide targets card-not-present and BNPL flows with sub-second decision SLAs. We assume you can add features without vendor re-certification and can route a small canary cohort. If your stack is vendor-locked, you can still apply the feature and feedback sections via pre-decision enrichment.

Design the Data Pipeline: Signals → Features → Labels

Signals you likely already have (but underuse)

Turn signals into high-lift features

High-lift features are context-aware and time-bounded. They raise or lower risk only when the pattern persists. Examples you can compute in batch and cache to your decision service:

Label service: build truth you can trust

False positives hide when appeals and post-hoc approvals don’t flow back as labels. Stand up a label service that joins decisions, outcomes (appeals, chargebacks, refunds), and time windows. Lock it behind an idempotent API so analytics and training read the same truth.

-- Minimal label model (illustrative)
TABLE decisions (
  decision_id STRING, user_id STRING, device_id STRING, merchant_id STRING,
  ts TIMESTAMP, policy_version STRING, action STRING  -- approve/review/stepup/decline
);

TABLE outcomes (
  decision_id STRING, outcome STRING, ts TIMESTAMP,  -- appeal_approved / chargeback / refund / none
  amount_cents INT64
);

VIEW labels AS
SELECT d.decision_id,
       CASE
         WHEN o.outcome='appeal_approved' THEN 'LEGIT_AFTER_DECLINE'  -- key to FP measurement
         WHEN o.outcome='chargeback' THEN 'FRAUD_CONFIRMED'
         ELSE 'UNRESOLVED'
       END AS label,
       d.ts AS decision_ts, o.ts AS outcome_ts
FROM decisions d
LEFT JOIN outcomes o USING (decision_id);
      

Decisioning & Experiments: How to Hit 30% in 90 Days

Policy stack: rules + ML working together

Keep deterministic rules for compliance and obvious fraud; let ML arbitrate the gray zone. Wrap both in a friction router with four outcomes: Approve, Step-Up (e.g., 3-DS/OTP), Manual Review, Decline.

30-60-90 plan (ship-to-learn)

Copy-paste templates

features:
  - name: device_trust
    freshness: "24h"
    inputs: [device_id, approvals_90d, stepup_pass_90d, stability_score]
    transform: >
      0.4*stability_score + 0.3*log1p(approvals_90d) + 0.3*stepup_pass_90d
  - name: merchant_prior
    freshness: "daily"
    inputs: [merchant_id, approvals_180d, disputes_180d]
    transform: "bayes_smooth(disputes_180d/approvals_180d, alpha=2, beta=50)"
          

WITH shadow AS (
  SELECT decision_id, old_score, new_score, label
  FROM shadow_eval
  WHERE ts >= TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 90 DAY)
)
SELECT
  APPROX_QUANTILES(new_score, 100)[OFFSET(10)] AS t1_10p,
  APPROX_QUANTILES(new_score, 100)[OFFSET(50)] AS t2_50p,
  SUM(label='LEGIT_AFTER_DECLINE') / COUNTIF(old_score>t2_50p) AS fp_recovery_rate
FROM shadow;
          

Benchmarks, Case Studies & How This Differs

Performance benchmarking (baseline vs improved)

Numbers are representative ranges for mid-market fintechs in the last 12 months; use your label service to compute exact values.

Mini case studies

How this differs from typical advice

• Counterfactuals first: We prove gains before rollout, rather than “test in prod and pray”.
• Context features vs. rule bloat: We reduce over-blocking by adding priors and trust, not hundreds of new rules.
• Safety budgets baked-in: Risk guardrails are part of the deployment, not an afterthought.

Operational Playbook, FAQ, Glossary & Conclusion

Runbook: daily/weekly cadence

Download-ready stubs (copy-paste)

{
  "policy_version": "v1.3",
  "thresholds": { "t1": 0.22, "t2": 0.44, "t3": 0.73 },
  "actions": ["APPROVE", "STEP_UP", "REVIEW", "DECLINE"],
  "guardrails": {
    "loss_budget_delta_pct": 3.0,
    "review_queue_max": 1.25
  },
  "canary": { "cohort": "returning_trusted_devices", "traffic_pct": 10 }
}
          

alerts:
  - name: feature_drift
    metric: "population_stability_index"
    threshold: 0.25
    evaluate: "daily"
  - name: score_drift
    metric: "kl_divergence"
    threshold: 0.08
    evaluate: "daily"
  - name: appeal_spike
    metric: "appeal_approved_rate"
    threshold: "mean+3sigma"
    evaluate: "hourly"
          

When not to use this playbook

• Your decisions are post-factum (e.g., only after fulfillment) and real-time signals are absent.
• You can’t route a canary or change thresholds within a rolling 90-day window.
• Regulatory or scheme rules force outcomes that a router can’t override.

Router vs. rule-only: quick comparison

References (authoritative)

• NIST Digital Identity Guidelines (800-63)
• Thresholding & Metrics (Google MLCC)
• ML Systems best practices (MLSys community)

Contextual internal links

• Feature Store Starter (this page)
• Counterfactual Replay Playbook (this page)
• Router vs. Rule-only Comparison (this page)

FAQ (intent-based)

Glossary (≤10 terms)

False Positive (FP)

A legit user incorrectly blocked; operational proxy: appeal-approved after decline.

Shadow Evaluation

Run new policy in parallel; compare outcomes without affecting live decisions.

Counterfactual Replay

Estimate what would have happened under different thresholds using logged data.

Safety Budget

Maximum allowed loss delta and review capacity during experiments.

Device Trust

Score capturing device stability, age, and prior approvals.

Merchant Prior

Merchant-level risk prior with Bayesian smoothing to prevent overfitting.

Graph Proximity

Relationship strength to known bad actors over shared attributes.

Step-Up

Additional verification (e.g., OTP) for gray-zone decisions.

Drift

Distribution change in features or scores that can degrade performance.

Canary Cohort

Small, safer traffic slice for initial rollout and monitoring.

Conclusion

Reducing false positives is a systems win, not just a model tweak. By instrumenting labels, adding a handful of high-lift context features, and deploying a router with budgets, you can unlock a meaningful improvement—on the order of 30%—within a rolling 90 days, while keeping losses in check. The secret is counterfactuals, safe canaries, and ruthless observability.