The Outlier Lab

Rule Governance and Change Management

How to govern a fraud rule portfolio: inventory fields, materiality tiers, approval paths, documentation, periodic performance review, emergency changes and sunsetting, so rules stay effective and explainable.

Core components

What counts as a "rule"

Govern anything that changes a transaction outcome and isn't already governed as a model:

Rule inventory

Start with the inventory. Every production rule gets a record:

FieldPurpose
Rule ID and nameUnique, stable identifier
Description (plain language)What it does, for a non-author
Money path and decision pointWhere it acts (see Real-Time Decisioning Architecture)
Mechanism(s) targetedLink to taxonomy
ActionDecline / hold / step-up / review / limit / flag
OwnerNamed person, not a team
Materiality tierSee below
StatusDraft / shadow / live / emergency / retired
Created date, author, approverAudit trail
Rationale and evidenceLink to strategy proposal / backtest (see Fraud Strategy Lifecycle)
Expected performanceHit rate, precision, catch, friction at approval
Last review date and outcomeGovernance hygiene
Next review dateSet by tier
Expiry date (emergency rules)Automatic sunset
DependenciesSignals, vendors, lists, other rules
Customer-facing reason code and messagingExplainability, complaints
Regulatory relevanceE.g., affects Reg E claims, adverse action, SAR referral

Materiality tiers (illustrative)

TierCriteriaApprovalReview cadence
Tier 1Hard decline or account action on money paths; affects a large share of traffic; regulatory relevance; large dollar exposureStrategy lead + head of fraud; second-line review where applicable; product consulted on frictionMonthly performance, quarterly deep review
Tier 2Holds, step-ups, limits; moderate volumeStrategy peer review + strategy leadQuarterly
Tier 3Review routing, flags, low-volume rulesPeer reviewSemi-annually
EmergencyCreated under incident authorityIncident commander; logged immediatelyExpires in N days unless backfilled and re-approved

How to build it

Change workflow

Diagram: Proposal (strategy template), Peer review: logic, backtest, impact, Approval by tier, Shadow (if Tier 1-2), Deploy (staged), Post-deploy verification (24-72h), Inventory updated, Scheduled review

Peer review checklist

  • Logic matches the stated intent; edge cases considered (nulls, new accounts, currency, time zones)
  • Backtest uses matured labels and no look-ahead features
  • Incremental value vs existing rules measured
  • Good-customer impact estimated by segment (see Friction and Customer Experience Tradeoffs)
  • Ops capacity confirmed for review actions (see Alert Review Operations)
  • Reason code and customer messaging defined
  • Rollback plan and trigger defined
  • Dependencies documented (signals, lists, vendors)
  • No conflict with other rules (order of evaluation, precedence)

Post-deploy verification

  • Hit volume within the expected range
  • No unexpected decline-rate shift in unrelated segments
  • No errors or latency regressions
  • QA spot-check of early hits (see Fraud QA Program)

Periodic performance review

MeasureQuestionAction
Hit volume vs expectationIs it firing as designed?Investigate drift
Precision (labels or QA sample)Are hits still mostly bad?Retune or retire
Incremental catchDoes it catch fraud nothing else catches?Consolidate if not
Friction costHow many good customers does it affect?Lighter action, narrower scope
OverlapWhat share of its hits do other rules also catch?Consolidate
Data healthAre its inputs available and stable?Fix or add fallback
OwnerIs the owner still here and responsible?Reassign

Sunsetting

Retire a rule when:

  • Incremental catch is below threshold for N reviews
  • Precision is below threshold and can't be fixed by tuning
  • It's superseded by a model or a broader rule
  • The product or flow no longer exists
  • It's an emergency rule past expiry without backfill
  • It has no owner and nobody will take it

Retire through shadow first: switch the action to log-only for a few weeks and watch the target population before full removal.

Separation of duties

  • The author can't be the sole approver for Tier 1–2
  • Production rule changes are logged with who, what, when and why. Rules engines should enforce this (see Rules Engine).
  • Access to change production rules is restricted and reviewed periodically. Rule access is an Insider Fraud vector: someone who can allow-list an account can launder through it (see Insider Access Controls).
  • Emergency changes are reviewed after the fact by someone other than the person who made them

Artifacts and templates

Rule change request (short form)

  1. Rule ID / new
  2. Change type: new / modify / retire / emergency
  3. What and why (2–3 sentences)
  4. Evidence: backtest link, shadow results
  5. Expected impact: hits/day, precision, catch $, good customers affected
  6. Tier and approvers
  7. Rollout and rollback plan
  8. Ops and support impact
  9. Reviewer sign-off

Quarterly rule portfolio review agenda

  1. Inventory stats: live rules by tier, emergency rules, rules past review date
  2. Top 10 rules by incremental catch
  3. Bottom 10 by precision or incremental value: retire or retune?
  4. Overlap clusters
  5. Friction leaders: rules affecting the most good customers
  6. Data dependency issues
  7. Decisions and owners

Governance health checklist

  • Every production rule has an owner, tier, description and review date
  • No emergency rules past expiry
  • Change log is complete and tamper-evident
  • Tier 1 rules reviewed in the last quarter
  • Rule edit access reviewed in the last 6 months
  • Reason codes mapped for every customer-facing action
  • Second-line sampling of rule changes, where applicable (see Three Lines of Defense)

Metrics

MetricTarget direction
Rules without an ownerZero
Rules past review dateNear zero
Emergency rules past expiryZero
Median change lead time (normal path)Short enough not to be bypassed
Rules retired per quarterNon-zero
Share of rules with documented backtestHigh
Post-deploy rollbacksLow; investigate causes

Pitfalls

  • Governance so heavy that people bypass it. If a Tier 3 rule change takes two weeks, analysts will ask engineers for "config tweaks" off the books.
  • No expiry on emergency rules. This is the main source of sprawl.
  • Inventory as a spreadsheet nobody updates. Generate the inventory from the rules engine and add metadata. Don't maintain two sources of truth.
  • Measuring rules individually. Overlap hides dead weight.
  • Vendor rules out of scope. Vendor-configured rules affect your customers. Govern them the same way.
  • No reason codes. When support and complaints teams can't explain a decision, customers escalate, and regulators may ask.

Sources and further reading