← All reportsRecorded BLOCK evidence
Device trust score contract change
device-trust-range-change--block--20260807T140453Z-857303.md
Provenance Impact Report
- Run ID:
20260807T140453Z-857303
- State: PUBLISHED
- Started: 2026-08-07 14:04:53.120572+00:00
- Completed: 2026-08-07 14:04:58.493510+00:00
Proposed Change
- Asset:
urn:li:dataset:(urn:li:dataPlatform:kafka,payments.authorized_transaction_events,PROD)
- Source column:
device_trust_score
- Scenario: device-trust-range-change
- Request: The device-risk producer will change device_trust_score from a 0–100 range to a 0–1 range without updating downstream expectations.
- Trigger event:
evt-dd8cd270-a249-4a94-9958-1328a6363c9d
- Trigger source:
device-risk-deployment-agent
DataHub Context (through MCP)
- Source:
urn:li:dataset:(urn:li:dataPlatform:kafka,payments.authorized_transaction_events,PROD) (payments.authorized_transaction_events)
- Selected field:
device_trust_score
- Owners:
urn:li:corpGroup:payment-fraud-monitoring
- Tags: Production, PaymentFraud, RealTimeML
Complete lineage
| Hop |
Upstream |
Downstream |
Relationship |
| 1 |
urn:li:dataset:(urn:li:dataPlatform:kafka,payments.authorized_transaction_events,PROD) |
urn:li:dataset:(urn:li:dataPlatform:postgres,fraud.realtime_payment_features,PROD) |
lineage |
| 2 |
urn:li:dataset:(urn:li:dataPlatform:postgres,fraud.realtime_payment_features,PROD) |
urn:li:mlFeature:(realtime_payment_risk_features,device_trust_score) |
lineage |
| 3 |
urn:li:mlFeature:(realtime_payment_risk_features,device_trust_score) |
urn:li:mlModel:(urn:li:dataPlatform:mlflow,payment_fraud_detector_v3,PROD) |
lineage |
Affected models
urn:li:mlModel:(urn:li:dataPlatform:mlflow,payment_fraud_detector_v3,PROD) ➔ payment_fraud_detector_v3
Evidence URNs
urn:li:dataset:(urn:li:dataPlatform:kafka,payments.authorized_transaction_events,PROD)urn:li:dataset:(urn:li:dataPlatform:postgres,fraud.realtime_payment_features,PROD)urn:li:mlFeature:(realtime_payment_risk_features,device_trust_score)urn:li:mlModel:(urn:li:dataPlatform:mlflow,payment_fraud_detector_v3,PROD)
Experiment Plan
- Target asset:
urn:li:dataset:(urn:li:dataPlatform:kafka,payments.authorized_transaction_events,PROD)
- Rationale Explanation: Reproduce the proposed device-score range change across the full evaluation population and measure its effect on the downstream fraud model.
- Expected signals: AUC drop, prediction flip rate, mean absolute probability shift
| # |
Operator |
Column |
Fraction |
Value |
| 1 |
scale_numeric |
device_trust_score |
1.0 |
0.01 |
Observed Results
Experiment 1: scale_numeric
- Column:
device_trust_score
- Duration (how long the mutation and mutated inference took)[ms]: 17.902000000049156
- Error: n/a
| Metric |
Observed value |
baseline_auc |
0.8092934782608696 |
mutated_auc |
0.7495108695652174 |
auc_drop |
0.05978260869565222 |
prediction_flip_rate |
0.974 |
mean_absolute_probability_shift |
0.8007552497510132 |
inference_error_rate |
0.0 |
- Rows selected: 500 / 500
- Before summary:
{"maximum": 99.42, "mean": 72.66107999999998, "minimum": 22.51, "missing": 0}
- After summary:
{"maximum": 0.9942000000000001, "mean": 0.7266107999999999, "minimum": 0.22510000000000002, "missing": 0}
Deterministic Decision
- Status: BLOCK
- Summary: Blocked by 2 deterministic safety rule(s).
Triggered policy rules
| Experiment |
Metric |
Observed |
Rule |
Result |
Reason |
| 1 |
auc_drop |
0.05978260869565222 |
> 0.05 |
BLOCK |
AUC drop 0.059783 exceeded block threshold 0.050000 |
| 1 |
prediction_flip_rate |
0.974 |
> 0.15 |
BLOCK |
Prediction flip rate 0.974000 exceeded block threshold 0.150000 |
Publication
- Local JSON:
mockup_data/artifacts/runs/20260807T140453Z-857303.json
- Local Markdown:
mockup_data/artifacts/runs/20260807T140453Z-857303.md
- Remote succeeded: yes
- DataHub tag: Provenance-BLOCK
- DataHub document: urn:li:document:shared-1f1429aa-2bb7-42c8-b76b-d717065e3b58
- Remote error: n/a
