← All reportsRecorded BLOCK evidence
Custom Device Trust Score
custom-device-trust-score--block--20260807T171733Z-08ef15.md
Provenance Impact Report
- Run ID:
20260807T171733Z-08ef15
- State: PUBLISHED
- Started: 2026-08-07 17:17:33.093175+00:00
- Completed: 2026-08-07 17:17:47.590168+00:00
Proposed Change
- Asset:
urn:li:dataset:(urn:li:dataPlatform:kafka,payments.authorized_transaction_events,PROD)
- Source column:
device_trust_score
- Scenario: n/a
- Request: During a mobile SDK rollout, device_trust_score may be missing from 20 percent of authorized payment events.
DataHub Context (through MCP)
- Source:
urn:li:dataset:(urn:li:dataPlatform:kafka,payments.authorized_transaction_events,PROD) (payments.authorized_transaction_events)
- Selected field:
device_trust_score
- Owners:
urn:li:corpGroup:payment-fraud-monitoring
- Tags: Production, PaymentFraud, RealTimeML
Complete lineage
| Hop |
Upstream |
Downstream |
Relationship |
| 1 |
urn:li:dataset:(urn:li:dataPlatform:kafka,payments.authorized_transaction_events,PROD) |
urn:li:dataset:(urn:li:dataPlatform:postgres,fraud.realtime_payment_features,PROD) |
lineage |
| 2 |
urn:li:dataset:(urn:li:dataPlatform:postgres,fraud.realtime_payment_features,PROD) |
urn:li:mlFeature:(realtime_payment_risk_features,device_trust_score) |
lineage |
| 3 |
urn:li:mlFeature:(realtime_payment_risk_features,device_trust_score) |
urn:li:mlModel:(urn:li:dataPlatform:mlflow,payment_fraud_detector_v3,PROD) |
lineage |
Affected models
urn:li:mlModel:(urn:li:dataPlatform:mlflow,payment_fraud_detector_v3,PROD) ➔ payment_fraud_detector_v3
Evidence URNs
urn:li:dataset:(urn:li:dataPlatform:kafka,payments.authorized_transaction_events,PROD)
urn:li:dataset:(urn:li:dataPlatform:postgres,fraud.realtime_payment_features,PROD)
urn:li:mlFeature:(realtime_payment_risk_features,device_trust_score)
urn:li:mlModel:(urn:li:dataPlatform:mlflow,payment_fraud_detector_v3,PROD)
Experiment Plan
- Target asset:
urn:li:dataset:(urn:li:dataPlatform:kafka,payments.authorized_transaction_events,PROD)
- Rationale Explanation: The mobile SDK rollout may cause 'device_trust_score' to be missing for 20% of events, directly simulated by the first experiment. The additional experiments test scenarios where the score might be incorrectly shifted or scaled down due to potential SDK integration issues, which could impact the 'payment_fraud_detector_v3' model's performance.
- Expected signals: none
| # |
Operator |
Column |
Fraction |
Value |
| 1 |
inject_missing |
device_trust_score |
0.2 |
n/a |
| 2 |
shift_numeric |
device_trust_score |
1.0 |
-10.0 |
| 3 |
scale_numeric |
device_trust_score |
1.0 |
0.8 |
Observed Results
Experiment 1: inject_missing
- Column:
device_trust_score
- Duration (how long the mutation and mutated inference took)[ms]: 13.122199999997974
- Error: n/a
| Metric |
Observed value |
baseline_auc |
0.8092934782608696 |
mutated_auc |
0.8098913043478261 |
auc_drop |
0.0 |
prediction_flip_rate |
0.004 |
mean_absolute_probability_shift |
0.009298107226132295 |
inference_error_rate |
0.0 |
- Rows selected: 100 / 500
- Before summary:
{"maximum": 99.42, "mean": 72.66107999999998, "minimum": 22.51, "missing": 0}
- After summary:
{"maximum": 99.42, "mean": 72.071375, "minimum": 22.51, "missing": 100}
Experiment 2: shift_numeric
- Column:
device_trust_score
- Duration (how long the mutation and mutated inference took)[ms]: 22.74799999941024
- Error: n/a
| Metric |
Observed value |
baseline_auc |
0.8092934782608696 |
mutated_auc |
0.8092934782608696 |
auc_drop |
0.0 |
prediction_flip_rate |
0.974 |
mean_absolute_probability_shift |
0.9209403910589938 |
inference_error_rate |
0.0 |
- Rows selected: 500 / 500
- Before summary:
{"maximum": 99.42, "mean": 72.66107999999998, "minimum": 22.51, "missing": 0}
- After summary:
{"maximum": -54.51840834359696, "mean": -81.27732834359695, "minimum": -131.42840834359697, "missing": 0}
Experiment 3: scale_numeric
- Column:
device_trust_score
- Duration (how long the mutation and mutated inference took)[ms]: 5.5388000000675675
- Error: n/a
| Metric |
Observed value |
baseline_auc |
0.8092934782608696 |
mutated_auc |
0.8127173913043478 |
auc_drop |
0.0 |
prediction_flip_rate |
0.036 |
mean_absolute_probability_shift |
0.07322892038109145 |
inference_error_rate |
0.0 |
- Rows selected: 500 / 500
- Before summary:
{"maximum": 99.42, "mean": 72.66107999999998, "minimum": 22.51, "missing": 0}
- After summary:
{"maximum": 79.536, "mean": 58.128864, "minimum": 18.008000000000003, "missing": 0}
Deterministic Decision
- Status: BLOCK
- Summary: Blocked by 1 deterministic safety rule(s).
Triggered policy rules
| Experiment |
Metric |
Observed |
Rule |
Result |
Reason |
| 2 |
prediction_flip_rate |
0.974 |
> 0.15 |
BLOCK |
Prediction flip rate 0.974000 exceeded block threshold 0.150000 |
Publication
- Local JSON:
mockup_data/artifacts/runs/20260807T171733Z-08ef15.json
- Local Markdown:
mockup_data/artifacts/runs/20260807T171733Z-08ef15.md
- Remote succeeded: yes
- DataHub tag: Provenance-BLOCK
- DataHub document: urn:li:document:shared-cb72bb30-07e7-42f3-b49c-0718bc966cd4
- Remote error: n/a