ⓘ Read-only hosted preview!! no analysis or DataHub mutation is executed. Clone the repository to run the live workflow.

Provenance Logo
Production ML agent View repository Run locally with DataHub →
← All reports

Recorded BLOCK evidence

Custom Device Trust Score

custom-device-trust-score--block--20260807T171733Z-08ef15.md

Provenance Impact Report

  • Run ID: 20260807T171733Z-08ef15
  • State: PUBLISHED
  • Started: 2026-08-07 17:17:33.093175+00:00
  • Completed: 2026-08-07 17:17:47.590168+00:00

Proposed Change

  • Asset: urn:li:dataset:(urn:li:dataPlatform:kafka,payments.authorized_transaction_events,PROD)
  • Source column: device_trust_score
  • Scenario: n/a
  • Request: During a mobile SDK rollout, device_trust_score may be missing from 20 percent of authorized payment events.

DataHub Context (through MCP)

  • Source: urn:li:dataset:(urn:li:dataPlatform:kafka,payments.authorized_transaction_events,PROD) (payments.authorized_transaction_events)
  • Selected field: device_trust_score
  • Owners: urn:li:corpGroup:payment-fraud-monitoring
  • Tags: Production, PaymentFraud, RealTimeML

Complete lineage

Hop Upstream Downstream Relationship
1 urn:li:dataset:(urn:li:dataPlatform:kafka,payments.authorized_transaction_events,PROD) urn:li:dataset:(urn:li:dataPlatform:postgres,fraud.realtime_payment_features,PROD) lineage
2 urn:li:dataset:(urn:li:dataPlatform:postgres,fraud.realtime_payment_features,PROD) urn:li:mlFeature:(realtime_payment_risk_features,device_trust_score) lineage
3 urn:li:mlFeature:(realtime_payment_risk_features,device_trust_score) urn:li:mlModel:(urn:li:dataPlatform:mlflow,payment_fraud_detector_v3,PROD) lineage

Affected models

  • urn:li:mlModel:(urn:li:dataPlatform:mlflow,payment_fraud_detector_v3,PROD) ➔ payment_fraud_detector_v3

Evidence URNs

  • urn:li:dataset:(urn:li:dataPlatform:kafka,payments.authorized_transaction_events,PROD)
  • urn:li:dataset:(urn:li:dataPlatform:postgres,fraud.realtime_payment_features,PROD)
  • urn:li:mlFeature:(realtime_payment_risk_features,device_trust_score)
  • urn:li:mlModel:(urn:li:dataPlatform:mlflow,payment_fraud_detector_v3,PROD)

Experiment Plan

  • Target asset: urn:li:dataset:(urn:li:dataPlatform:kafka,payments.authorized_transaction_events,PROD)
  • Rationale Explanation: The mobile SDK rollout may cause 'device_trust_score' to be missing for 20% of events, directly simulated by the first experiment. The additional experiments test scenarios where the score might be incorrectly shifted or scaled down due to potential SDK integration issues, which could impact the 'payment_fraud_detector_v3' model's performance.
  • Expected signals: none
# Operator Column Fraction Value
1 inject_missing device_trust_score 0.2 n/a
2 shift_numeric device_trust_score 1.0 -10.0
3 scale_numeric device_trust_score 1.0 0.8

Observed Results

Experiment 1: inject_missing

  • Column: device_trust_score
  • Duration (how long the mutation and mutated inference took)[ms]: 13.122199999997974
  • Error: n/a
Metric Observed value
baseline_auc 0.8092934782608696
mutated_auc 0.8098913043478261
auc_drop 0.0
prediction_flip_rate 0.004
mean_absolute_probability_shift 0.009298107226132295
inference_error_rate 0.0
  • Rows selected: 100 / 500
  • Before summary: {"maximum": 99.42, "mean": 72.66107999999998, "minimum": 22.51, "missing": 0}
  • After summary: {"maximum": 99.42, "mean": 72.071375, "minimum": 22.51, "missing": 100}

Experiment 2: shift_numeric

  • Column: device_trust_score
  • Duration (how long the mutation and mutated inference took)[ms]: 22.74799999941024
  • Error: n/a
Metric Observed value
baseline_auc 0.8092934782608696
mutated_auc 0.8092934782608696
auc_drop 0.0
prediction_flip_rate 0.974
mean_absolute_probability_shift 0.9209403910589938
inference_error_rate 0.0
  • Rows selected: 500 / 500
  • Before summary: {"maximum": 99.42, "mean": 72.66107999999998, "minimum": 22.51, "missing": 0}
  • After summary: {"maximum": -54.51840834359696, "mean": -81.27732834359695, "minimum": -131.42840834359697, "missing": 0}

Experiment 3: scale_numeric

  • Column: device_trust_score
  • Duration (how long the mutation and mutated inference took)[ms]: 5.5388000000675675
  • Error: n/a
Metric Observed value
baseline_auc 0.8092934782608696
mutated_auc 0.8127173913043478
auc_drop 0.0
prediction_flip_rate 0.036
mean_absolute_probability_shift 0.07322892038109145
inference_error_rate 0.0
  • Rows selected: 500 / 500
  • Before summary: {"maximum": 99.42, "mean": 72.66107999999998, "minimum": 22.51, "missing": 0}
  • After summary: {"maximum": 79.536, "mean": 58.128864, "minimum": 18.008000000000003, "missing": 0}

Deterministic Decision

  • Status: BLOCK
  • Summary: Blocked by 1 deterministic safety rule(s).

Triggered policy rules

Experiment Metric Observed Rule Result Reason
2 prediction_flip_rate 0.974 > 0.15 BLOCK Prediction flip rate 0.974000 exceeded block threshold 0.150000

Publication

  • Local JSON: mockup_data/artifacts/runs/20260807T171733Z-08ef15.json
  • Local Markdown: mockup_data/artifacts/runs/20260807T171733Z-08ef15.md
  • Remote succeeded: yes
  • DataHub tag: Provenance-BLOCK
  • DataHub document: urn:li:document:shared-cb72bb30-07e7-42f3-b49c-0718bc966cd4
  • Remote error: n/a