ⓘ Read-only hosted preview!! no analysis or DataHub mutation is executed. Clone the repository to run the live workflow.

Provenance Logo
Production ML agent View repository Run locally with DataHub →
← All reports

Recorded BLOCK evidence

Custom Device Trust Score

custom-device-trust-score--block--20260807T102721Z-ebb37c.md

Download Markdown

Provenance Impact Report

  • Run ID: 20260807T102721Z-ebb37c
  • State: PUBLISHED
  • Started: 2026-08-07 10:27:21.951596+00:00
  • Completed: 2026-08-07 10:27:37.773747+00:00

Proposed Change

  • Asset: urn:li:dataset:(urn:li:dataPlatform:kafka,payments.authorized_transaction_events,PROD)
  • Source column: device_trust_score
  • Scenario: n/a
  • Request: During a mobile SDK rollout, device_trust_score may be missing from 20 percent of authorized payment events.

DataHub Context (through MCP)

  • Source: urn:li:dataset:(urn:li:dataPlatform:kafka,payments.authorized_transaction_events,PROD) (payments.authorized_transaction_events)
  • Selected field: device_trust_score
  • Owners: urn:li:corpGroup:payment-fraud-monitoring
  • Tags: Production, PaymentFraud, RealTimeML

Complete lineage

Hop Upstream Downstream Relationship
1 urn:li:dataset:(urn:li:dataPlatform:kafka,payments.authorized_transaction_events,PROD) urn:li:dataset:(urn:li:dataPlatform:postgres,fraud.realtime_payment_features,PROD) lineage
2 urn:li:dataset:(urn:li:dataPlatform:postgres,fraud.realtime_payment_features,PROD) urn:li:mlFeature:(realtime_payment_risk_features,device_trust_score) lineage
3 urn:li:mlFeature:(realtime_payment_risk_features,device_trust_score) urn:li:mlModel:(urn:li:dataPlatform:mlflow,payment_fraud_detector_v3,PROD) lineage

Affected models

  • urn:li:mlModel:(urn:li:dataPlatform:mlflow,payment_fraud_detector_v3,PROD) ➔ payment_fraud_detector_v3

Evidence URNs

  • urn:li:dataset:(urn:li:dataPlatform:kafka,payments.authorized_transaction_events,PROD)
  • urn:li:dataset:(urn:li:dataPlatform:postgres,fraud.realtime_payment_features,PROD)
  • urn:li:mlFeature:(realtime_payment_risk_features,device_trust_score)
  • urn:li:mlModel:(urn:li:dataPlatform:mlflow,payment_fraud_detector_v3,PROD)

Experiment Plan

  • Target asset: urn:li:dataset:(urn:li:dataPlatform:kafka,payments.authorized_transaction_events,PROD)
  • Rationale Explanation: Simulate the impact of missing device_trust_score on payment fraud detection due to a mobile SDK rollout. Experiment 1 directly tests the described 20% missing data. Experiment 2 assesses the effect of a general degradation in trust scores, potentially indicating a broader issue with the SDK's data quality. Experiment 3 evaluates the model's resilience if the device_trust_score feature becomes entirely unavailable, providing insight into its dependency on this critical input.
  • Expected signals: none
# Operator Column Fraction Value
1 inject_missing device_trust_score 0.2 n/a
2 shift_numeric device_trust_score 1.0 -10.0
3 drop_column device_trust_score 1.0 n/a

Observed Results

Experiment 1: inject_missing

  • Column: device_trust_score
  • Duration (how long the mutation and mutated inference took)[ms]: 29.910900000686524
  • Error: n/a
Metric Observed value
baseline_auc 0.8092934782608696
mutated_auc 0.8098913043478261
auc_drop 0.0
prediction_flip_rate 0.004
mean_absolute_probability_shift 0.009298107226132295
inference_error_rate 0.0
  • Rows selected: 100 / 500
  • Before summary: {"maximum": 99.42, "mean": 72.66107999999998, "minimum": 22.51, "missing": 0}
  • After summary: {"maximum": 99.42, "mean": 72.071375, "minimum": 22.51, "missing": 100}

Experiment 2: shift_numeric

  • Column: device_trust_score
  • Duration (how long the mutation and mutated inference took)[ms]: 6.056299998817849
  • Error: n/a
Metric Observed value
baseline_auc 0.8092934782608696
mutated_auc 0.8092934782608696
auc_drop 0.0
prediction_flip_rate 0.974
mean_absolute_probability_shift 0.9209403910589938
inference_error_rate 0.0
  • Rows selected: 500 / 500
  • Before summary: {"maximum": 99.42, "mean": 72.66107999999998, "minimum": 22.51, "missing": 0}
  • After summary: {"maximum": -54.51840834359696, "mean": -81.27732834359695, "minimum": -131.42840834359697, "missing": 0}

Experiment 3: drop_column

  • Column: device_trust_score
  • Duration (how long the mutation and mutated inference took)[ms]: 0.6868999989819713
  • Error: KeyError: "['device_trust_score'] not in index"
Metric Observed value
baseline_auc 0.8092934782608696
mutated_auc n/a
auc_drop n/a
prediction_flip_rate 0.0
mean_absolute_probability_shift 0.0
inference_error_rate 1.0
  • Rows selected: 500 / 500
  • Before summary: {"maximum": 99.42, "mean": 72.66107999999998, "minimum": 22.51, "missing": 0}
  • After summary: {"column_present": "false"}

Deterministic Decision

  • Status: BLOCK
  • Summary: Blocked by 2 deterministic safety rule(s).

Triggered policy rules

Experiment Metric Observed Rule Result Reason
2 prediction_flip_rate 0.974 > 0.15 BLOCK Prediction flip rate 0.974000 exceeded block threshold 0.150000
3 inference_error_rate 1.0 > 0.0 BLOCK Inference error rate 1.000000 exceeded block threshold 0.000000

Publication

  • Local JSON: mockup_data/artifacts/runs/20260807T102721Z-ebb37c.json
  • Local Markdown: mockup_data/artifacts/runs/20260807T102721Z-ebb37c.md
  • Remote succeeded: yes
  • DataHub tag: Provenance-BLOCK
  • DataHub document: urn:li:document:shared-5e0e21cf-ead9-4924-a049-3f893f4801c0
  • Remote error: n/a