# Provenance Impact Report

- Run ID: `20260807T102721Z-ebb37c`
- State: **PUBLISHED**
- Started: 2026-08-07 10:27:21.951596+00:00
- Completed: 2026-08-07 10:27:37.773747+00:00

## Proposed Change

- Asset: `urn:li:dataset:(urn:li:dataPlatform:kafka,payments.authorized_transaction_events,PROD)`
- Source column: `device_trust_score`
- Scenario: n/a
- Request: During a mobile SDK rollout, device_trust_score may be missing from 20 percent of authorized payment events.

## DataHub Context (through MCP)

- Source: `urn:li:dataset:(urn:li:dataPlatform:kafka,payments.authorized_transaction_events,PROD)` (payments.authorized_transaction_events)
- Selected field: `device_trust_score`
- Owners: `urn:li:corpGroup:payment-fraud-monitoring`
- Tags: Production, PaymentFraud, RealTimeML

### Complete lineage

| Hop | Upstream | Downstream | Relationship |
| ---: | --- | --- | --- |
| 1 | `urn:li:dataset:(urn:li:dataPlatform:kafka,payments.authorized_transaction_events,PROD)` | `urn:li:dataset:(urn:li:dataPlatform:postgres,fraud.realtime_payment_features,PROD)` | lineage |
| 2 | `urn:li:dataset:(urn:li:dataPlatform:postgres,fraud.realtime_payment_features,PROD)` | `urn:li:mlFeature:(realtime_payment_risk_features,device_trust_score)` | lineage |
| 3 | `urn:li:mlFeature:(realtime_payment_risk_features,device_trust_score)` | `urn:li:mlModel:(urn:li:dataPlatform:mlflow,payment_fraud_detector_v3,PROD)` | lineage |

### Affected models

- `urn:li:mlModel:(urn:li:dataPlatform:mlflow,payment_fraud_detector_v3,PROD)` ➔ payment_fraud_detector_v3

### Evidence URNs

- `urn:li:dataset:(urn:li:dataPlatform:kafka,payments.authorized_transaction_events,PROD)`
- `urn:li:dataset:(urn:li:dataPlatform:postgres,fraud.realtime_payment_features,PROD)`
- `urn:li:mlFeature:(realtime_payment_risk_features,device_trust_score)`
- `urn:li:mlModel:(urn:li:dataPlatform:mlflow,payment_fraud_detector_v3,PROD)`

## Experiment Plan

- Target asset: `urn:li:dataset:(urn:li:dataPlatform:kafka,payments.authorized_transaction_events,PROD)`
- Rationale Explanation: **_Simulate the impact of missing device_trust_score on payment fraud detection due to a mobile SDK rollout. Experiment 1 directly tests the described 20% missing data. Experiment 2 assesses the effect of a general degradation in trust scores, potentially indicating a broader issue with the SDK's data quality. Experiment 3 evaluates the model's resilience if the device_trust_score feature becomes entirely unavailable, providing insight into its dependency on this critical input._**
- Expected signals: none

| # | Operator | Column | Fraction | Value |
| ---: | --- | --- | ---: | --- |
| 1 | `inject_missing` | `device_trust_score` | 0.2 | n/a |
| 2 | `shift_numeric` | `device_trust_score` | 1.0 | -10.0 |
| 3 | `drop_column` | `device_trust_score` | 1.0 | n/a |

## Observed Results

### Experiment 1: `inject_missing`

- Column: `device_trust_score`
- Duration (_how long the mutation and mutated inference took_)[ms]: 29.910900000686524
- Error: n/a

| Metric | Observed value |
| --- | ---: |
| `baseline_auc` | 0.8092934782608696 |
| `mutated_auc` | 0.8098913043478261 |
| `auc_drop` | 0.0 |
| `prediction_flip_rate` | 0.004 |
| `mean_absolute_probability_shift` | 0.009298107226132295 |
| `inference_error_rate` | 0.0 |

- Rows selected: 100 / 500
- Before summary: `{"maximum": 99.42, "mean": 72.66107999999998, "minimum": 22.51, "missing": 0}`
- After summary: `{"maximum": 99.42, "mean": 72.071375, "minimum": 22.51, "missing": 100}`

### Experiment 2: `shift_numeric`

- Column: `device_trust_score`
- Duration (_how long the mutation and mutated inference took_)[ms]: 6.056299998817849
- Error: n/a

| Metric | Observed value |
| --- | ---: |
| `baseline_auc` | 0.8092934782608696 |
| `mutated_auc` | 0.8092934782608696 |
| `auc_drop` | 0.0 |
| `prediction_flip_rate` | 0.974 |
| `mean_absolute_probability_shift` | 0.9209403910589938 |
| `inference_error_rate` | 0.0 |

- Rows selected: 500 / 500
- Before summary: `{"maximum": 99.42, "mean": 72.66107999999998, "minimum": 22.51, "missing": 0}`
- After summary: `{"maximum": -54.51840834359696, "mean": -81.27732834359695, "minimum": -131.42840834359697, "missing": 0}`

### Experiment 3: `drop_column`

- Column: `device_trust_score`
- Duration (_how long the mutation and mutated inference took_)[ms]: 0.6868999989819713
- Error: KeyError: "['device_trust_score'] not in index"

| Metric | Observed value |
| --- | ---: |
| `baseline_auc` | 0.8092934782608696 |
| `mutated_auc` | n/a |
| `auc_drop` | n/a |
| `prediction_flip_rate` | 0.0 |
| `mean_absolute_probability_shift` | 0.0 |
| `inference_error_rate` | 1.0 |

- Rows selected: 500 / 500
- Before summary: `{"maximum": 99.42, "mean": 72.66107999999998, "minimum": 22.51, "missing": 0}`
- After summary: `{"column_present": "false"}`

## Deterministic Decision

- Status: **BLOCK**
- Summary: Blocked by 2 deterministic safety rule(s).

### Triggered policy rules

| Experiment | Metric | Observed | Rule | Result | Reason |
| ---: | --- | ---: | --- | --- | --- |
| 2 | `prediction_flip_rate` | 0.974 | > 0.15 | BLOCK | Prediction flip rate 0.974000 exceeded block threshold 0.150000 |
| 3 | `inference_error_rate` | 1.0 | > 0.0 | BLOCK | Inference error rate 1.000000 exceeded block threshold 0.000000 |

## Publication

- Local JSON: `mockup_data/artifacts/runs/20260807T102721Z-ebb37c.json`
- Local Markdown: `mockup_data/artifacts/runs/20260807T102721Z-ebb37c.md`
- Remote succeeded: yes
- DataHub tag: Provenance-BLOCK
- DataHub document: urn:li:document:shared-5e0e21cf-ead9-4924-a049-3f893f4801c0
- Remote error: n/a