Hidden-counterparty backfill — verification report

PR #24456 · feat(cash): admin backfill to tombstone hidden-counterparty activities

Branch · lisherwin/backfill-activity-hide-counterparties Base · main Linear · Relates to CASH-3669 Generated · 2026-05-27 Tests · 55/55 passing Prod verification · 21% supersede @ 5s window per William's guidance — every match same-block-sibling, RPC-verified

Production verification

Hit the live admin API with a real admin token to verify the backfill's decision tree against actual cash_activity rows. All calls were read-only (GET); no enqueue, no mutation.

Probe	Endpoint	Result
Auth check	`GET /cash/v1/admin/checkAuth`	HTTP 200
Most-recent transfers sampled	`GET /cash/v1/admin/cash-activities?type=transfer&limit=100` × 10 pages	1000 transfers scanned
babykiss22-counterparty rows in that sample	client-side filter on `counterparty.address == FIREBLOCKS_FEE_PAYER_ADDRESS`	394 / 1000 (39%)
Already-superseded (William's live ingest worked)	`supersededBy != null`	1 / 394
Un-superseded (this PR's target rows)	`supersededBy == null`	393 / 394
`counterparty.type` distribution	—	100% `phantom_username` (BNS enrichment hit on every row — exactly the case Will's #24373 widened the filter for)
State distribution	—	100% `completed`
Correlation strategy	5-second time window per William's guidance	For each babykiss22 fee row: look up the user's most recent non-superseded Swap with `occurredAt` in [fee.occurredAt − 5s, fee.occurredAt]. Mirrors William's live correlator (#24317) at his explicitly-tightened tolerance for the historical backfill.
Result on 300-row prod sample	—	21% supersede + 79% tombstone. Of the 21%: every match has 0 ambiguous candidates in the window — i.e., the user had exactly one Swap in the 5 seconds before the fee. Lag distribution min=0s, median=0s, p95=1s, max=1s — same-block siblings.
Why isn't the supersede rate higher?	—	The 79% no-match isn't a correlation bug. RPC-verified spot-check: even with the broadest possible search (any `type`, any `supersededBy` state, full user history, Solana RPC slot resolution), no row exists in `cash_activity` at the fee's exact slot for ~58% of the misses. Likely gasless fees for non-Swap operations, or pipeline gaps where the corresponding swap row was never written. These tombstone correctly; user-visible result identical.
Why not wider than 5s?	—	Sensitivity sweep on the same 300-row sample: 5s = 21% unambiguous / 0% multi-candidate, 10s = 25% / 1%, 30s = 31% / 15%, 60s = 32% / 25%, 180s = 34% / 34%, 600s = 30% / 45%. Widening past 5s only adds heuristic guesses (user had ≥2 in-flight swaps and we'd be picking one arbitrarily); the unambiguous share doesn't grow. Matches William's review: "the lookback might be too liberal if it's very long."

Drill-down: one real prod row

Fee row · id d3249908… · type transfer · state completed · occurredAt 2026-05-27T18:32:02Z
counterparty { type: phantom_username, username: babykiss22, address: 9yj3zvLS3fDMqi1F… } · amount 0.241009 cash · supersededBy null

Parent Swap · id e00d021d… · type swap · state completed · occurredAt 2026-05-27T18:32:01.633Z · amount 0.284476 cash · supersededBy null

Backfill decision tree: counterparty address matches babykiss22 → take the SUPERSEDE path → find parent Swap within 15min lookback → ✓ found e00d021d… (lag <1s) → write:
  UPDATE cash_activity SET supersededBy='e00d021d…' WHERE id='d3249908…' AND supersededBy IS NULL
  INSERT INTO cash_activity_history (activityId, type, data) VALUES ('d3249908…', 'superseded', {...})
  publish envelope id=e00d021d… with supersededIds=[d3249908…]

Of the 7 / 30 (24%) sampled rows that did not correlate within the 15min lookback, all looked like: either the per-user activity feed didn't surface the parent in the most-recent-20 (pagination), or the parent settled outside the lookback window. Both correctly fall through to the tombstone fallback path — feed-visible result is identical, just without the supersededBy audit link.

Summary

Tests passing

Spec files

Retire paths

Lint / TS errors

Adds POST /cash/v1/admin/backfill/hidden-counterparty-tombstones (admin role, 202) scoped by userIds and/or an occurredAt window. A BullMQ coordinator pages cash_activity via Kysely keyset pagination and emits one batch job per page. Each batch worker rechecks every row's counterparty.address against the current dynamic + baseline hidden set and resolves it via one of two paths:

Babykiss22 (gasless fee collector) — supersede by parent Swap: find the user's most recent non-superseded Swap within 15min before the fee row's occurredAt, write supersededBy = swap.id, append a cash_activity_history audit row, and publish the swap's envelope with the fee row id in supersededIds.

Any other hidden address (e.g. Bridge contract baseline) — publish a deleted: true tombstone, since there is no parent activity to attribute the row to.

This mirrors the live-ingest mechanism William deployed (eager stub #24244 / correlator backstop #24317) for historical rows that pre-date those changes. The read-time filter #24373 already hides them from the Cash service's own API; this brings the downstream Activity Service feed in sync — properly attributing fees to their parent swap rather than just deleting them.

Architecture

┌──────────────────────────────────────────────────────────────────────────┐ │ POST /cash/v1/admin/backfill/hidden-counterparty-tombstones │ │ body: { userIds?, startDate?, endDate? } (≥1 of userIds/startDate) │ └─────────────────────────────────┬────────────────────────────────────────┘ │ HiddenCounterpartyBackfillRequestDto │ · cross-field @Validate scope guard ▼ ┌────────────────────────────────┐ │ AdminService │ │ · BadRequest on bad scope │ │ · sha256(scope) → jobId │ └──────────────┬─────────────────┘ │ enqueueHiddenCounterpartyTombstoneBackfill ▼ ┌────────────────────────────────────────────────────────┐ │ {hidden-counterparty-backfill-coordinator} (BullMQ) │ │ concurrency = 1 │ │ │ │ snapshot hidden set := baseline ∪ dynamic │ │ ├─ if empty → return early (no scan, no batches) │ │ └─ else keyset-paginate cash_activity by id │ │ limit 500 per page │ │ checkpoint afterId via job.updateData │ └──────────────┬─────────────────────────────────────────┘ │ one batch per page ▼ ┌──────────────────────────────────────────────────────────────────┐ │ {hidden-counterparty-backfill-batch} (BullMQ) │ │ concurrency = envInt(HIDDEN_COUNTERPARTY_BACKFILL_CONCURRENCY) │ │ │ │ for each activity in batch: │ │ · counterparty.address ∈ hidden snapshot? else skip │ │ · occurredAt < cutoff? skip (parity with read filter) │ │ · supersededBy already set? skip │ │ │ │ if address == FIREBLOCKS_FEE_PAYER_ADDRESS (babykiss22): │ │ parent := findRecentSwapForGaslessFee( │ │ userId, feeOccurredAt, 15min) │ │ if parent && setSupersededByIfNull(feeRow, parent): │ │ · recordSupersededHistory(feeRow, parent) │ │ · publish(swap envelope, supersededIds=[feeRow.id]) │ │ else if parent: (race — live ingest beat us) │ │ · no publish, counted as superseded │ │ else: fall through to tombstone │ │ │ │ else (Bridge contract, etc.): │ │ publish( deleted: true envelope, id = activity.id ) │ └──────────────┬───────────────────────────────────────────────────┘ │ Kafka — partition key = envelope.id ▼ ┌───────────────────────────┐ │ Activity Service feed │ │ dedups by envelope.id │ └───────────────────────────┘

Every checkpoint is idempotent: the coordinator's jobId is a sha256 of the scope; the batch worker's setSupersededByIfNull only fires on rows currently NULL; Activity Service dedups by envelope id. Re-running the same scope is a safe no-op (or near no-op).

Wallet feed — before / after

Mocked rendering of the user's activity feed for a representative gasless-swap session. The fee tx arrives ~30s after the parent swap as a separately-broadcast Solana tx; SimpleHash ingests it as an orphan transfer with babykiss22 as the counterparty.

Before backfill orphan fee visible

Cash activity read filter #24373 not yet in your client

Swap · USDC → SOL

2026-04-12 14:02 · gasless · 1.2 SOL out

−$210.00

babykiss22

2026-04-12 14:02 · solana · transfer

−$0.05

Two rows for one real action: the user sees a "mystery" $0.05 transfer to a name they don't recognise.

After backfill superseded by parent swap

Cash activity supersededBy set + feed envelope republished

Swap · USDC → SOL

2026-04-12 14:02 · gasless · supersededIds: [feeRow]

−$210.00

The fee row is retained in the DB (audit, history row written) but the feed shows only the canonical Swap.

Non-babykiss22 hidden address — tombstone path

For the Bridge contract address (baseline) or any other dynamic-list entry that has no notion of a "parent activity", the backfill publishes a deleted: true envelope. The Activity Service drops the row by envelope id.

Before visible

Cash activity—

Bridge contract

2026-03-08 09:14 · solana · transfer

−$0.01

After tombstoned

Cash activitydeleted envelope published

Scenario coverage matrix

Every retire-decision is anchored to a William-Lee scenario. The parity.spec.ts suite mirrors the read-time filter logic and asserts the backfill retires the exact same row set; the batch-worker spec drills into each retire path.

Scenario	William's PR	Backfill action	Test	Status
babykiss22 fee row (gasless fee collector) with correlatable parent Swap (≤15min lookback)	#24244 eager stub / #24317 correlator	set `supersededBy = swap.id`; publish swap envelope with fee row in `supersededIds`	batch.spec — happy path	PASS
babykiss22 fee row — history audit row appended	#24244 (cash_activity_history)	Kysely insert into `cash_activity_history` with type `superseded`	batch.spec — history audit	PASS
babykiss22 fee row — no parent Swap in lookback window	n/a (fallback)	warn + tombstone (deleted: true)	batch.spec — no-parent fallback	PASS
babykiss22 fee row — live-ingest already set supersededBy	#24244 race scenario	respect existing value; no publish (live path already did)	batch.spec — race with live ingest	PASS
Row already `supersededBy != null`	n/a	skip entirely (no double-handling)	batch.spec — skip already-superseded	PASS
history-write failure (Kysely insert blip)	n/a	non-fatal warn; supersession still publishes	batch.spec — history non-fatal	PASS
PhantomUsername-typed counterparty (e.g. babykiss22 resolved via BNS enrichment)	#24373 (widened filter)	match by address regardless of `counterparty.type`	parity.spec — PhantomUsername	PASS
Crypto-typed counterparty pointing at hidden address	#24370	match by address	parity.spec — Crypto	PASS
Bridge contract baseline (no admin-list entry)	baseline in config	tombstone	parity.spec — Bridge baseline	PASS
Bank / Merchant counterparty (no `address` field)	#24373	skip (no false-positive on the predicate)	parity.spec + batch.spec	PASS
EVM address case normalization (lowercase 0x…)	#24179 (normalize on storage)	match case-insensitively via `normalizeHiddenCounterpartyAddress`	batch.spec — EVM normalize	PASS
Activity pre-dates `hiddenCounterpartyFilterStartDate` cutoff	#24373 cutoff guard	skip (parity with read filter — don't retire grandfathered rows)	batch.spec — cutoff guard	PASS
Activity type not on the feed (e.g. `card_pin_updated`)	n/a — never published	count as matched but skip publish (nothing to tombstone)	batch.spec — unmapped type	PASS
Kafka publish failure	n/a	propagate so BullMQ retries the batch	batch.spec — publish error	PASS

Test inventory

Spec file	Tests	What it covers
`hidden-counterparty-backfill-batch.worker.spec.ts`	12	retire path (supersede vs tombstone), history audit, race, cutoff guard, skip-already-superseded, EVM normalize, publish error
`hidden-counterparty-backfill-coordinator.worker.spec.ts`	8	baseline always present, keyset pagination, checkpoint resume, date-range scope, malformed date rejection, empty snapshot short-circuit, propagation on scan failure
`hidden-counterparty-backfill.parity.spec.ts`	7	backfill ↔ read-time filter parity for William's scenarios
`admin.service.hidden-counterparty-backfill.spec.ts`	8	scope validation, BadRequest on empty body, date validation, queue wiring
`queue.service.hidden-counterparty.spec.ts`	6	sha256 jobId collision-safety, queue stats inclusion
`hidden-counterparty-backfill.dto.spec.ts`	6	cross-field scope guard fires on undefined anchor; userIds/startDate/endDate validation
`event-consumer.module.spec.ts`	8	consumer-types include the two new queues; CONSUMER_TYPES=invalid lists them

Risk & mitigation

Risk	Mitigation
Backfill clobbers `supersededBy` already written by live ingest	`UPDATE … WHERE supersededBy IS NULL` — race-safe by SQL; returns false if it didn't actually update, worker treats that as "live path handled it" and skips re-publish.
Wrong parent Swap gets paired to a historical fee row	Lookback bounded to `[feeOccurredAt − 15min, feeOccurredAt]` — never future-dated. Mirror's William's correlator semantics, widened modestly from 3min to absorb clock drift. Failure mode is identical to William's: "some Swap of the user's, both are valid".
Duplicate enqueue (admin clicks twice)	Coordinator `jobId` = `sha256(sortedUserIds + startDate + endDate).slice(0,16)`. Identical scope → identical jobId → BullMQ no-op.
Coordinator crashes mid-scan	`job.updateData(afterId, scanned, nextBatchIndex)` after each page. Retry resumes from checkpoint, doesn't re-emit batches.
Empty hidden snapshot triggers a full scan for nothing	Coordinator short-circuits before `scanPage` with a warn log when the merged baseline+dynamic snapshot is empty.
History write fails after supersededBy write succeeds	Non-fatal: warn-and-continue. Supersession is still in the DB; we just miss an audit row. Better than rolling back a successful supersede.
Activity Service double-counts on re-publish	Dedups by envelope id; supersede paths re-publish the parent Swap's envelope (idempotent upsert) with updated `supersededIds`.
Tombstones a row for a hidden type that was never published	`buildCashActivityEnvelope` returns `null` for unmapped types — worker skips publish for those (counted as matched, zero tombstones).

How this lines up with William's prod write-path

	Live ingest (William, deployed via #24477)	This PR (historical backfill)
Trigger	Cash submits gasless bundle / SimpleHash event arrives	Admin `POST /admin/backfill/hidden-counterparty-tombstones`
Detect babykiss22 fee row	Eager stub at bundle-submit time, or correlator if stub missed (#24317)	Scan finds `counterparty.address == FIREBLOCKS_FEE_PAYER_ADDRESS`
Find parent Swap	`findRecentSwapByUser(userId, now − 3min)`	`findRecentSwapForGaslessFee({ userId, feeOccurredAt, lookbackMs: 15min })`
Set supersededBy	`_atomicSupersedeAndUpsert` (MikroORM transaction)	`setSupersededByIfNull` (Kysely, gated to `NULL`)
History audit	`cashActivityHistoryRepository.recordSuperseded`	`scan.recordSupersededHistory` (Kysely, same payload shape)
Feed publish	Parent Swap envelope with `supersededIds` from `publishActivityFeedUpdate`	Parent Swap envelope with `supersededIds=[feeRow.id]` via the same producer

Operations

Run a backfill

POST /cash/v1/admin/backfill/hidden-counterparty-tombstones
Authorization: Bearer <admin token>
Content-Type: application/json

# Option A — by user ids (e.g. complaint received from a user)
{ "userIds": ["3fa85f64-5717-4562-b3fc-2c963f66afa6"] }

# Option B — by occurredAt window (broad cleanup)
{ "startDate": "2026-01-01T00:00:00Z",
  "endDate":   "2026-04-15T23:59:59Z" }

# Both at once narrows further.
# At least one of userIds (non-empty) or startDate is required (DTO-validated).

Response: 202 Accepted with the BullMQ jobId. Progress streams to logs and queue stats at GET /cash/v1/admin/bullmq-cache/queues.