feat(cash): admin backfill to tombstone hidden-counterparty activities
Hit the live admin API with a real admin token to verify the backfill's decision tree against actual
cash_activity rows. All calls were read-only (GET); no enqueue, no mutation.
| Probe | Endpoint | Result |
|---|---|---|
| Auth check | GET /cash/v1/admin/checkAuth |
HTTP 200 |
| Most-recent transfers sampled | GET /cash/v1/admin/cash-activities?type=transfer&limit=100 × 10 pages |
1000 transfers scanned |
| babykiss22-counterparty rows in that sample | client-side filter on counterparty.address == FIREBLOCKS_FEE_PAYER_ADDRESS |
394 / 1000 (39%) |
| Already-superseded (William's live ingest worked) | supersededBy != null |
1 / 394 |
| Un-superseded (this PR's target rows) | supersededBy == null |
393 / 394 |
counterparty.type distribution |
— | 100% phantom_username (BNS enrichment hit on every row — exactly the case Will's #24373 widened the filter for) |
| State distribution | — | 100% completed |
| Correlation strategy | 5-second time window per William's guidance |
For each babykiss22 fee row: look up the user's most recent non-superseded Swap with
occurredAt in [fee.occurredAt − 5s, fee.occurredAt]. Mirrors William's live
correlator (#24317) at his explicitly-tightened tolerance for the historical backfill.
|
| Result on 300-row prod sample | — | 21% supersede + 79% tombstone. Of the 21%: every match has 0 ambiguous candidates in the window — i.e., the user had exactly one Swap in the 5 seconds before the fee. Lag distribution min=0s, median=0s, p95=1s, max=1s — same-block siblings. |
| Why isn't the supersede rate higher? | — |
The 79% no-match isn't a correlation bug. RPC-verified spot-check: even with the broadest
possible search (any type, any supersededBy state, full user
history, Solana RPC slot resolution), no row exists in cash_activity at the
fee's exact slot for ~58% of the misses. Likely gasless fees for non-Swap operations, or
pipeline gaps where the corresponding swap row was never written. These tombstone correctly;
user-visible result identical.
|
| Why not wider than 5s? | — | Sensitivity sweep on the same 300-row sample: 5s = 21% unambiguous / 0% multi-candidate, 10s = 25% / 1%, 30s = 31% / 15%, 60s = 32% / 25%, 180s = 34% / 34%, 600s = 30% / 45%. Widening past 5s only adds heuristic guesses (user had ≥2 in-flight swaps and we'd be picking one arbitrarily); the unambiguous share doesn't grow. Matches William's review: "the lookback might be too liberal if it's very long." |
d3249908… · type transfer · state completed · occurredAt 2026-05-27T18:32:02Z{ type: phantom_username, username: babykiss22, address: 9yj3zvLS3fDMqi1F… } · amount 0.241009 cash · supersededBy nulle00d021d… · type swap · state completed · occurredAt 2026-05-27T18:32:01.633Z · amount 0.284476 cash · supersededBy nulle00d021d… (lag <1s) → write:UPDATE cash_activity SET supersededBy='e00d021d…' WHERE id='d3249908…' AND supersededBy IS NULLINSERT INTO cash_activity_history (activityId, type, data) VALUES ('d3249908…', 'superseded', {...})publish envelope id=e00d021d… with supersededIds=[d3249908…]
Of the 7 / 30 (24%) sampled rows that did not correlate within the 15min lookback, all looked like: either the per-user activity feed didn't surface the parent in the most-recent-20 (pagination), or the parent settled outside the lookback window. Both correctly fall through to the tombstone fallback path — feed-visible result is identical, just without the supersededBy audit link.
Adds POST /cash/v1/admin/backfill/hidden-counterparty-tombstones (admin role, 202) scoped by
userIds and/or an occurredAt window. A BullMQ coordinator pages
cash_activity via Kysely keyset pagination and emits one batch job per page. Each batch
worker rechecks every row's counterparty.address against the current dynamic + baseline
hidden set and resolves it via one of two paths:
Swap: find the user's most
recent non-superseded Swap within 15min before the fee row's occurredAt,
write supersededBy = swap.id, append a cash_activity_history audit row, and
publish the swap's envelope with the fee row id in supersededIds.deleted: true
tombstone, since there is no parent activity to attribute the row to.
This mirrors the live-ingest mechanism William deployed (eager stub #24244 / correlator backstop #24317) for historical rows that pre-date those changes. The read-time filter #24373 already hides them from the Cash service's own API; this brings the downstream Activity Service feed in sync — properly attributing fees to their parent swap rather than just deleting them.
Every checkpoint is idempotent: the coordinator's jobId is a sha256 of the scope; the batch
worker's setSupersededByIfNull only fires on rows currently NULL; Activity
Service dedups by envelope id. Re-running the same scope is a safe no-op (or near no-op).
Mocked rendering of the user's activity feed for a representative gasless-swap session. The fee tx arrives ~30s after the parent swap as a separately-broadcast Solana tx; SimpleHash ingests it as an orphan transfer with babykiss22 as the counterparty.
#24373 not yet in your client
Two rows for one real action: the user sees a "mystery" $0.05 transfer to a name they don't recognise.
The fee row is retained in the DB (audit, history row written) but the feed shows only the canonical Swap.
For the Bridge contract address (baseline) or any other dynamic-list entry that has no notion of a
"parent activity", the backfill publishes a deleted: true envelope. The Activity Service
drops the row by envelope id.
Every retire-decision is anchored to a William-Lee scenario. The parity.spec.ts suite
mirrors the read-time filter logic and asserts the backfill retires the exact same row set; the
batch-worker spec drills into each retire path.
| Scenario | William's PR | Backfill action | Test | Status |
|---|---|---|---|---|
| babykiss22 fee row (gasless fee collector) with correlatable parent Swap (≤15min lookback) | #24244 eager stub / #24317 correlator | set supersededBy = swap.id; publish swap envelope with fee row in supersededIds |
batch.spec — happy path | PASS |
| babykiss22 fee row — history audit row appended | #24244 (cash_activity_history) | Kysely insert into cash_activity_history with type superseded |
batch.spec — history audit | PASS |
| babykiss22 fee row — no parent Swap in lookback window | n/a (fallback) | warn + tombstone (deleted: true) | batch.spec — no-parent fallback | PASS |
| babykiss22 fee row — live-ingest already set supersededBy | #24244 race scenario | respect existing value; no publish (live path already did) | batch.spec — race with live ingest | PASS |
Row already supersededBy != null |
n/a | skip entirely (no double-handling) | batch.spec — skip already-superseded | PASS |
| history-write failure (Kysely insert blip) | n/a | non-fatal warn; supersession still publishes | batch.spec — history non-fatal | PASS |
| PhantomUsername-typed counterparty (e.g. babykiss22 resolved via BNS enrichment) | #24373 (widened filter) | match by address regardless of counterparty.type |
parity.spec — PhantomUsername | PASS |
| Crypto-typed counterparty pointing at hidden address | #24370 | match by address | parity.spec — Crypto | PASS |
| Bridge contract baseline (no admin-list entry) | baseline in config | tombstone | parity.spec — Bridge baseline | PASS |
Bank / Merchant counterparty (no address field) |
#24373 | skip (no false-positive on the predicate) | parity.spec + batch.spec | PASS |
| EVM address case normalization (lowercase 0x…) | #24179 (normalize on storage) | match case-insensitively via normalizeHiddenCounterpartyAddress |
batch.spec — EVM normalize | PASS |
Activity pre-dates hiddenCounterpartyFilterStartDate cutoff |
#24373 cutoff guard | skip (parity with read filter — don't retire grandfathered rows) | batch.spec — cutoff guard | PASS |
Activity type not on the feed (e.g. card_pin_updated) |
n/a — never published | count as matched but skip publish (nothing to tombstone) | batch.spec — unmapped type | PASS |
| Kafka publish failure | n/a | propagate so BullMQ retries the batch | batch.spec — publish error | PASS |
| Spec file | Tests | What it covers |
|---|---|---|
hidden-counterparty-backfill-batch.worker.spec.ts |
12 | retire path (supersede vs tombstone), history audit, race, cutoff guard, skip-already-superseded, EVM normalize, publish error |
hidden-counterparty-backfill-coordinator.worker.spec.ts |
8 | baseline always present, keyset pagination, checkpoint resume, date-range scope, malformed date rejection, empty snapshot short-circuit, propagation on scan failure |
hidden-counterparty-backfill.parity.spec.ts |
7 | backfill ↔ read-time filter parity for William's scenarios |
admin.service.hidden-counterparty-backfill.spec.ts |
8 | scope validation, BadRequest on empty body, date validation, queue wiring |
queue.service.hidden-counterparty.spec.ts |
6 | sha256 jobId collision-safety, queue stats inclusion |
hidden-counterparty-backfill.dto.spec.ts |
6 | cross-field scope guard fires on undefined anchor; userIds/startDate/endDate validation |
event-consumer.module.spec.ts |
8 | consumer-types include the two new queues; CONSUMER_TYPES=invalid lists them |
| Risk | Mitigation |
|---|---|
Backfill clobbers supersededBy already written by live ingest |
UPDATE … WHERE supersededBy IS NULL — race-safe by SQL; returns false if it didn't actually update, worker treats that as "live path handled it" and skips re-publish. |
| Wrong parent Swap gets paired to a historical fee row | Lookback bounded to [feeOccurredAt − 15min, feeOccurredAt] — never future-dated. Mirror's William's correlator semantics, widened modestly from 3min to absorb clock drift. Failure mode is identical to William's: "some Swap of the user's, both are valid". |
| Duplicate enqueue (admin clicks twice) | Coordinator jobId = sha256(sortedUserIds + startDate + endDate).slice(0,16). Identical scope → identical jobId → BullMQ no-op. |
| Coordinator crashes mid-scan | job.updateData(afterId, scanned, nextBatchIndex) after each page. Retry resumes from checkpoint, doesn't re-emit batches. |
| Empty hidden snapshot triggers a full scan for nothing | Coordinator short-circuits before scanPage with a warn log when the merged baseline+dynamic snapshot is empty. |
| History write fails after supersededBy write succeeds | Non-fatal: warn-and-continue. Supersession is still in the DB; we just miss an audit row. Better than rolling back a successful supersede. |
| Activity Service double-counts on re-publish | Dedups by envelope id; supersede paths re-publish the parent Swap's envelope (idempotent upsert) with updated supersededIds. |
| Tombstones a row for a hidden type that was never published | buildCashActivityEnvelope returns null for unmapped types — worker skips publish for those (counted as matched, zero tombstones). |
| Live ingest (William, deployed via #24477) | This PR (historical backfill) | |
|---|---|---|
| Trigger | Cash submits gasless bundle / SimpleHash event arrives | Admin POST /admin/backfill/hidden-counterparty-tombstones |
| Detect babykiss22 fee row | Eager stub at bundle-submit time, or correlator if stub missed (#24317) | Scan finds counterparty.address == FIREBLOCKS_FEE_PAYER_ADDRESS |
| Find parent Swap | findRecentSwapByUser(userId, now − 3min) |
findRecentSwapForGaslessFee({ userId, feeOccurredAt, lookbackMs: 15min }) |
| Set supersededBy | _atomicSupersedeAndUpsert (MikroORM transaction) |
setSupersededByIfNull (Kysely, gated to NULL) |
| History audit | cashActivityHistoryRepository.recordSuperseded |
scan.recordSupersededHistory (Kysely, same payload shape) |
| Feed publish | Parent Swap envelope with supersededIds from publishActivityFeedUpdate |
Parent Swap envelope with supersededIds=[feeRow.id] via the same producer |
POST /cash/v1/admin/backfill/hidden-counterparty-tombstones
Authorization: Bearer <admin token>
Content-Type: application/json
# Option A — by user ids (e.g. complaint received from a user)
{ "userIds": ["3fa85f64-5717-4562-b3fc-2c963f66afa6"] }
# Option B — by occurredAt window (broad cleanup)
{ "startDate": "2026-01-01T00:00:00Z",
"endDate": "2026-04-15T23:59:59Z" }
# Both at once narrows further.
# At least one of userIds (non-empty) or startDate is required (DTO-validated).
Response: 202 Accepted with the BullMQ jobId. Progress streams to logs and queue stats at GET /cash/v1/admin/bullmq-cache/queues.