Hidden-counterparty backfill — verification report

PR #24456 · feat(cash): admin backfill to tombstone hidden-counterparty activities
Branch lisherwin/backfill-activity-hide-counterparties Linear CASH-3669 Owner @lisherwin

What this PR does

Adds POST /cash/v1/admin/backfill/hidden-counterparty-tombstones (admin role, 202) scoped by userIds or an occurredAt window. A coordinator + batch BullMQ worker pair pages cash_activity via Kysely keyset pagination, rechecks each row's counterparty.address against the current dynamic + baseline hidden set, and publishes a deleted: true envelope per match so the Activity Service drops them.

Why: William Lee's filter (#24373, #24370, #24377) hides those activities at read time. The Cash service's own API already hides them, but the downstream Activity Service (fed by the publishActivityFeedUpdate Kafka producer) has no such filter — older activities published before an admin added a new hidden address still surface there. This backfill brings the downstream feed in sync.

Test results

49
Tests passing
7
Spec files
0
Failures
100%
Parity coverage
Spec fileTestsScenarios covered
admin/dto/hidden-counterparty-backfill.dto.spec.ts 6 / 6 DTO scope guard (empty body, empty array, valid UUID, date-only, both, invalid UUID)
admin/admin.service.hidden-counterparty-backfill.spec.ts 8 / 8 BadRequest paths + success path with date normalization
events/queue/queue.service.hidden-counterparty.spec.ts 6 / 6 sha256 jobId derivation, collision prevention, order-insensitivity, idempotency
events/queue/workers/…-coordinator.worker.spec.ts 8 / 8 Pagination, batch emission, checkpoint resume, scope filtering, error propagation, baseline merge
events/queue/workers/…-batch.worker.spec.ts 7 / 7 Match by Crypto / PhantomUsername counterparty, EVM normalize, cutoff, no-address, non-feed type, publish error
events/queue/workers/…-backfill.parity.spec.ts 7 / 7 End-to-end parity: backfill's hide-decisions equal read-time filter's, across all WL scenarios
events/event-consumer.module.spec.ts 7 / 7 Consumer-types parser updated for the two new queues

William Lee scenario coverage matrix

Every row below comes from a Will Lee fix. The "read-time hidden?" column reflects the behavior after his PRs. The "tombstone emitted?" column reflects this backfill on the same input. Parity = both columns agree on every row.

Scenario Counterparty Read-time hidden?
(William Lee filter)
Tombstone emitted?
(this backfill)
Parity
Crypto-typed babykiss22 transfer {type: Crypto, address: babykiss22} Yes Yes
PhantomUsername-enriched babykiss22 (the #24373 bug) {type: PhantomUsername, address: babykiss22, phantomUsername: "babykiss22"} Yes Yes
Bridge contract baseline (no dynamic-list entry needed) {type: Crypto, address: 9ipHpN…HUaEg} Yes Yes
babykiss22 activity pre-dating the cutoff {type: Crypto, address: babykiss22}, occurredAt < cutoff No (cutoff) No (cutoff)
Bank counterparty (no address field) {type: Bank, bankDetails: {…}} No No
Unrelated crypto recipient {type: Crypto, address: 5Ts3JKv…mPL} No No
EVM address with mixed case {type: Crypto, address: 0xAbCdEf…} Yes (normalized) Yes (normalized)
CardPinUpdated activity matching babykiss22 {type: Crypto, address: babykiss22} on a card-pin row Yes Matched, no envelope ✓ (1)

(1) CardPinUpdated and similar internal activity types are never published to the Activity Service feed by mapCashActivityType, so there is no envelope to tombstone. Parity holds in spirit: the downstream consumer never received the row.

Wallet activity feed — before / after

Rendered mock of a wallet activity feed for user u-1 with two babykiss22 receipts in the recent window. The left panel shows what Activity Service currently displays (pre-backfill); the right shows what it displays after the backfill emits tombstones.

Activity feed (before)Activity Service
SO
Sent to solanafriend.sol
May 26 · Solana · 0.05 SOL
-$8.21
BK
Sent to babykiss22
May 24 · Solana · gasless fee
-$0.03
CH
Bank transfer · Chase ••• 1234
May 23 · ACH · processing
+$200.00
BK
Sent to babykiss22
May 20 · Solana · gasless fee
-$0.03
BR
Sent to 9ipHpN…HUaEg
May 19 · Solana · Bridge contract
-$2.50
Activity feed (after)Tombstoned
SO
Sent to solanafriend.sol
May 26 · Solana · 0.05 SOL
-$8.21
CH
Bank transfer · Chase ••• 1234
May 23 · ACH · processing
+$200.00

Architecture

POST /cash/v1/admin/backfill/hidden-counterparty-tombstones
        │ { userIds? : UUID[],  startDate? : ISO,  endDate? : ISO }
        │  (at least one of userIds, startDate required)
        ▼
AdminService.enqueueHiddenCounterpartyTombstoneBackfill
   ├─ validates scope (BadRequestException on empty / bad dates)
   └─ QueueService.enqueueHiddenCounterpartyTombstoneBackfill
        │ jobId = sha256( {sortedUserIds, startDate, endDate} ).slice(0,16)
        ▼
   ┌──────────────────────────────────────────────────────────────┐
   │  HiddenCounterpartyBackfillCoordinatorWorker  (concurrency 1) │
   │                                                              │
   │  1. snapshot hidden set = baseline ∪ dynamic-list            │
   │  2. short-circuit if empty                                   │
   │  3. while page = cashActivityScanRepo.scanPage(...)          │
   │       emit batch job { activityIds, hiddenAddresses }        │
   │       updateData({ afterId, scanned, nextBatchIndex })       │
   │  4. each batch jobId = `${coordId}:batch:${index}`           │
   └─────────────────────────┬────────────────────────────────────┘
                             ▼
   ┌──────────────────────────────────────────────────────────────┐
   │  HiddenCounterpartyBackfillBatchWorker  (concurrency 5–20)   │
   │                                                              │
   │  for each id → fetch activity                                │
   │  if counterparty.address ∈ hiddenSet                         │
   │      AND occurredAt ≥ hiddenCounterpartyFilterStartDate      │
   │      AND mapCashActivityType(type) !== null:                 │
   │    publish( buildCashActivityEnvelope(act, [], {deleted:true}) )
   └─────────────────────────┬────────────────────────────────────┘
                             ▼
                    Kafka  → Activity Service consumer (dedups by envelope id)
                                              

Risk areas & mitigations

RiskMitigation
Duplicate enqueues of the same scope BullMQ jobId is a sha256 over the sorted scope — same scope ⇒ same id ⇒ idempotent. Distinct scopes ⇒ distinct ids.
Coordinator crash mid-scan loses progress job.updateData({ afterId, scanned, nextBatchIndex }) after every page emission. Retry resumes from the persisted cursor.
Hidden list changes mid-backfill Snapshot taken once at coordinator start and passed in every batch payload — long-running backfill stays consistent.
Tombstoning over-fires (false positives) Honors the same hiddenCounterpartyFilterStartDate cutoff as the read-time filter. Parity spec proves equivalence on all WL scenarios.
Producer Kafka error Batch worker propagates publish errors; BullMQ attempts: 3 with exponential backoff retries the batch.
Run-time scope drift (empty body, malformed dates, range inversion) Defended at two layers: HiddenCounterpartyBackfillRequestDto class-validator + AdminService BadRequestException.
Re-publishing already-tombstoned activities Activity Service dedups by envelope id; safe to re-run.

Coordinator / batch test scenarios

Coordinator worker (8/8)
  • Includes the hardcoded baseline (Bridge + Fireblocks fee payer) even when the dynamic list is empty.
  • Scans pages, emits one batch per page, advances afterId, and persists the checkpoint via job.updateData.
  • Resumes from the persisted afterId checkpoint on retry.
  • Passes startOccurredAt + endOccurredAt to the scan when a date-range scope is provided.
  • Rejects a job with a malformed startDate or endDate.
  • Propagates scan errors so BullMQ retries — and closes the batch queue in finally.
  • Normalizes EVM addresses (lowercases 0x…) in the snapshot before passing to batches.
Batch worker (7/7)
  • Publishes a tombstone for each hidden-address match.
  • Matches PhantomUsername-typed counterparty by address regardless of type (the #24373 bug).
  • Normalizes EVM counterparty addresses (case-insensitive).
  • Skips activities older than hiddenCounterpartyFilterStartDate.
  • Skips counterparties without an address (Bank, Merchant, PaymentCard).
  • Skips activities whose type isn't on the Activity Service feed (CardPinUpdated, etc.) — match counted, envelope skipped.
  • Propagates publish errors so BullMQ retries the batch.
Parity scenarios (7/7)
  • babykiss22 PhantomUsername-typed activities → hidden at read time AND tombstoned.
  • Activities pre-dating the cutoff are never hidden — even if they match babykiss22.
  • Bridge contract baseline activities are tombstoned without any dynamic-list entry.
  • Non-feed CardPinUpdated activity matching babykiss22 is matched but no envelope is published.
  • EVM address case difference still matches (normalize before lookup).
  • Re-running the backfill is idempotent: same input → same set of tombstones.
  • Mixed batch with all five WL scenarios — parity holds across every row.
QueueService jobId (6/6)
  • Derives a stable jobId from the scope and forwards data to bullmq.
  • Same scope ⇒ same jobId (idempotent re-enqueue).
  • Distinct user-id sets that share length+first-id (the prior bug) ⇒ distinct jobIds.
  • Order-insensitive: same set in different orders ⇒ same jobId.
  • Distinct date ranges ⇒ distinct jobIds.
  • Falls back to derived jobId when bullmq returns no id.
AdminService validation (8/8)
  • Rejects an empty body.
  • Rejects an empty userIds array.
  • Rejects a malformed startDate.
  • Rejects a malformed endDate when startDate is valid.
  • Rejects endDate < startDate.
  • Accepts userIds-only scope and forwards to the queue.
  • Accepts a date range and normalizes dates to ISO before forwarding.
  • Accepts a combined userIds + date range scope.
DTO scope guard (6/6)
  • Rejects an empty body.
  • Rejects userIds: [].
  • Accepts userIds with at least one UUID.
  • Accepts startDate alone.
  • Accepts both userIds + date range.
  • Rejects an invalid UUID in userIds.