feat(cash): admin backfill to tombstone hidden-counterparty activities
Adds POST /cash/v1/admin/backfill/hidden-counterparty-tombstones (admin role, 202)
scoped by userIds or an occurredAt window. A coordinator + batch BullMQ
worker pair pages cash_activity via Kysely keyset pagination, rechecks each row's
counterparty.address against the current dynamic + baseline hidden set, and publishes
a deleted: true envelope per match so the Activity Service drops them.
Why: William Lee's filter
(#24373,
#24370,
#24377)
hides those activities at read time. The Cash service's own API already hides them, but
the downstream Activity Service (fed by the publishActivityFeedUpdate Kafka
producer) has no such filter — older activities published before an admin added a new hidden
address still surface there. This backfill brings the downstream feed in sync.
| Spec file | Tests | Scenarios covered |
|---|---|---|
admin/dto/hidden-counterparty-backfill.dto.spec.ts |
6 / 6 | DTO scope guard (empty body, empty array, valid UUID, date-only, both, invalid UUID) |
admin/admin.service.hidden-counterparty-backfill.spec.ts |
8 / 8 | BadRequest paths + success path with date normalization |
events/queue/queue.service.hidden-counterparty.spec.ts |
6 / 6 | sha256 jobId derivation, collision prevention, order-insensitivity, idempotency |
events/queue/workers/…-coordinator.worker.spec.ts |
8 / 8 | Pagination, batch emission, checkpoint resume, scope filtering, error propagation, baseline merge |
events/queue/workers/…-batch.worker.spec.ts |
7 / 7 | Match by Crypto / PhantomUsername counterparty, EVM normalize, cutoff, no-address, non-feed type, publish error |
events/queue/workers/…-backfill.parity.spec.ts |
7 / 7 | End-to-end parity: backfill's hide-decisions equal read-time filter's, across all WL scenarios |
events/event-consumer.module.spec.ts |
7 / 7 | Consumer-types parser updated for the two new queues |
Every row below comes from a Will Lee fix. The "read-time hidden?" column reflects the behavior after his PRs. The "tombstone emitted?" column reflects this backfill on the same input. Parity = both columns agree on every row.
| Scenario | Counterparty | Read-time hidden? (William Lee filter) |
Tombstone emitted? (this backfill) |
Parity |
|---|---|---|---|---|
| Crypto-typed babykiss22 transfer | {type: Crypto, address: babykiss22} |
Yes | Yes | ✓ |
| PhantomUsername-enriched babykiss22 (the #24373 bug) | {type: PhantomUsername, address: babykiss22, phantomUsername: "babykiss22"} |
Yes | Yes | ✓ |
| Bridge contract baseline (no dynamic-list entry needed) | {type: Crypto, address: 9ipHpN…HUaEg} |
Yes | Yes | ✓ |
| babykiss22 activity pre-dating the cutoff | {type: Crypto, address: babykiss22}, occurredAt < cutoff |
No (cutoff) | No (cutoff) | ✓ |
Bank counterparty (no address field) |
{type: Bank, bankDetails: {…}} |
No | No | ✓ |
| Unrelated crypto recipient | {type: Crypto, address: 5Ts3JKv…mPL} |
No | No | ✓ |
| EVM address with mixed case | {type: Crypto, address: 0xAbCdEf…} |
Yes (normalized) | Yes (normalized) | ✓ |
| CardPinUpdated activity matching babykiss22 | {type: Crypto, address: babykiss22} on a card-pin row |
Yes | Matched, no envelope | ✓ (1) |
(1) CardPinUpdated and similar internal activity types are never published to the Activity
Service feed by mapCashActivityType, so there is no envelope to tombstone. Parity
holds in spirit: the downstream consumer never received the row.
Rendered mock of a wallet activity feed for user u-1 with two babykiss22 receipts
in the recent window. The left panel shows what Activity Service currently displays (pre-backfill);
the right shows what it displays after the backfill emits tombstones.
solanafriend.solbabykiss22babykiss229ipHpN…HUaEgsolanafriend.solPOST /cash/v1/admin/backfill/hidden-counterparty-tombstones
│ { userIds? : UUID[], startDate? : ISO, endDate? : ISO }
│ (at least one of userIds, startDate required)
▼
AdminService.enqueueHiddenCounterpartyTombstoneBackfill
├─ validates scope (BadRequestException on empty / bad dates)
└─ QueueService.enqueueHiddenCounterpartyTombstoneBackfill
│ jobId = sha256( {sortedUserIds, startDate, endDate} ).slice(0,16)
▼
┌──────────────────────────────────────────────────────────────┐
│ HiddenCounterpartyBackfillCoordinatorWorker (concurrency 1) │
│ │
│ 1. snapshot hidden set = baseline ∪ dynamic-list │
│ 2. short-circuit if empty │
│ 3. while page = cashActivityScanRepo.scanPage(...) │
│ emit batch job { activityIds, hiddenAddresses } │
│ updateData({ afterId, scanned, nextBatchIndex }) │
│ 4. each batch jobId = `${coordId}:batch:${index}` │
└─────────────────────────┬────────────────────────────────────┘
▼
┌──────────────────────────────────────────────────────────────┐
│ HiddenCounterpartyBackfillBatchWorker (concurrency 5–20) │
│ │
│ for each id → fetch activity │
│ if counterparty.address ∈ hiddenSet │
│ AND occurredAt ≥ hiddenCounterpartyFilterStartDate │
│ AND mapCashActivityType(type) !== null: │
│ publish( buildCashActivityEnvelope(act, [], {deleted:true}) )
└─────────────────────────┬────────────────────────────────────┘
▼
Kafka → Activity Service consumer (dedups by envelope id)
| Risk | Mitigation |
|---|---|
| Duplicate enqueues of the same scope | BullMQ jobId is a sha256 over the sorted scope — same scope ⇒ same id ⇒ idempotent. Distinct scopes ⇒ distinct ids. |
| Coordinator crash mid-scan loses progress | job.updateData({ afterId, scanned, nextBatchIndex }) after every page emission. Retry resumes from the persisted cursor. |
| Hidden list changes mid-backfill | Snapshot taken once at coordinator start and passed in every batch payload — long-running backfill stays consistent. |
| Tombstoning over-fires (false positives) | Honors the same hiddenCounterpartyFilterStartDate cutoff as the read-time filter. Parity spec proves equivalence on all WL scenarios. |
| Producer Kafka error | Batch worker propagates publish errors; BullMQ attempts: 3 with exponential backoff retries the batch. |
| Run-time scope drift (empty body, malformed dates, range inversion) | Defended at two layers: HiddenCounterpartyBackfillRequestDto class-validator + AdminService BadRequestException. |
| Re-publishing already-tombstoned activities | Activity Service dedups by envelope id; safe to re-run. |
afterId, and persists the checkpoint via job.updateData.afterId checkpoint on retry.startOccurredAt + endOccurredAt to the scan when a date-range scope is provided.startDate or endDate.finally.0x…) in the snapshot before passing to batches.PhantomUsername-typed counterparty by address regardless of type (the #24373 bug).hiddenCounterpartyFilterStartDate.address (Bank, Merchant, PaymentCard).CardPinUpdated, etc.) — match counted, envelope skipped.CardPinUpdated activity matching babykiss22 is matched but no envelope is published.id.userIds array.startDate.endDate when startDate is valid.endDate < startDate.userIds: [].startDate alone.userIds.