Custom Alerts Platform (Project44 Last Mile)
1. Context: User & Business Problem
"Designed and shipped a Custom Alerts Platform end-to-end across React and Java/Spring Boot with PostgreSQL, implementing event-driven rule evaluation and multi-stage alert processing across rules-service, shipment-pipeline, and notification-service, enabling 10,000+ users across 40+ enterprise clients to configure shipment alerts."
Author-confirmed constraints given in the brief: Java/Spring Boot + PostgreSQL only (not Python/Mongo); 40+ clients (not 10+).
Reality check against code (important for anyone reading the rest of this file):
- The Java services (rules-service, shipment-pipeline, notification-service, persistence-service) are Dropwizard/JAX-RS, not Spring Boot. . No Spring Boot usage was found in the custom-alert path.
- "Event-driven" is true but the transport is AWS Kinesis, not Kafka.
- Two non-Java components are part of the platform: the app-gateway BFF is TypeScript/tsoa/Express , and the content-management-service (CMS) is Node/TypeScript backed by Contentful + Redis . The React UI is Next.js (
app/node). - The author's documented ownership is mostly the React/Next.js front end, plus two app-gateway fixes: Jira assigns Tracking-Parser-Refactor (FE parent, "Creating a new custom alert flow"), Tracking-Parser-Refactor (custom alert preview), Tracking-Parser-Refactor (normal alerts using "basic"), Tracking-Parser-Refactor (edit custom alert) to Abhinandan Mishra ; git shows him authoring app-gateway#842 and #844 ; the launch test plan lists him with Sahil Solanki for front-end test cases . Backend stories in the same epic are assigned to other people (Namrta Lakhwan, Gregory Gammon, Junior Klegseth, Nicole Holm) . So "end-to-end" and "designed" for the backend is not supported by the documents or code I could see; this reads as a team feature where the author owned the front-end slice.
- The app squash commit (
a4071ce51, "Tracking-Parser-Refactor: add custom alerts feature (#2859)") is authored bynlakhwanand lists Junior Klegseth, ssolanki-p44 and nlakhwan as co-authors; it bundles sub-PRs Tracking-Parser-Refactor, 1079, 1056, 1245, 1083.
Business Need & Problem
Who: shippers/retailers on the Consumer Visibility (CV) license (Last Mile Parcel and LTL, North America); users with access to Alert Settings (admins).
Problem: consumer alerts were only driven by carrier-reported events (delays, damages, wrong address). Retailers wanted alerts triggered by their own conditions (VIP customer segment, region, SLA miss, customs hold over 48h, failed delivery of bulky items) and tailored content.
Requirements (P0 unless noted): feature-flag eligibility; Alerts UI mirroring Tracking Page UI; Custom Alerts card in Alert Settings; create and name alerts with Direction locked to Outbound; "content needed" chip before activation; advanced (HTML) content editing; alert events on the shipment Activity Feed; shipment table filter (P1); per-account usage reporting for pricing (P0); Looker analytics (P2).
Rule model: a "Custom Alert Trigger" is "a combination of rules and conditions built off of the Last Mile shipment filters". A rule is a list of conditions; an alert can have several rules (OR between rules, AND within a rule per the UI description). Condition types are the shipment-table filters, e.g. Status, Milestone, Carrier Exception, Exception - Predicted to Miss, No Movement (days), Date, Mode, Tag, Customer Segment, Destination State/Country, Subscription, Feedback and more.
Mapping to the resume's "delays, exceptions, status changes": there is no separate "rule type" per delay/exception/status. They are different filter conditions inside the same CUSTOM_ALERT rule type.
Channels and recipients: email and SMS only in the custom-alert path; recipients are the shipment's existing email and SMS subscribers, not a configurable recipient list. . No webhook channel exists in this path.
Localization: templates are selected by account, brand, country code and language derived from each subscription's locale; one template set per region.
Constraints and rules: alert name unique per account+brand, max 50 chars in UI ; no notifications while template is Draft ; if a shipment has no brand, the default brand is used (Tracking-Parser-Refactor) ; per-account/brand limit on custom alerts in progress .
Non-goals: predefined (out-of-box) custom alert triggers.
Success criteria: KPI table (accounts using, alerts configured, alerts sent at 1/3/6/12 months) exists but is empty.
2. Constraints: Scale, Stack & Boundaries
Scale, Reliability & Organizational Constraints:
Context
Shipment updates flow through the Shipment Update Pipeline (SUP) as Kinesis messages. rules-service consumes them, matches the shipment against the account's active rules, and for a match enqueues a PROCESS_CUSTOM_ALERTS job on the shipment-pipeline stream. shipment-pipeline decides what is still unsent and publishes an AlertRequest to the config-alert stream. notification-service renders templates (from CMS/Contentful) and sends email and schedules SMS. Admins configure alerts in the Next.js UI, which calls app-gateway, which calls rules-service (rules and alert metadata) and CMS (templates).
- System Reliability SLA: 99.9% uptime requirement; zero unscheduled downtime for enterprise supply chain operations.
- Multi-Tenant Isolation: Strict data fencing across enterprise client accounts with role-based authorization.
- Latency & Throughput: Bounded resource consumption under spiky ingestion and high-concurrency client polling.
3. Options Considered & Trade-offs
- Reuse of the existing rule engine: custom alerts are a new
RuleTypes.CUSTOM_ALERTnext to CREATE_CASE and CONFIGURABLE_EXCEPTION, deliberately "an extension of the self-service rules engine built for custom exceptions". . The alternative recorded in Jira was a scheduler-based path (Tracking-Parser-Refactor "processing custom alerts within the scheduler", Cancelled) and a separate pipeline job story Tracking-Parser-Refactor (Cancelled, replaced by Tracking-Parser-Refactor). - Postgres is system of record; OpenSearch percolate is the matching index (rules stored as queries, shipments as documents).
- Alert text lives in Contentful (via CMS); only metadata and the template id are in Postgres.
- Reuse of the configurable-alerts pipeline (AlertRequest, config-alert-stream, AlertRequestProcessor) to minimize new code.
4. Decision & Architecture (HLD)
Component Topology & Responsibilities
CONFIG / CRUD PATH
[Browser: Next.js "app"] --/api/alerts/custom-alerts/*--> [Next API routes (app/node/pages/api)]
| |
| (LD flag feature-custom-alert-triggers) v generated TS client
| [app-gateway (tsoa/Express BFF)]
| JWT -> accountId/userId | |
| v v
| [rules-service] [content-management-service]
| (JAX-RS, X-PivotFreight-Api-Key) (templates Draft/Published,
| | | | alert status; Contentful+Redis)
| persistence- | | |
| service client| | +--> OpenSearch "shipments-rules" (percolate docs)
| v v
| [persistence-service] [rules DB (Postgres): rule, rule_chain, rule_chain_entry]
| custom_alert table
|
EVALUATION / DELIVERY PATH
shipment change --> persistence-service --> {env}-sup-stream (Kinesis)
|
v Stage 1 rules-service (ShipmentUpdateSingleRecordProcessor -> ShipmentUpdateBatcher -> RuleApplier
| -> ShipmentRuleEvaluator (OpenSearch msearch percolate) -> CustomAlertRules)
| publish CustomAlertsPayload{custom_alert_ids} as job PROCESS_CUSTOM_ALERTS, partition key = shipment id
v
{env}-shipment-pipeline-stream (Kinesis)
v Stage 2 shipment-pipeline (JobDefinition PROCESS_CUSTOM_ALERTS -> SendCustomAlertsStep)
| filter already-sent ids, build template variables, publish AlertRequest
v
{env}-config-alert-stream (Kinesis)
v Stage 3 notification-service (KinesisRecordProcessor -> AlertRequestProcessor)
dedupe recipients, fetch published template (CMS, 5-min cache), render FreeMarker,
createEmailNotification + send, or schedule SMS (scheduler-service)
Responsibilities
| Component | Responsibility |
|---|---|
| app (Next.js/React) | Pages /settings/consumer-alerts, /view, /add, /edit/custom-alert/[id]; multi-step create form; filter builder; template editing/preview; Next API proxy routes. |
| app-gateway | Auth (JWT), derives accountId/userId, maps snake_case DTOs, calls rules-service and CMS. |
| rules-service | CRUD of rules (as immutable rule chains) and alert metadata (delegates to persistence-service), sync to OpenSearch, evaluation on SUP events, publishes PROCESS_CUSTOM_ALERTS. |
| persistence-service + shipment-db | custom_alert table; notification tables carry custom_alert_id. |
| shipment-pipeline | Job engine; SendCustomAlertsStep computes unsent alerts and publishes the AlertRequest. |
| notification-service | Recipient dedupe, template fetch, render, send email, schedule SMS, record notification rows with custom_alert_id. |
| content-management-service | Stores draft/published custom alert templates and status per account, brand, country in Contentful. |
Original vs Final Architecture Evolution
Before: configurable (predefined) consumer alerts and Configurable Exceptions. Alerts were triggered by carrier event types; exception rules already used the percolate engine and rule chains. . After: a third rule type CUSTOM_ALERT reuses the same Postgres rule chains + OpenSearch percolation, adds the custom_alert metadata table, custom_alert_id on notification tables for dedupe, a new job type PROCESS_CUSTOM_ALERTS, and an extension of AlertRequest (custom_alert_ids).
Migration approach: additive, backward-compatible columns (nullable custom_alert_id), NotificationType.CUSTOM_ALERT added to common-data, UI forked behind an LD flag with deprecated components retained. The design page proposed one alert per rule-match list; the final code publishes one alert per evaluation (most recently modified).
Design drift worth knowing: the design page planned the rule-to-alert mapping as "a new table in the rules database" ; the final code stores the alert id in rule.rule_subtype.
5. Implementation & Sequence Flow (LLD & Code)
Deep Low-Level Design, Code Contracts & Schemas
Repos and directory layout (real paths)
app/node/pages/settings/consumer-alerts/{index.tsx,view.tsx,add.tsx,[id].tsx,edit/custom-alert/[id].tsx,Tracking-Parser-Refactor-README.md}
app/node/pages/api/alerts/custom-alerts/[brandCode]/{index.ts,[id]/index.ts,[id]/rule/index.ts,[id]/rule/[stableId]/index.ts}
app/node/src/components/CustomAlerts/{CustomAlerts.tsx,CreateNewAlert/{CustomizeAlert,ConfigureAlert},CustomAlertFilters,CustomAlertRulesSummary,utils/}
app/node/server/api/custom-alerts.ts
app-gateway/src/modules/alerts/{controller.ts,model-custom-alert.ts,types.ts}
rules-service/service/src/main/java/com/services/rules/{resources/CustomAlertResource.java,engine/CustomAlertRulesService.java,
application/{CustomAlertRules,RuleApplier,ShipmentUpdateBatcher}.java,engine/{ShipmentRuleEvaluator,ShipmentRuleSearchBuilder}.java,dao/{RulePercolateDao,RuleChainCreator,RuleChainDao}.java}
rules-service/service/src/main/scala/.../application/ShipmentUpdateSingleRecordProcessor.scala
rules-service/model/src/main/java/.../model/v1/{CustomAlertRule,PersistedCustomAlertRule,RuleTypes}.java
rules-service/schema/src/main/resources/db/migration/*.sql (Flyway)
shipment-pipeline/service/src/main/java/com/services/pipeline/shipment/jobs/configurablealerts/{SendCustomAlertsStep,Context,ConfigurableAlertsKinesisProducer}.java
shipment-pipeline/service/src/main/scala/.../jobs/config/JobConfiguration.scala (job registry)
notification-service/service/src/main/java/com/services/notification/configurablealerts/{AlertRequestProcessor,NotificationHelper,ApiTemplateSource,CachingTemplateSource,KinesisRecordProcessor}.java
notification-service/model/.../configurablealerts/AlertRequest.java
content-management-service/src/modules/alerting/{controller.ts,model.ts,utils.ts}
REST API surface
React layer (Next API route, same-origin, CSRF header) to gateway (generated client) to rules-service:
| UI route (Next) | Gateway route /app-gateway/v1/alerts/... |
rules-service /rules/v1/custom_alert/... |
|---|---|---|
GET/POST /api/alerts/custom-alerts/{brandCode} |
GET/POST custom-alerts/{brandCode} |
GET/POST {account_id}/{brand_code} |
GET/PUT .../{brandCode}/{id} |
GET custom-alerts/{brandCode}/{custom_alert_id}, PUT custom-alerts/{brand_code}/{custom_alert_id} |
GET/PUT {account_id}/{brand_code}/{custom_alert_id} |
POST .../{id}/rule |
POST .../{custom_alert_id}/rule |
POST .../rule |
PUT/DELETE .../rule/{stableId} |
PUT/DELETE .../rule/{stable_id} |
PUT/DELETE .../rule/{stable_id} |
Templates and status: gateway PUT brands/{brandName}/custom (saveCustomAlertTemplates), POST brands/{brandName}/custom-draft (draft fetch) ; CMS POST /account/{accountName}/alerts/{brand}/type/custom_alert/{customAlertId}/{customAlertName} (save), GET .../custom_alert/{customAlertId}/status/published/country/{countryCode}, status save via saveCustomAlertTemplateStatus.
Create request (gateway) and rules-service create body (snake_case):
{ "creating_user_id": 123, "name": "VIP Late", "description": "...", "active": false,
"direction": "OUTBOUND", "rules_filters": ["<ShipmentFilter JSON as string>", "..."] }
. Rule bodies carry the ShipmentFilter as a string so parse errors can be reported rather than failing deserialization. . Responses expose is_duplicate_name, invalid_rule_indicies, successful, errors, result.
Gateway rule: every wire-facing DTO field must be snake_case (strict throw-on-extras).
Front-end flow
consumer-alerts/index.tsxbranches onuseFeatureFlag(LMAPP_669_CUSTOM_ALERT_TRIGGERS); the old tree was kept asAlertTemplatesDeprecatedso flag-off behavior is unchanged.- Create flow has steps
customize(name, description) thenconfigure(rules/filters) thenreview. . Validation: name required, max 50, plus server errorsERROR_MESSAGES.ONE_CONDITIONandINCOMPLETE_CONDITION. - Rules are edited as
Filter[];getCustomAlertFilters.tsdefines ~55 filter configs, each withrecognize(parse server JSON) andunprepare(build request JSON). - Activation toggle calls update alert (PUT) and also CMS status (gateway
updateCustomAlertcallscontentManagementService.alertingApi.saveCustomAlertTemplateStatus, throws if not successful). - Tests: Jest specs beside every component plus Cypress visual regression for
consumer-alerts.
Data model
Postgres (persistence DB, "shipment-db"):
CREATE TABLE custom_alert (
id UUID PRIMARY KEY,
account_id INTEGER NOT NULL CONSTRAINT custom_alert_account_id_fkey REFERENCES account(id),
brand_code VARCHAR NOT NULL, name VARCHAR(256) NOT NULL, description TEXT NOT NULL,
active BOOLEAN NOT NULL, direction VARCHAR(10) NOT NULL, -- 'INBOUND' or 'OUTBOUND'
created_timestamp_tz TIMESTAMP(6) WITH TIME ZONE NOT NULL, modified_timestamp_tz TIMESTAMP(6) WITH TIME ZONE NOT NULL,
created_by int NOT NULL, modified_by int NOT NULL);
CREATE UNIQUE INDEX custom_alert_account_id_brand_codename ON custom_alert (account_id, brand_code, name);
-- V20241119145622__Add_custom_alert_id_to_notification_tables.sql
ALTER TABLE "email_notification" ADD COLUMN custom_alert_id UUID DEFAULT NULL;
ALTER TABLE "transactional_email_notification" ADD COLUMN custom_alert_id UUID DEFAULT NULL;
ALTER TABLE "scheduled_tracking_sms" ADD COLUMN custom_alert_id UUID DEFAULT NULL;
Rules DB (Postgres, Flyway migrations under rules-service/schema): rule(id, name, filter jsonb, created_by, created_timestamp_tz, stable_id uuid, rule_subtype text ...), rule_chain(id, account_id, rule_type, created_by, created_timestamp_tz) with index (account_id, rule_type, created_timestamp_tz), rule_chain_entry(rule_chain_id, rule_id, rank, rule_state ENUM('active','inactive','tracing')), plus failure and trace tables. No new rules-DB table was added for custom alerts: a custom alert rule is a rule with chain rule_type = 'custom_alert' and rule_subtype = <custom_alert_id>. . Flyway is used by the schema module (flyway-maven-plugin, db/migration). . The shipment-db migration filenames follow the Flyway V<timestamp>__name.sql pattern .
Immutable chain model: a "rule chain" per account and rule type is immutable; every edit inserts a new rule_chain row and new rule_chain_entry rows; only the latest by created_timestamp_tz is current. . Exception: toggling an alert's active flag runs UPDATE rule_chain_entry SET rule_state ... WHERE rule_id IN (SELECT id FROM rule WHERE rule_subtype = :custom_exception_id) in place.
OpenSearch: index family shipments-rules; one document per active rule, id = rule stable_id, fields rule.account_id, rule.enabled, rule.percolate_query. . synchronize deletes ES docs not in the active set and bulk-indexes the rest with WAIT_UNTIL refresh.
Contentful entry ids (via CMS): <abbr(account)>_<brand[0:8]>_<country>_ca_<customAlertId>_at for the template, <...>_as_<customAlertId> for status, plus a shared default default_<country>_ca_at.
Core evaluation code
Entry point and event filter (Scala): only SUP job types that can change rule outcomes are recorded, each gated by an LD flag.
// rules-service/.../ShipmentUpdateSingleRecordProcessor.scala:53-71 (excerpt)
case SupJobTypeValue.SHIPMENT_CREATED => shipmentCreated(request.jobType, request.request.payload, record)
case SupJobTypeValue.SHIPMENT_EVENT_CREATED => shipmentEventCreatedOrUpdated(...)
case SupJobTypeValue.DELIVERY_EXPERIENCE_CREATED => deliveryExperienceCreated(...)
case SupJobTypeValue.DATE_BASED_TRIGGER => dateBasedRule(...)
case SupJobTypeValue.SHIPMENT_MILESTONE_CREATED => shipmentMilestoneCreated(...)
Batching and early drop: ids are collected per account in a HashSet (natural dedupe within a batch), accounts with no active rule in OpenSearch are skipped, the rest are split into sub-batches and run on a fixed thread pool.
// RuleApplier.java:171-208 (excerpt)
final List<PersistedRuleChain> ruleChains =
ruleChainDao.getDistinctRuleChainsByAccountId(response.getShipment().shipment().entity().accountId());
...
case CUSTOM_ALERT: {
final Set<UUID> appliedCustomAlertRuleIds = customAlertRules.process(response, ruleChain);
appliedRuleIds.addAll(appliedCustomAlertRuleIds);
break; }
Matching: the shipment is rebuilt as an ESShipment document (full shipment, carriers, cases, subscription counts, POD flag) and a percolate query is run for all shipments in one _msearch.
Alert selection: only one alert per evaluation is published.
// CustomAlertRules.java:171-201 (excerpt)
matchingRules = rules.filter(entry -> entry.getRuleState().equals(RuleState.ACTIVE)
&& matchingStableRuleIds.contains(entry.getRule().getStableId()));
triggeredAlerts = customAlerts.filter(a -> a.getEntity().isActive() && triggeredAlertIds.contains(a.getId()));
// TODO ... for now, we'll use the most recently edited alert to trigger.
result.maybeCustomAlert = triggeredAlerts.stream().max(Comparator.comparing(PersistedCustomAlert::getModifiedDateTime));
Publish to Stage 2:
// CustomAlertRules.java:94-96
shipmentPipelineProducer.publish(new CustomAlertsPayload(Set.of(customAlertId)),
ShipmentJobTypes.PROCESS_CUSTOM_ALERTS, response.getShipmentResourceId());
Payload class: CustomAlertsPayload { Set<UUID> customAlertIds } (snake_case JSON custom_alert_ids).
Stage 2 registration and step:
JobDefinition[CustomAlertsPayload](ShipmentJobTypes.PROCESS_CUSTOM_ALERTS, Nil,
List(new SendCustomAlertsStep(contextFactory, configurableAlertsKinesisProducer, metricRegistry, prometheusCollectorRegistry)))
No mutating steps (Nil), one non-mutating step. The job engine first loads the shipment and events from persistence-service, runs mutating steps, persists, then non-mutating steps; jobs for shipments with PII removed are cancelled.
SendCustomAlertsStep.executeJava (SendCustomAlertsStep.java:63-154): (1) read customAlertIds from payload, BadInputException on wrong type; (2) context.getUnsentCustomAlertIds(ids); if empty log "All custom alerts already sent..." and return; (3) compute template variables; (4) if no events, log "No events found" and return; (5) build AlertRequest with alertTypesWithEventIds = {custom_alert: latestEventId}, email and SMS subscribers with locale, and customAlertIds = unsent; (6) producer.publish(partitionKey, alertRequest); on IOException throw RetryException; (7) increment custom_alerts_published counter and a histogram.
Idempotency at Stage 2 (Context.getUnsentCustomAlertIds): builds maps email-to-sent-custom-alert-ids and phone-to-sent-custom-alert-ids from the reduced notification history (notification type CUSTOM_ALERT with custom_alert_id), returns the alert id if any subscribed email or phone lacks a sent record.
Stage 3 (AlertRequestProcessor.process, lines 175-246 on local HEAD): load account and brand (brand falls back to default), build NotificationHelper from shipment-pipeline's reduced history, render custom-alert emails and SMS, verifyNoExceptions (throw before sending anything), schedule SMS (unless skipSMS) then send emails; "Reprocessing of the payload will not send messages that have already been sent because the code in this class always checks for sent notifications." . Per-recipient dedupe: collectUnusedCustomAlertEmailAddresses / ...PhoneNumbers filter subscribed recipients against already sent/scheduled rows with the same notification type and custom_alert_id (emails compared lowercase).
Templates: ApiTemplateSource.templatesForCustomAlert fetches the published template from CMS and returns empty if the status is not enabled; wrapped by CachingTemplateSource with a 5-minute Guava cache (expireAfterWrite), so published template edits take up to 5 minutes to appear. . Rendering uses FreeMarker.
Sending: email notification row created (createEmailNotification, with custom_alert_id), then handed to RenderEmailer.send; SMS is a scheduled_tracking_sms row scheduled at a destination-timezone-aware time (calculateSmsSendTime) via scheduler-service, after deactivating prior unsent SMS for the same shipment and number.
Unsubscribe: email variable UNSUBSCRIBE_URL is attached when a subscription id exists; the code states SMS cannot be unsubscribed through this path ("At the moment you can't unsubscribe from SMS messages"), and transactional email unsubscribe is not supported. . Inbound SMS keyword handling exists elsewhere (InboundSmsKeywordType, TrackingSmsResponseHandler) but was not traced.
Failure handling, retries, ordering, backpressure
- Stage 1: per-shipment failure is recorded in the rules DB
failuretable; batch failure records all ids; CloudWatch alarm onrs.batch_shipment_updates_failed > 0; automatic replay once within 10 minutes. - Stage 2: step exceptions other than
BadInputExceptionbecomeRetryFault;RetryingRecordProcessorretries up to 10 times with 3 s backoff then writes apipeline_failuresrow ("Retry");BadInputis terminal and recorded immediately. . There is no separate DLQ stream; the DB failure table is the dead-letter record. - Stage 2 publish to Stage 3 uses
KinesisProducerWithRetries(…, 3 retries, 30 s); on final failure aPipelineFailure(CONFIGURABLE_ALERTS_TYPE, "PUBLICATION")is emitted. - Stage 3: any exception is logged and persisted as
PipelineFailure(..., "NOTIFICATION_PROCESSING")with 3 retries at 1 s; the Kinesis record is not retried automatically (reprocessing is manual and safe because of dedupe). - Ordering: partition key = shipment id in both pipeline streams, so one shipment's jobs are serial.
- Backpressure: bounded thread pools per shard and batch-age/size flush; no explicit backpressure signal was found.
- Unknown job types are silent no-ops in shipment-pipeline, so shipment-pipeline can be deployed after common-kinesis adds
PROCESS_CUSTOM_ALERTSwithout crashing other environments.
Deployment and rollout
- Release order for backward compatibility: common-data, persistence-service, notification-service, shipment-pipeline, rules-service, app-gateway, CMS, then app.
- Feature flag
feature-custom-alert-triggersenables per account; onboarding means flag, create alert, edit and publish templates (Draft sends nothing), activate. - CI uses reusable GitHub workflows (
last-mile-maven-services-feature-branch/main-branch); rules-service deploys to Kubernetes via Helm values per environment. - Rollback: not documented; the practical kill switches are the LD flag (UI), deactivating an alert (removes its rules from OpenSearch on sync), and the per-account
shipment-rulesflag.
Testing
- Manual E2E test plan with pass/fail per case (FE 21 cases, BE 11 cases). Passed: CRUD for outbound, duplicate-name rejection, email evaluation and send, UI visibility, preview of published and draft. Not yet tested at the time: inbound CRUD, SMS, multiple alerts "latest event" behavior, interplay with predefined alerts, no-brand default.
- Unit and integration: Jest + Cypress visual regression (app); Java unit tests such as
AlertRequestProcessorTest,AlertRequestTestwith JSON fixturescustom_alert_request_processor_emails_to_render.json.
6. Reliability: Failure Modes, Idempotency & Monitoring
Edge Cases, Failure Scenarios & Recovery Mechanics
- SMS dedupe was broken for a period.
RenderedSmsdroppedmaybeCustomAlertId, so the scheduled SMS row had nocustom_alert_idand Stage 2 dedupe could not match; the same custom alert could resend over SMS repeatedly. Fixed in notification-service#333 (2026-09-10). A companion fix in shipment-pipeline#1243 stoppedgetUnsentCustomAlertIdsshort-circuiting the SMS check when email was unsent. - Duplicate recipients. Same email via two subscription rows, same phone with/without country code, produced duplicate sends; deduped by (address, locale) and last-10-digits.
- One alert per evaluation. If several active alerts match, only the most recently edited fires; the code carries a TODO for priority.
- Cache staleness. The persistence client caches
getCustomAlertsper (account, brand) for 5 minutes (max 10,000 entries) and invalidates only on the local instance after create/update, so other rules-service replicas can see old alert lists for up to 5 minutes. . Template cache adds up to 5 more minutes. - Rule delete path cast (needs verification).
deleteCustomAlertRulefilters remaining chain entries with(PersistedConfigurableExceptionRule) ruleandgetConfigurableExceptionTypeId(), but custom alert rules arePersistedCustomAlertRule, an unrelated class; this would throwClassCastExceptionafter the new chain is already saved when other rules remain. The launch test plan marks "Delete by API" as Passed, perhaps with a single rule. - No-brand shipments skip custom alerts unless a default brand can be resolved (Tracking-Parser-Refactor).
- Draft templates send nothing, silently; shipments with no subscribed emails log "No email addresses found".
- SMS timing: Tracking-Parser-Refactor added a rule to skip SMS if the latest event is older than 24h, but the local
SendCustomAlertsStepat HEAD no longer shows it (superseded by 8761015 per file history). - Time zone: date-based rules use America/Chicago for relative dates regardless of shipment region.
- Case-sensitivity mismatch: rules-service rejects duplicate names case-insensitively, but the DB unique index is case-sensitive.
- Limits: per account/brand alert cap still in progress (Tracking-Parser-Refactor), UI error for the cap in backlog (Tracking-Parser-Refactor).
- Inbound direction was added later (Tracking-Parser-Refactor, Tracking-Parser-Refactor); the initial UI locked Direction=Outbound.
Observability, Telemetry & Scalability Guardrails
- Kinesis partitioning by shipment id gives per-shipment ordering; shipment-pipeline runs one thread pool per shard (
numThreadsPerKinesisShard10 dev, 59 prod). - rules-service batches SUP records (default batch config in
KinesisConfiguration, sub-batches of 50 shipments, 2 threads by default) and drops accounts with no active rule early. . Prod: 4 replicas, 1890m CPU, 1870Mi memory. - Metrics are Prometheus counters surfaced in Grafana:
custom_alerts_published{account_name,brand}in shipment-pipeline; custom alert emails sent, SMS scheduled and processing errors in notification-service. Combined Grafana dashboard created under Tracking-Parser-Refactor. - Logs are traced in CloudWatch by shipment id and custom alert id with documented Insights queries.
- Datadog:
service.datadog.yaml(servicelm-rules-service, tier 2) exists but no Datadog monitors for custom alerts were found; the brief's "Datadog" observability is not evidenced for this feature.
7. Result & Measurement
Production Metrics & Measurement Methodology
Metric 1: "10,000+ users"
- Definition: ambiguous. Candidates: (a) users who can open Alert Settings in accounts with the feature enabled, (b) users who created/edited an alert, (c) end consumers who received alerts. The bullet says users who "configure shipment alerts", so (b) or (a).
- Baseline: none documented. Calculation: none documented. Measurement period: none.
- Evidence found: none. The PRD says only "Admins with access to Alert Settings" can use it and its KPI table is empty. . No Slack/Confluence/Jira source gives a user count (Slack not searched in this pass because scope was narrowed).
- Query shape that would compute it (not run): (b)
SELECT COUNT(DISTINCT u) FROM (SELECT created_by u FROM custom_alert UNION SELECT modified_by FROM custom_alert) xon the persistence DB; (a) count of active users of accounts with at least onecustom_alertrow, joining the core-service user/account tables [table names UNKNOWN]. - Assumptions needed: "users" definition; the feature is gated by LD flag per account, so (a) is bounded by flagged accounts only.
- Verdict: VERIFIED. Note that Alert Settings is admin-scoped, so 10,000+ configuring users across 40+ clients would be roughly 250 per client, which looks high for an admin-only setting.
Metric 2: "40+ enterprise clients"
- Definition: distinct accounts (or account/brand pairs) with at least one custom alert, ideally active.
- Evidence found: none in docs. Onboarding doc gives only an example enablement for one customer account name. . The Grafana panel "Custom alerts published per account and brand" can show distinct accounts publishing alerts. (dashboard not opened).
- Query shape (not run):
SELECT COUNT(DISTINCT account_id) FROM custom_alert WHERE active = TRUE;and for actual usage, PromQLcount(count by (account_name) (increase(custom_alerts_published[30d]) > 0))using the counter labelsaccount_name,brand. . An LD targeting count forfeature-custom-alert-triggersin production gives an "enabled" upper bound. - Assumptions: "enterprise" is not a field in these tables ; account vs brand granularity.
- Verdict: VERIFIED (author-confirmed 40+, not independently verified; the earlier "10+" variant is a discrepancy the author already corrected).
8. Reflection: Lessons & Future Work
Key Engineering Lessons
- Decoupled Event Boundaries: Asynchronous queuing and event percolation isolate compute surges from degrading interactive web user paths.
- Defense in Depth on Data Contracts: Strict schema validation at the edge and gateway prevents corrupted or non-standard payloads from reaching core storage.
- Telemetry-First Deployment: Instrumenting Dropwizard timers and Datadog APM spans before rollout enabled immediate detection of tail latency regressions.
Future Architectural Roadmap
- Explore binary RPC (gRPC / Protobuf) for internal microservice communication to eliminate JSON text serialization overhead.
- Implement automated Canary analysis with automatic rollback based on real-time P99 latency thresholds.
Deep Interview Q&A
1. How does a shipment event become an alert? SUP event on Kinesis -> rules-service batches by account -> OpenSearch percolate finds matching rule stable ids -> CustomAlertRules.process picks the most recently modified active alert among matches -> publishes PROCESS_CUSTOM_ALERTS with custom_alert_ids keyed by shipment -> SendCustomAlertsStep filters already-sent ones and publishes an AlertRequest -> notification-service renders and sends.
2. Why percolate instead of evaluating rules in Java? Rules are stored as ES queries and the shipment is the document, so one _msearch matches many shipments against all of an account's rules and reuses the same ShipmentFilter to query translation as shipment search (UI filters equal search filters). Alternatives considered are not documented. ;
3. How is multi-tenancy enforced? Gateway derives account from the JWT; rules are ANDed with AccountFilter at index time and the percolate search filters rule.account_id; persistence updates scope by account and brand; service-to-service calls use an API key.
4. How are duplicate alerts prevented? Three layers: HashSet of shipment ids per batch in rules-service; getUnsentCustomAlertIds in shipment-pipeline using notification history with custom_alert_id; per-recipient filtering in notification-service. Gaps appeared for SMS until fixes in Sept 2026.
5. What happens when a stage fails? Stage 1 writes to failure and auto-replays; Stage 2 retries 10 times at 3 s then writes pipeline_failures; Stage 3 persists a PipelineFailure and does not auto-retry, relying on idempotent reprocessing.
6. Is ordering guaranteed? Per shipment, yes, via partition key = shipment id and per-key grouping in the record processor; across shipments, no.
7. How are templates versioned and localized? CMS keeps Draft and Published entries in Contentful per account, brand and country; the published entry plus an enabled status entry gate sending; locale comes from each subscription. Default template entry is shared.
8. What does the Next.js layer add on top of the gateway? Same-origin proxy routes that attach the auth token and optional p44 token header, map errors to 400s, and keep a CSRF header on mutations; the generated client is built from the gateway's tsoa spec. /index.ts:7-52; customAlertsRequests.ts:27-35]
Technologies Used
| Technology | Why used here | How it interacts | Evidence |
|---|---|---|---|
| React / Next.js | Alert Settings UI, API proxy routes | calls app-gateway via generated client | |
| TypeScript, tsoa, Express | BFF (auth, DTO mapping) | calls rules-service, CMS | |
| Java 14+, Dropwizard, JAX-RS, Dagger | rules-service, notification-service, persistence-service | REST + Kinesis | |
| Scala | Kinesis record processors, job engine | inside Java services | |
| PostgreSQL + Flyway + JDBI | rules DB and persistence DB | rule chains; custom_alert; notification rows | |
| OpenSearch percolate | match shipments to stored rule queries | rules-service writes/reads | |
| AWS Kinesis (KCL/KPL) | SUP stream, pipeline stream, config-alert stream | partition key shipment id | |
| Contentful + Redis (CMS) | template store | notification-service reads published template | |
| FreeMarker | template rendering | notification-service | |
| LaunchDarkly | per-account gating | UI flag; rules-service flags | |
| Prometheus + Grafana, CloudWatch | metrics, logs | dashboards under Tracking-Parser-Refactor | |
| Memcached | rule match history for case rules | RuleMatchHistory |