sxchnge/backend/notifications/README.md

Notifications backend

Independent internal gRPC service with durable asynchronous email delivery. The public contract belongs to interface/proto/v1/interface.proto. Generated clients are build artifacts, not committed source.

Architecture

The gRPC transport maps protobuf DTOs to domain requests. The service validates and renders immutable template revisions, then saves a notification and its pending-work state atomically in one document through the shared DB wrapper. The outbox dispatcher publishes opaque IDs using shared messaging's durable transport, adapted from Sendico. A durable NATS worker claims the notification with an atomic conditional patch and saves the outcome before acknowledging. DriverConfig selects the email driver: sendgrid, resend or smtp.

Email adapters only map rendered email models to provider wire payloads. SendGrid uses adapted Sendico SDK construction; Resend uses its email HTTP API. Both support subject, text and HTML, bounded timeouts and safe error classification.

The owned SMTP adapter uses authenticated implicit TLS (normally port 465), verified server certificates and context-bounded connections. It sends MIME multipart/alternative with quoted-printable UTF-8 text/HTML and encoded subjects. It uses Go's standard SMTP client and does not import Sendico modules or share a mutable connection between workers. Configure it with:

email:
  driver: smtp
  settings:
    host: mail.example.com
    port: 465
    username: notifications@example.com
    password_env: SMTP_PASSWORD
    sender_address: notifications@example.com
    sender_name: SXCHNGE
    network_timeout: 10

network_timeout must be 1–20 seconds; the worker's expiry/shutdown context also bounds every operation. Only the password belongs in Vault. There is no plaintext fallback or certificate-verification bypass.

Run

Requires Go 1.27, protoc, an SXCHNGE MongoDB database and a NATS server with JetStream enabled. Copy config.example.yml to an untracked config.yml here. Set SXCHNGE_MONGO_URI, SXCHNGE_NATS_URL and the selected email provider's API key environment variable. Configure a verified sender address and template catalog. Paths are relative to the process working directory.

From the root, run make run-notifications. The gRPC listener uses plaintext on the private local network. Configure both tls_cert and tls_key only when TLS is explicitly required. Use a private authenticated NATS endpoint; tls:// URLs and credentials in the URL are supported by the shared driver. Mongo and messaging both use shared DriverConfig settings; email settings reject unknown keys.

The queue uses SXCHNGE_NOTIFICATIONS, subject sxchnge.notifications.email, and durable consumer email. Configure queue.stream.replicas as 1, 3 or 5 to match the NATS cluster, and queue.stream.max_age_hours for retention (48 hours minimum). File storage and the owned stream/subject names are enforced. queue.provisioning.environment accepts production or non-production; mode: ops validates existing resources without changing them, mode: app provisions missing resources, and mode: hybrid provisions only outside production. Production retention and consumer-setting drift fail startup instead of being silently accepted. Provisioning operations honor their startup context. The example defaults to one replica and app provisioning for local development. Multiple service workers can safely compete for jobs through atomic claims.

Templates and parameters

See templates.example.json, including the auth.activation and auth.login English/Russian templates used by product authentication. Parameters use Go template map syntax such as {{.first_name}}. Each revision declares the exact allowed parameter names and locale-specific content. Missing/extra parameters, unknown revisions/locales and invalid content are rejected before acceptance. Empty locale uses the default. Values are data, not executable templates; HTML is escaped by html/template. The catalog is loaded and compiled at startup. Never edit a published revision; add a new version. Rendered payloads are persisted so retries do not depend on a later catalog change. Recipient, body and subject are cleared on terminal state; remaining status and request fingerprints expire through a seven-day TTL.

Delivery semantics

• Send persists before returning; it does not wait for NATS or the email provider.
• Outbox scans every five seconds and republishes safely after a bus failure. Pending notifications expire and have their rendered content cleared even when the bus is unavailable or a retry is scheduled later.
• Duplicate RPCs and duplicate broker deliveries do not create extra jobs/sends.
• Provider HTTP 2xx becomes ACCEPTED. No receipt/webhook or read tracking is claimed.
• Explicit rate limiting (429) retries with bounded exponential backoff and Retry-After, at most five provider attempts, bounded by message expiry.
• Transport errors, 408, 5xx and expired worker leases become UNKNOWN. These may have sent the email; operators must reconcile before requesting another send.
• Other provider failures are terminal. provider_authentication_failed signals credentials/permissions, provider_rejected a nonretryable rejection, retry_exhausted the attempt limit, and expired the message deadline.
• Resend receives the notification ID as its provider idempotency key. Its window is 24 hours. SendGrid has no equivalent guarantee; exactly-once external email delivery is not promised.
• SMTP acceptance requires the final DATA 250 reply. A later QUIT failure cannot undo acceptance. Explicit SMTP 4xx replies become retryable provider_temporarily_unavailable; 5xx replies are terminal (authentication failures retain provider_authentication_failed). Transport/TLS errors and a lost final DATA reply become UNKNOWN. No immediate reconnect/resend occurs. A stable Message-ID is included for correlation, not an exactly-once promise.
• Changing the configured driver fails queued jobs pinned to the old driver with driver_changed. Drain queued jobs before changing sender credentials/config.
• Callers authorize recipients and preferences. No direct customer-database access, bulk campaigns, provider failover, SMS, push or Telegram delivery is included.

Structured logs report notification IDs, driver, state and attempt count. Provider HTTP outcomes include provider_status; failures also include provider_error and provider_message when available. The adapter reads at most 16 KiB plus an overflow byte and extracts only the Resend name/message or first SendGrid errors[].message. Diagnostic messages are bounded to 512 Unicode characters plus an ellipsis. Credentials, idempotency keys, outgoing request string values, email addresses, URLs, long token-like strings and numeric verification/account values embedded in the request are redacted; controls and formatting characters are normalized. Malformed, incomplete and oversized error responses produce a diagnostic fallback instead of raw content. Successful response bodies are not logged. Diagnostic fields are transient: they are neither persisted in notifications nor exposed in gRPC responses. The existing bounded failure codes and delivery/retry decisions remain unchanged. Transport failures include a sanitized cause. Provider outcome logging happens before the result write so a storage failure cannot discard the only provider diagnostic.

Message content, template values, addresses and raw provider bodies must remain excluded from logs. Provider text is sanitized external diagnostic data, never an instruction. Use provider_status, provider_error and provider_message together: the generic provider_authentication_failed category combines HTTP 401 and 403, including sender/domain authorization failures. A sending-only Resend key is valid for sending, but cannot read the provider's log API. SMTP outcomes use provider_smtp_status for protocol reply codes and the same sanitized provider error/message fields, including password/auth-payload redaction. Monitor pending age, UNKNOWN/FAILED states and repeated outbox/worker warnings.

Validate and distribute

make test and make lint-backend generate interfaces and validate all modules. Notifications tests exercise real disposable MongoDB and NATS containers, concurrent idempotency and claims, lease recovery, retries, transport authentication, rendering and both provider payloads through local HTTP servers. An authenticated TLS gRPC Send/Get test covers the full shared-database → NATS → rendered-email driver path. No live emails are sent. make package-notifications-interface prepares source and Go-client archives. See provenance and contract usage.

Provider references: SendGrid Mail Send, Resend Send Email.