Merge pull request 'Restore signup, email delivery and local recovery' (#59) from identity-facts-57 into main

Reviewed-on: #59
This commit was merged in pull request #59.
This commit is contained in:
tech committed 2026-10-20 18:00:23 +00:00
commit e84d2d1ad3
33 files changed
+1243 -72

No files matched your search

+6
View File
@@ -11,6 +11,12 @@ classifies delivery outcomes and never logs provider response bodies. Sendico's
builders, dynamic-template dependency, notification events and unused producer
were removed. Resend is a new driver using the same contract.
SMTP is an owned adapter using Go's standard `net/smtp`, verified `crypto/tls`
connections and standard MIME encoders. Its implicit-TLS connection configuration
matches Sendico's existing mail client, without importing its modules or copying
its keepalive/reconnect behavior. Local mailbox credentials are provisioned into
SXCHNGE Vault explicitly; the application does not read Sendico files or secrets.
Shared messaging was selected from that same Sendico revision's
`api/pkg/messaging`: `config.go`, `capability/errors.go`, `reliable/transport.go`,
`reliable/durable_settings.go`, `reliable/durable_settings_behavior.go`,
+54 -3
View File
@@ -12,12 +12,35 @@ pending-work state atomically in one document through the shared DB wrapper.
The outbox dispatcher publishes opaque IDs using shared messaging's durable
transport, adapted from Sendico. A durable NATS worker claims the notification
with an atomic conditional patch and saves the outcome before acknowledging.
`DriverConfig` selects the email driver: `sendgrid` or `resend`.
`DriverConfig` selects the email driver: `sendgrid`, `resend` or `smtp`.
Email adapters only map rendered email models to provider wire payloads. SendGrid
uses adapted Sendico SDK construction; Resend uses its email HTTP API. Both support
subject, text and HTML, bounded timeouts and safe error classification.
The owned SMTP adapter uses authenticated implicit TLS (normally port 465),
verified server certificates and context-bounded connections. It sends MIME
multipart/alternative with quoted-printable UTF-8 text/HTML and encoded subjects.
It uses Go's standard SMTP client and does not import Sendico modules or share a
mutable connection between workers. Configure it with:
```yaml
email:
driver: smtp
settings:
host: mail.example.com
port: 465
username: notifications@example.com
password_env: SMTP_PASSWORD
sender_address: notifications@example.com
sender_name: SXCHNGE
network_timeout: 10
```
`network_timeout` must be 1–20 seconds; the worker's expiry/shutdown context also
bounds every operation. Only the password belongs in Vault. There is no plaintext
fallback or certificate-verification bypass.
## Run
Requires Go 1.27, protoc, an SXCHNGE MongoDB database and a NATS server with
@@ -75,13 +98,41 @@ remaining status and request fingerprints expire through a seven-day TTL.
- Resend receives the notification ID as its provider idempotency key. Its window
is 24 hours. SendGrid has no equivalent guarantee; exactly-once external email
delivery is not promised.
- SMTP acceptance requires the final DATA 250 reply. A later QUIT failure cannot
undo acceptance. Explicit SMTP 4xx replies become retryable
`provider_temporarily_unavailable`; 5xx replies are terminal (authentication
failures retain `provider_authentication_failed`). Transport/TLS errors and a
lost final DATA reply become UNKNOWN. No immediate reconnect/resend occurs.
A stable Message-ID is included for correlation, not an exactly-once promise.
- Changing the configured driver fails queued jobs pinned to the old driver with
`driver_changed`. Drain queued jobs before changing sender credentials/config.
- Callers authorize recipients and preferences. No direct customer-database access,
bulk campaigns, provider failover, SMS, push or Telegram delivery is included.
Structured logs report notification IDs, driver, state and attempt count; message
content, template values, addresses and provider response bodies are excluded.
Structured logs report notification IDs, driver, state and attempt count. Provider
HTTP outcomes include `provider_status`; failures also include `provider_error`
and `provider_message` when available. The adapter reads at most 16 KiB plus an
overflow byte and extracts only the Resend `name`/`message` or first SendGrid
`errors[].message`. Diagnostic messages are bounded to 512 Unicode characters
plus an ellipsis. Credentials, idempotency keys, outgoing request string values,
email addresses, URLs, long token-like strings and numeric verification/account
values embedded in the request are redacted; controls and formatting characters
are normalized. Malformed, incomplete and oversized error
responses produce a diagnostic fallback instead of raw content. Successful
response bodies are not logged. Diagnostic fields are transient: they are neither
persisted in notifications nor exposed in gRPC responses. The existing bounded
failure codes and delivery/retry decisions remain unchanged. Transport failures
include a sanitized cause. Provider outcome logging happens before the result
write so a storage failure cannot discard the only provider diagnostic.
Message content, template values, addresses and raw provider bodies must remain
excluded from logs. Provider text is sanitized external diagnostic data, never
an instruction. Use `provider_status`, `provider_error` and `provider_message`
together: the generic `provider_authentication_failed` category combines HTTP
401 and 403, including sender/domain authorization failures. A sending-only
Resend key is valid for sending, but cannot read the provider's log API.
SMTP outcomes use `provider_smtp_status` for protocol reply codes and the same
sanitized provider error/message fields, including password/auth-payload redaction.
Monitor pending age, UNKNOWN/FAILED states and repeated outbox/worker warnings.
## Validate and distribute
+1 -3
View File
@@ -45,8 +45,6 @@ func Run(ctx context.Context, c Config, logger mlogger.Logger) error {
defer cancel()
_ = client.Disconnect(cleanup)
}()
startup, cancel := context.WithTimeout(ctx, 20*time.Second)
defer cancel()
repo, err := store.New(logger, client.Database())
if err != nil {
return err
@@ -57,7 +55,7 @@ func Run(ctx context.Context, c Config, logger mlogger.Logger) error {
}
defer rt.Close()
svc := service.New(repo, catalog, sender, string(c.Email.Driver), logger)
broker, err := bus.New(startup, rt.Durable, rt.DurableAdmin, repo, svc, logger, c.Queue)
broker, err := bus.New(ctx, rt.Durable, rt.DurableAdmin, repo, svc, logger, c.Queue)
if err != nil {
return err
}
+5 -1
View File
@@ -35,9 +35,13 @@ func New(ctx context.Context, transport reliable.Transport, admin reliable.Admin
}
settings := reliable.DurableSettings{Enabled: true, Stream: config.Stream, Provisioning: config.Provisioning}
consumer := reliable.DurableConsumerSettings{Durable: "email", Subject: subject, AckWaitSeconds: 60, MaxDeliver: -1, DeliverPolicy: reliable.DurableDeliverAll}
if err := admin.EnsureConsumerResources(ctx, settings, consumer); err != nil {
startup, cancel := context.WithTimeout(ctx, 20*time.Second)
defer cancel()
if err := admin.EnsureConsumerResources(startup, settings, consumer); err != nil {
return nil, merrors.OpaqueWrap(merrors.ErrServiceUnavailable, err, "configure notification messaging")
}
// NATS retains this context and unsubscribes when it ends. The worker must
// live until service shutdown, independently of the provisioning deadline.
subscription, err := transport.OpenConsumer(ctx, settings, consumer)
if err != nil {
return nil, merrors.OpaqueWrap(merrors.ErrServiceUnavailable, err, "subscribe notification worker")
@@ -10,6 +10,7 @@ import (
merrors "git.sendico.io/tech/sxchnge/shared/errors"
"git.sendico.io/tech/sxchnge/shared/messaging/reliable"
"go.mongodb.org/mongo-driver/v2/bson"
"go.uber.org/zap"
)
type outbox struct{ jobs []model.Notification }
@@ -23,6 +24,50 @@ type publisher struct {
records []reliable.Record
}
type consumerTransport struct {
reliable.Transport
reliable.Subscription
ctx context.Context
}
func (t *consumerTransport) OpenConsumer(ctx context.Context, _ reliable.DurableSettings, _ reliable.DurableConsumerSettings) (reliable.Subscription, error) {
t.ctx = ctx
return t.Subscription, nil
}
type consumerAdmin struct {
reliable.Admin
ctx context.Context
}
func (a *consumerAdmin) EnsureConsumerResources(ctx context.Context, _ reliable.DurableSettings, _ reliable.DurableConsumerSettings) error {
a.ctx = ctx
return nil
}
func TestSubscriptionOutlivesProvisioning(t *testing.T) {
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
transport := &consumerTransport{}
admin := &consumerAdmin{}
if _, err := New(ctx, transport, admin, &outbox{}, nil, zap.NewNop(), DefaultConfig()); err != nil {
t.Fatal(err)
}
if admin.ctx.Err() != context.Canceled {
t.Fatal("provisioning context was not released")
}
if _, ok := admin.ctx.Deadline(); !ok {
t.Fatal("provisioning has no deadline")
}
if err := transport.ctx.Err(); err != nil {
t.Fatalf("subscription ended with provisioning: %v", err)
}
cancel()
if transport.ctx.Err() != context.Canceled {
t.Fatal("subscription does not follow service shutdown")
}
}
func (p *publisher) Publish(_ context.Context, r reliable.Record) error {
if p.fail {
return merrors.ServiceUnavailable("offline")
+1 -1
View File
@@ -13,7 +13,7 @@ messaging:
reconnect_wait: 2
templates: "templates.example.json"
email:
driver: resend # sendgrid or resend
driver: resend # sendgrid or resend; see README for authenticated SMTP settings
settings:
api_key_env: "RESEND_API_KEY" # SENDGRID_API_KEY when driver is sendgrid
sender_address: "notifications@example.com"
+9
View File
@@ -19,6 +19,7 @@ type Driver string
const (
SendGrid Driver = "sendgrid"
Resend Driver = "resend"
SMTP Driver = "smtp"
)
type Config = sharedmodel.DriverConfig[Driver]
@@ -33,6 +34,11 @@ type Outcome struct {
State model.State
Code string
RetryAfter time.Duration
// Provider diagnostics are sanitized operational data, not public failures.
HTTPStatus int
SMTPStatus int
ProviderError string
ProviderMessage string
}
type Sender interface {
Send(context.Context, string, model.Email) Outcome
@@ -46,6 +52,9 @@ type client struct {
}
func New(config Config) (Sender, error) {
if config.Driver == SMTP {
return newSMTP(config)
}
var s Settings
decoder, err := mapstructure.NewDecoder(&mapstructure.DecoderConfig{Result: &s, ErrorUnused: true, TagName: "mapstructure"})
if err != nil {
+97
View File
@@ -0,0 +1,97 @@
package email
import (
"encoding/json"
"regexp"
"sort"
"strings"
"unicode"
)
const maxErrorBody = 16 << 10
const maxErrorMessage = 512
var (
errorName = regexp.MustCompile(`^[a-zA-Z0-9_.-]{1,64}$`)
addresses = regexp.MustCompile(`[^\s<>"'(),;:]+@[^\s<>"'(),;:]+`)
urls = regexp.MustCompile(`(?i)https?://[^\s<>"']+`)
tokens = regexp.MustCompile(`[a-zA-Z0-9_+/-]{32,}={0,2}`)
numbers = regexp.MustCompile(`[0-9]{4,}`)
)
// Only the documented human-readable error fields cross the adapter boundary.
// Full bodies, headers and request data must never reach logs or public failures.
func providerFailure(body, request []byte, secrets ...string) (string, string) {
var dto struct {
Name string `json:"name"`
Message string `json:"message"`
Errors []struct {
Message string `json:"message"`
} `json:"errors"`
}
if err := json.Unmarshal(body, &dto); err != nil {
return "", "Provider error details unavailable: invalid JSON response"
}
if dto.Message == "" && len(dto.Errors) > 0 {
dto.Message = dto.Errors[0].Message
}
// Providers can echo any scalar string from their request DTOs. Strip those
// values before truncation, in addition to credentials and embedded tokens.
var payload any
if json.Unmarshal(request, &payload) == nil {
for _, value := range requestStrings(payload) {
secrets = append(secrets, value)
// Rendered login/verification codes and numeric account references
// can be echoed as fragments of an otherwise longer email body.
secrets = append(secrets, numbers.FindAllString(value, -1)...)
}
}
name := safeErrorText(dto.Name, secrets)
if !errorName.MatchString(name) {
name = ""
}
message := safeErrorText(dto.Message, secrets)
if message == "" {
message = "Provider did not include an error message"
}
return name, message
}
func requestStrings(value any) []string {
var values []string
switch value := value.(type) {
case string:
values = append(values, value)
case []any:
for _, item := range value {
values = append(values, requestStrings(item)...)
}
case map[string]any:
for _, item := range value {
values = append(values, requestStrings(item)...)
}
}
return values
}
func safeErrorText(text string, secrets []string) string {
sort.Slice(secrets, func(i, j int) bool { return len(secrets[i]) > len(secrets[j]) })
var replacements []string
for _, secret := range secrets {
if secret != "" {
replacements = append(replacements, secret, "[redacted]")
}
}
text = strings.NewReplacer(replacements...).Replace(text)
text = addresses.ReplaceAllString(text, "[redacted-address]")
text = urls.ReplaceAllString(text, "[redacted-url]")
text = tokens.ReplaceAllString(text, "[redacted-token]")
text = strings.Join(strings.FieldsFunc(text, func(r rune) bool {
return unicode.IsSpace(r) || unicode.IsControl(r) || unicode.Is(unicode.Cf, r)
}), " ")
runes := []rune(text)
if len(runes) > maxErrorMessage {
text = string(runes[:maxErrorMessage]) + "…"
}
return text
}
@@ -0,0 +1,90 @@
package email
import (
"context"
"encoding/json"
"io"
"net/http"
"net/http/httptest"
"strings"
"testing"
"unicode/utf8"
"git.sendico.io/tech/sxchnge/notifications/model"
)
func TestProviderErrorDiagnostics(t *testing.T) {
for _, tc := range []struct {
name, body, code, message string
}{
{"resend", `{"name":"validation_error","message":"The sxchnge.com domain is not verified"}`, "validation_error", "The sxchnge.com domain is not verified"},
{"sendgrid", `{"errors":[{"message":"The sender identity is not verified","field":"from","help":"private URL"}]}`, "", "The sender identity is not verified"},
{"no message", `{"unrecognized":"private response"}`, "", "Provider did not include an error message"},
{"malformed", `{"message":"private response"`, "", "Provider error details unavailable: invalid JSON response"},
{"wrong type", `{"message":{"secret":"private response"}}`, "", "Provider error details unavailable: invalid JSON response"},
{"trailing", `{"message":"private response"} {}`, "", "Provider error details unavailable: invalid JSON response"},
{"oversized", `{"message":"` + strings.Repeat("x", maxErrorBody) + `"}`, "", "Provider error details unavailable: incomplete or oversized response"},
} {
t.Run(tc.name, func(t *testing.T) {
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
w.WriteHeader(http.StatusForbidden)
_, _ = io.WriteString(w, tc.body)
}))
defer server.Close()
c := client{http: server.Client(), endpoint: server.URL}
out := c.post(context.Background(), "", struct{}{})
if out.State != model.Failed || out.Code != "provider_authentication_failed" || out.HTTPStatus != 403 || out.ProviderError != tc.code || out.ProviderMessage != tc.message {
t.Fatalf("unexpected provider outcome: %+v", out)
}
})
}
}
func TestProviderDiagnosticsRedactRequestDataBeforeTruncation(t *testing.T) {
key := "re_credential-secret"
request := []byte(`{"from":"Sender <sender@example.com>","to":["recipient@example.com"],"personalizations":[{"dynamic_template_data":{"secret":"short-private-value"}}],"subject":"Private subject","text":"Private text","html":"<p>Private HTML</p>","content":[{"value":"Your verification code is 123456; account 9876543210"}]}`)
token := strings.Repeat("a", 43)
message := "Credential " + key + "; recipient@example.com; Private subject; Private text; <p>Private HTML</p>; short-private-value; https://example.com/verify?token=" + token + "; " + token + "; 123456; 9876543210; job-secret\n\r\x00\u202eforged-log"
body, err := json.Marshal(map[string]string{"name": "validation_error", "message": message})
if err != nil {
t.Fatal(err)
}
name, result := providerFailure(body, request, key, "job-secret")
if name != "validation_error" || !strings.Contains(result, "forged-log") || !strings.Contains(result, "[redacted]") {
t.Fatal("lost useful diagnostic", name, result)
}
for _, forbidden := range []string{key, "recipient@example.com", "Private subject", "Private text", "Private HTML", "short-private-value", "https://", token, "123456", "9876543210", "job-secret", "\n", "\r", "\x00", "\u202e"} {
if strings.Contains(result, forbidden) {
t.Fatalf("diagnostic contains sensitive value or control %q", forbidden)
}
}
body, err = json.Marshal(map[string]string{"name": "validation_error", "message": strings.Repeat("ą", maxErrorMessage+20) + key})
if err != nil {
t.Fatal(err)
}
_, result = providerFailure(body, request, key)
if !utf8.ValidString(result) || utf8.RuneCountInString(result) != maxErrorMessage+1 || !strings.HasSuffix(result, "…") || strings.Contains(result, "re_") {
t.Fatal("invalid or unbounded diagnostic")
}
}
func TestSuccessfulResponsesDoNotExposeProviderContent(t *testing.T) {
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
_, _ = io.WriteString(w, `{"name":"private","message":"private provider response"}`)
}))
defer server.Close()
c := client{http: server.Client(), endpoint: server.URL}
if out := c.post(context.Background(), "", struct{}{}); out.State != model.Accepted || out.HTTPStatus != 200 || out.ProviderMessage != "" || out.ProviderError != "" {
t.Fatal("success leaked provider content", out)
}
}
func TestTransportFailureHasSanitizedDiagnostic(t *testing.T) {
ctx, cancel := context.WithCancel(context.Background())
cancel()
c := client{http: http.DefaultClient, key: "credential-secret", endpoint: "https://example.com/emails?credential=credential-secret"}
out := c.post(ctx, "job-secret", struct{}{})
if out.State != model.Unknown || !strings.Contains(out.ProviderMessage, "context canceled") || strings.Contains(out.ProviderMessage, "credential-secret") || strings.Contains(out.ProviderMessage, "https://") {
t.Fatal("transport cause missing or unsafe", out)
}
}
+17 -8
View File
@@ -28,14 +28,15 @@ func (c *client) post(ctx context.Context, key string, payload any) Outcome {
}
resp, err := c.http.Do(req)
if err != nil {
return Outcome{State: model.Unknown, Code: "provider_outcome_unknown"}
return Outcome{State: model.Unknown, Code: "provider_outcome_unknown", ProviderMessage: safeErrorText(err.Error(), []string{c.key, key})}
}
defer resp.Body.Close()
// Discard a bounded response without logging provider-controlled content.
_, _ = io.Copy(io.Discard, io.LimitReader(resp.Body, 64<<10))
// Never pass raw provider bodies through the outcome or log them directly.
details, readErr := io.ReadAll(io.LimitReader(resp.Body, maxErrorBody+1))
outcome := Outcome{HTTPStatus: resp.StatusCode}
switch {
case resp.StatusCode >= 200 && resp.StatusCode < 300:
return Outcome{State: model.Accepted}
outcome.State = model.Accepted
case resp.StatusCode == 429:
delay := time.Minute
if n, e := strconv.Atoi(resp.Header.Get("Retry-After")); e == nil && n > 0 && n <= 86400 {
@@ -43,12 +44,20 @@ func (c *client) post(ctx context.Context, key string, payload any) Outcome {
} else if date, e := http.ParseTime(resp.Header.Get("Retry-After")); e == nil && time.Until(date) > 0 {
delay = time.Until(date)
}
return Outcome{State: model.RetryWait, Code: "provider_rate_limited", RetryAfter: delay}
outcome.State, outcome.Code, outcome.RetryAfter = model.RetryWait, "provider_rate_limited", delay
case resp.StatusCode >= 500 || resp.StatusCode == 408:
return Outcome{State: model.Unknown, Code: "provider_outcome_unknown"}
outcome.State, outcome.Code = model.Unknown, "provider_outcome_unknown"
case resp.StatusCode == 401 || resp.StatusCode == 403:
return Outcome{State: model.Failed, Code: "provider_authentication_failed"}
outcome.State, outcome.Code = model.Failed, "provider_authentication_failed"
default:
return Outcome{State: model.Failed, Code: "provider_rejected"}
outcome.State, outcome.Code = model.Failed, "provider_rejected"
}
if outcome.State != model.Accepted {
if readErr != nil || len(details) > maxErrorBody {
outcome.ProviderMessage = "Provider error details unavailable: incomplete or oversized response"
} else {
outcome.ProviderError, outcome.ProviderMessage = providerFailure(details, body, c.key, key)
}
}
return outcome
}
+49
View File
@@ -0,0 +1,49 @@
package email
import (
"bytes"
"crypto/sha256"
"fmt"
"mime"
"mime/multipart"
"mime/quotedprintable"
"net/mail"
"net/textproto"
"strings"
"time"
"git.sendico.io/tech/sxchnge/notifications/model"
merrors "git.sendico.io/tech/sxchnge/shared/errors"
)
func smtpMessage(s smtpSettings, key string, email model.Email) ([]byte, error) {
if !ValidAddress(email.Address) || strings.ContainsAny(email.Subject, "\r\n") || (email.Text == "" && email.HTML == "") {
return nil, merrors.InvalidArgument("invalid email content")
}
var body bytes.Buffer
parts := multipart.NewWriter(&body)
for _, part := range []struct{ kind, value string }{{"text/plain", email.Text}, {"text/html", email.HTML}} {
if part.value == "" {
continue
}
writer, err := parts.CreatePart(textproto.MIMEHeader{"Content-Type": {part.kind + "; charset=utf-8"}, "Content-Transfer-Encoding": {"quoted-printable"}})
if err != nil {
return nil, merrors.InternalWrap(err, "create email MIME part")
}
encoded := quotedprintable.NewWriter(writer)
if _, err = encoded.Write([]byte(part.value)); err != nil {
return nil, merrors.InternalWrap(err, "encode email MIME part")
}
if err = encoded.Close(); err != nil {
return nil, merrors.InternalWrap(err, "finish email MIME part")
}
}
if err := parts.Close(); err != nil {
return nil, merrors.InternalWrap(err, "finish email MIME body")
}
from := (&mail.Address{Name: s.SenderName, Address: s.SenderAddress}).String()
to := (&mail.Address{Address: email.Address}).String()
domain := s.SenderAddress[strings.LastIndex(s.SenderAddress, "@")+1:]
headers := fmt.Sprintf("From: %s\r\nTo: %s\r\nSubject: %s\r\nDate: %s\r\nMessage-ID: <%x@%s>\r\nMIME-Version: 1.0\r\nContent-Type: multipart/alternative; boundary=%q\r\n\r\n", from, to, mime.QEncoding.Encode("utf-8", email.Subject), time.Now().UTC().Format(time.RFC1123Z), sha256.Sum256([]byte(key)), domain, parts.Boundary())
return append([]byte(headers), body.Bytes()...), nil
}
+122
View File
@@ -0,0 +1,122 @@
package email
import (
"bytes"
"context"
"crypto/tls"
"encoding/base64"
"errors"
"io"
"net"
"net/smtp"
"net/textproto"
"os"
"strconv"
"strings"
"time"
"git.sendico.io/tech/sxchnge/notifications/model"
merrors "git.sendico.io/tech/sxchnge/shared/errors"
"github.com/go-viper/mapstructure/v2"
)
type smtpSettings struct {
Host string `mapstructure:"host"`
Port int `mapstructure:"port"`
Username string `mapstructure:"username"`
PasswordEnv string `mapstructure:"password_env"`
SenderAddress string `mapstructure:"sender_address"`
SenderName string `mapstructure:"sender_name"`
NetworkTimeout int `mapstructure:"network_timeout"`
}
type smtpSender struct {
settings smtpSettings
password string
tls *tls.Config
}
func newSMTP(config Config) (Sender, error) {
var s smtpSettings
decoder, err := mapstructure.NewDecoder(&mapstructure.DecoderConfig{Result: &s, ErrorUnused: true, TagName: "mapstructure"})
if err != nil {
return nil, merrors.InternalWrap(err, "create SMTP settings decoder")
}
if decoder.Decode(config.Settings) != nil || s.Host == "" || strings.ContainsAny(s.Host, " /@\r\n\t") || s.Port < 1 || s.Port > 65535 || !ValidAddress(s.Username) || !ValidAddress(s.SenderAddress) || strings.ContainsAny(s.SenderName, "\r\n") || s.NetworkTimeout < 1 || s.NetworkTimeout > 20 {
return nil, merrors.InvalidArgument("invalid SMTP settings")
}
password := os.Getenv(s.PasswordEnv)
if s.PasswordEnv == "" || password == "" || strings.ContainsAny(password, "\r\n\x00") {
return nil, merrors.FailedPrecondition("SMTP password environment variable is missing or invalid")
}
return &smtpSender{settings: s, password: password, tls: &tls.Config{ServerName: s.Host, MinVersion: tls.VersionTLS12}}, nil
}
// SMTP uses a fresh authenticated TLS connection per attempt. No automatic
// resend follows an uncertain result, since SMTP has no idempotency guarantee.
func (s *smtpSender) Send(ctx context.Context, key string, email model.Email) Outcome {
body, err := smtpMessage(s.settings, key, email)
if err != nil {
return Outcome{State: model.Failed, Code: "invalid_payload"}
}
ctx, cancel := context.WithTimeout(ctx, time.Duration(s.settings.NetworkTimeout)*time.Second)
defer cancel()
dialer := tls.Dialer{NetDialer: &net.Dialer{}, Config: s.tls}
conn, err := dialer.DialContext(ctx, "tcp", net.JoinHostPort(s.settings.Host, strconv.Itoa(s.settings.Port)))
if err != nil {
return s.failure(err, key, email)
}
defer conn.Close()
stop := context.AfterFunc(ctx, func() { _ = conn.Close() })
defer stop()
deadline, _ := ctx.Deadline()
if err = conn.SetDeadline(deadline); err != nil {
return s.failure(err, key, email)
}
client, err := smtp.NewClient(conn, s.settings.Host)
if err == nil {
err = client.Auth(smtp.PlainAuth("", s.settings.Username, s.password, s.settings.Host))
}
if err == nil {
err = client.Mail(s.settings.SenderAddress)
}
if err == nil {
err = client.Rcpt(email.Address)
}
if err != nil {
return s.failure(err, key, email)
}
writer, err := client.Data()
if err != nil {
return s.failure(err, key, email)
}
if _, err = io.Copy(writer, bytes.NewReader(body)); err == nil {
err = writer.Close()
}
if err != nil {
return s.failure(err, key, email)
}
// DATA's final 250 is acceptance. A later QUIT error cannot undo it.
_ = client.Quit()
return Outcome{State: model.Accepted, SMTPStatus: 250}
}
func (s *smtpSender) failure(err error, key string, email model.Email) Outcome {
secrets := []string{s.password, s.settings.Username, key, email.Address, email.Subject, email.HTML, email.Text}
secrets = append(secrets, base64.StdEncoding.EncodeToString([]byte("\x00"+s.settings.Username+"\x00"+s.password)))
secrets = append(secrets, numbers.FindAllString(email.Text+" "+email.HTML, -1)...)
out := Outcome{State: model.Unknown, Code: "provider_outcome_unknown", ProviderError: "smtp", ProviderMessage: safeErrorText(err.Error(), secrets)}
var reply *textproto.Error
if errors.As(err, &reply) {
out.SMTPStatus = reply.Code
if reply.Code >= 400 && reply.Code < 500 {
out.State, out.Code, out.RetryAfter = model.RetryWait, "provider_temporarily_unavailable", time.Minute
} else if reply.Code >= 500 && reply.Code < 600 {
out.State, out.Code = model.Failed, "provider_rejected"
if reply.Code == 530 || reply.Code == 534 || reply.Code == 535 || reply.Code == 538 {
out.Code = "provider_authentication_failed"
}
}
}
return out
}
+223
View File
@@ -0,0 +1,223 @@
package email
import (
"context"
"crypto/tls"
"crypto/x509"
"encoding/base64"
"fmt"
"io"
"mime"
"mime/multipart"
"net"
"net/http/httptest"
"net/mail"
"net/textproto"
"strconv"
"strings"
"sync/atomic"
"testing"
"time"
"git.sendico.io/tech/sxchnge/notifications/model"
)
type smtpFixture struct {
settings smtpSettings
root *x509.CertPool
body chan []byte
auth atomic.Bool
}
func smtpServer(t *testing.T, reject string, code int) *smtpFixture {
t.Helper()
certServer := httptest.NewTLSServer(nil)
cert := certServer.TLS.Certificates[0]
root := x509.NewCertPool()
root.AddCert(certServer.Certificate())
certServer.Close()
listener, err := tls.Listen("tcp", "127.0.0.1:0", &tls.Config{Certificates: []tls.Certificate{cert}, MinVersion: tls.VersionTLS12})
if err != nil {
t.Fatal(err)
}
t.Cleanup(func() { _ = listener.Close() })
host, port, _ := net.SplitHostPort(listener.Addr().String())
n, _ := strconv.Atoi(port)
f := &smtpFixture{settings: smtpSettings{Host: host, Port: n, Username: "sender@example.com", SenderAddress: "sender@example.com", SenderName: "SXCHNGE", NetworkTimeout: 2}, root: root, body: make(chan []byte, 1)}
go func() {
conn, err := listener.Accept()
if err != nil {
return
}
defer conn.Close()
_ = conn.SetDeadline(time.Now().Add(5 * time.Second))
if reject == "greeting" {
_ = conn.(*tls.Conn).Handshake()
_, _ = io.Copy(io.Discard, conn)
return
}
c := textproto.NewConn(conn)
if c.PrintfLine("220 localhost ESMTP") != nil {
return
}
for {
line, err := c.ReadLine()
if err != nil {
return
}
command := strings.SplitN(line, " ", 2)[0]
if command == reject {
_ = c.PrintfLine("%d rejected recipient user@example.com credential test-password", code)
continue
}
switch command {
case "EHLO":
_ = c.PrintfLine("250-localhost\r\n250 AUTH PLAIN")
case "AUTH":
want := "AUTH PLAIN " + base64.StdEncoding.EncodeToString([]byte("\x00sender@example.com\x00test-password"))
f.auth.Store(line == want)
_ = c.PrintfLine("235 authenticated")
case "MAIL", "RCPT":
_ = c.PrintfLine("250 OK")
case "DATA":
_ = c.PrintfLine("354 send message")
body, err := c.ReadDotBytes()
if err != nil {
return
}
f.body <- body
if reject == "final" {
_ = c.PrintfLine("%d final rejection", code)
} else if reject == "disconnect" {
return
} else {
_ = c.PrintfLine("250 queued")
}
case "QUIT":
// A lost QUIT reply must not turn accepted DATA into UNKNOWN.
return
default:
_ = c.PrintfLine("500 unexpected command")
}
}
}()
return f
}
func (f *smtpFixture) sender() *smtpSender {
return &smtpSender{settings: f.settings, password: "test-password", tls: &tls.Config{ServerName: f.settings.Host, RootCAs: f.root, MinVersion: tls.VersionTLS12}}
}
func TestSMTPAcceptanceAndMIME(t *testing.T) {
f := smtpServer(t, "", 0)
sender := f.sender()
email := model.Email{Address: "user@example.com", Subject: "Подтвердите почту", Text: "Confirm code 123456\n.line", HTML: "<p>Confirm code 123456</p>"}
out := sender.Send(context.Background(), "notification-id", email)
if out.State != model.Accepted || out.SMTPStatus != 250 || !f.auth.Load() {
t.Fatal("SMTP acceptance/authentication failed", out)
}
msg, err := mail.ReadMessage(strings.NewReader(string(<-f.body)))
if err != nil {
t.Fatal(err)
}
subject, err := new(mime.WordDecoder).DecodeHeader(msg.Header.Get("Subject"))
if err != nil || subject != email.Subject {
t.Fatal("wrong MIME subject", subject, err)
}
if msg.Header.Get("From") != "\"SXCHNGE\" <sender@example.com>" || !strings.HasSuffix(msg.Header.Get("Message-ID"), "@example.com>") {
t.Fatal("invalid sender/message identifier")
}
kind, params, err := mime.ParseMediaType(msg.Header.Get("Content-Type"))
if err != nil || kind != "multipart/alternative" {
t.Fatal("wrong MIME type", kind, err)
}
parts := multipart.NewReader(msg.Body, params["boundary"])
for _, want := range []string{email.Text, email.HTML} {
part, err := parts.NextPart()
if err != nil {
t.Fatal(err)
}
body, err := io.ReadAll(part)
if err != nil || strings.ReplaceAll(string(body), "\r\n", "\n") != want {
t.Fatal("incorrect MIME content", err)
}
}
}
func TestSMTPFailureSafety(t *testing.T) {
for _, tc := range []struct {
stage string
code int
state model.State
failure string
}{
{"AUTH", 535, model.Failed, "provider_authentication_failed"},
{"RCPT", 550, model.Failed, "provider_rejected"},
{"final", 451, model.RetryWait, "provider_temporarily_unavailable"},
{"disconnect", 0, model.Unknown, "provider_outcome_unknown"},
} {
t.Run(fmt.Sprintf("%s-%d", tc.stage, tc.code), func(t *testing.T) {
f := smtpServer(t, tc.stage, tc.code)
out := f.sender().Send(context.Background(), "id", model.Email{Address: "user@example.com", Subject: "Subject", Text: "Body"})
if out.State != tc.state || out.Code != tc.failure || out.SMTPStatus != tc.code {
t.Fatal("incorrect SMTP classification", out)
}
if strings.Contains(out.ProviderMessage, "test-password") || strings.Contains(out.ProviderMessage, "user@example.com") {
t.Fatal("SMTP rejection leaked private data")
}
if tc.code == 451 && out.RetryAfter <= 0 {
t.Fatal("temporary rejection did not schedule retry")
}
})
}
}
func TestSMTPCertificateAndCancellation(t *testing.T) {
t.Run("untrusted certificate", func(t *testing.T) {
f := smtpServer(t, "", 0)
sender := f.sender()
sender.tls.RootCAs = x509.NewCertPool()
if out := sender.Send(context.Background(), "id", model.Email{Address: "user@example.com", Text: "Body"}); out.State != model.Unknown || f.auth.Load() {
t.Fatal("untrusted TLS accepted", out)
}
})
t.Run("cancel blocked greeting", func(t *testing.T) {
f := smtpServer(t, "greeting", 0)
ctx, cancel := context.WithTimeout(context.Background(), 100*time.Millisecond)
defer cancel()
started := time.Now()
if out := f.sender().Send(ctx, "id", model.Email{Address: "user@example.com", Text: "Body"}); out.State != model.Unknown || time.Since(started) > time.Second {
t.Fatal("SMTP ignored cancellation", out)
}
})
}
func TestSMTPConfigAndHeaderValidation(t *testing.T) {
t.Setenv("TEST_SMTP_PASSWORD", "test-password")
valid := map[string]any{"host": "mail.example.com", "port": 465, "username": "sender@example.com", "password_env": "TEST_SMTP_PASSWORD", "sender_address": "sender@example.com", "network_timeout": 10}
if _, err := New(Config{Driver: SMTP, Settings: valid}); err != nil {
t.Fatal(err)
}
for field, value := range map[string]any{"port": 0, "host": "https://bad", "username": "bad", "network_timeout": 0, "password_env": "MISSING_SMTP_PASSWORD", "typo": true} {
settings := map[string]any{}
for key, item := range valid {
settings[key] = item
}
settings[field] = value
if _, err := New(Config{Driver: SMTP, Settings: settings}); err == nil {
t.Fatalf("invalid setting %s accepted", field)
}
}
if _, err := smtpMessage(smtpSettings{}, "id", model.Email{Address: "user@example.com", Subject: "Subject\r\nBcc: stolen@example.com", Text: "Body"}); err == nil {
t.Fatal("header injection accepted")
}
}
func TestSMTPAuthenticationDiagnosticRedactsEncodedCredential(t *testing.T) {
f := &smtpSender{settings: smtpSettings{Username: "sender@example.com"}, password: "a"}
auth := base64.StdEncoding.EncodeToString([]byte("\x00sender@example.com\x00a"))
out := f.failure(&textproto.Error{Code: 535, Msg: "Rejected AUTH " + auth}, "id", model.Email{})
if out.SMTPStatus != 535 || strings.Contains(out.ProviderMessage, auth) || !strings.Contains(out.ProviderMessage, "AUTH [redacted]") {
t.Fatalf("encoded SMTP credential was not fully redacted: %q", out.ProviderMessage)
}
}
+19 -4
View File
@@ -5,6 +5,7 @@ import (
"errors"
"time"
"git.sendico.io/tech/sxchnge/notifications/email"
"git.sendico.io/tech/sxchnge/notifications/model"
merrors "git.sendico.io/tech/sxchnge/shared/errors"
"go.uber.org/zap"
@@ -23,6 +24,7 @@ func (s *Service) Dispatch(ctx context.Context, id string) (bool, error) {
if err != nil {
return false, err
}
var outcome email.Outcome
switch {
case !n.ExpiresAt.After(now):
n.State = model.Expired
@@ -36,7 +38,7 @@ func (s *Service) Dispatch(ctx context.Context, id string) (bool, error) {
deadline = n.ExpiresAt
}
sendCtx, cancel := context.WithDeadline(ctx, deadline)
outcome := s.sender.Send(sendCtx, n.ID.Hex(), n.Email)
outcome = s.sender.Send(sendCtx, n.ID.Hex(), n.Email)
cancel()
n.State = outcome.State
n.FailureCode = outcome.Code
@@ -62,14 +64,27 @@ func (s *Service) Dispatch(ctx context.Context, id string) (bool, error) {
// Persist the result even when shutdown canceled the provider request.
finishCtx, cancel := context.WithTimeout(context.WithoutCancel(ctx), 5*time.Second)
defer cancel()
if err = s.repo.Finish(finishCtx, n); err != nil {
return true, err
}
fields := []zap.Field{zap.String("notification_id", n.ID.Hex()), zap.String("driver", n.Driver), zap.String("state", string(n.State)), zap.Uint32("attempts", n.Attempts), zap.String("failure_code", n.FailureCode)}
if outcome.HTTPStatus != 0 {
fields = append(fields, zap.Int("provider_status", outcome.HTTPStatus))
}
if outcome.SMTPStatus != 0 {
fields = append(fields, zap.Int("provider_smtp_status", outcome.SMTPStatus))
}
if outcome.ProviderError != "" {
fields = append(fields, zap.String("provider_error", outcome.ProviderError))
}
if outcome.ProviderMessage != "" {
fields = append(fields, zap.String("provider_message", outcome.ProviderMessage))
}
// Keep the provider outcome observable even if its local result write fails.
if n.State == model.Accepted {
s.logger.Info("Email accepted by provider", fields...)
} else {
s.logger.Warn("Email delivery attempt finished", fields...)
}
if err = s.repo.Finish(finishCtx, n); err != nil {
return true, err
}
return true, nil
}
+44 -3
View File
@@ -2,18 +2,23 @@ package service
import (
"context"
"errors"
"fmt"
"testing"
"time"
"git.sendico.io/tech/sxchnge/notifications/email"
"git.sendico.io/tech/sxchnge/notifications/model"
merrors "git.sendico.io/tech/sxchnge/shared/errors"
"go.uber.org/zap"
"go.uber.org/zap/zaptest/observer"
)
type workerRepo struct {
Repository
job model.Notification
saved *model.Notification
job model.Notification
saved *model.Notification
finishErr error
}
func (r *workerRepo) Recover(context.Context, time.Time) error { return nil }
@@ -21,7 +26,10 @@ func (r *workerRepo) Claim(context.Context, string, time.Time) (*model.Notificat
n := r.job
return &n, nil
}
func (r *workerRepo) Finish(_ context.Context, n *model.Notification) error { r.saved = n; return nil }
func (r *workerRepo) Finish(_ context.Context, n *model.Notification) error {
r.saved = n
return r.finishErr
}
type workerSender struct {
calls int
@@ -62,3 +70,36 @@ func TestWorkerTerminalAndRetryOutcomes(t *testing.T) {
})
}
}
func TestWorkerLogsProviderFailureOnce(t *testing.T) {
for _, failPersistence := range []bool{false, true} {
t.Run(fmt.Sprint(failPersistence), func(t *testing.T) {
core, logs := observer.New(zap.WarnLevel)
repo := &workerRepo{job: model.Notification{Driver: "resend", ExpiresAt: time.Now().Add(time.Hour), Attempts: 1, Email: model.Email{Address: "private@example.com", Subject: "Private subject", Text: "Private text"}}}
if failPersistence {
repo.finishErr = merrors.ServiceUnavailable("storage offline")
}
sender := &workerSender{outcome: email.Outcome{State: model.Failed, Code: "provider_authentication_failed", HTTPStatus: 403, ProviderError: "validation_error", ProviderMessage: "The sxchnge.com domain is not verified"}}
svc := New(repo, nil, sender, "resend", zap.New(core))
if _, err := svc.Dispatch(context.Background(), "id"); !errors.Is(err, repo.finishErr) {
t.Fatal("unexpected persistence error", err)
}
entries := logs.All()
if len(entries) != 1 || entries[0].Message != "Email delivery attempt finished" {
t.Fatal("expected one provider outcome log", entries)
}
fields := entries[0].ContextMap()
if fields["provider_status"] != int64(403) || fields["provider_error"] != "validation_error" || fields["provider_message"] != "The sxchnge.com domain is not verified" {
t.Fatal("provider diagnostic missing", fields)
}
if repo.saved.FailureCode != "provider_authentication_failed" {
t.Fatal("public failure code changed", repo.saved.FailureCode)
}
for _, field := range []string{"address", "email", "subject", "text", "html", "parameters"} {
if _, ok := fields[field]; ok {
t.Fatalf("private notification field %s logged", field)
}
}
})
}
}
+4 -4
View File
@@ -12,7 +12,7 @@ import (
"go.mongodb.org/mongo-driver/v2/bson"
)
func (c *Client) post(ctx context.Context, resource, requestRef string, payload any) (receiptDTO, error) {
func (c *Client) post(ctx context.Context, resource, requestRef string, payload any, expectedStatus int) (receiptDTO, error) {
var receipt receiptDTO
token, err := c.token()
if err != nil {
@@ -38,7 +38,7 @@ func (c *Client) post(ctx context.Context, resource, requestRef string, payload
}
defer func() { _ = response.Body.Close() }()
// Never decode or expose upstream error bodies, which can contain assertions.
if response.StatusCode != http.StatusOK {
if response.StatusCode != expectedStatus {
return receipt, merrors.ServiceUnavailable("Sendico request was not accepted")
}
decoder := json.NewDecoder(io.LimitReader(response.Body, 16385))
@@ -54,14 +54,14 @@ func (c *Client) post(ctx context.Context, resource, requestRef string, payload
}
func (c *Client) Register(ctx context.Context, requestRef string) (string, error) {
dto, err := c.post(ctx, "individuals", requestRef, nil)
dto, err := c.post(ctx, "individuals", requestRef, nil, http.StatusCreated)
if err != nil {
return "", err
}
return mapSubject(dto)
}
func (c *Client) Submit(ctx context.Context, subject, source string, facts identity.Facts) (identity.Receipt, error) {
dto, err := c.post(ctx, "identity_facts", bson.NewObjectID().Hex(), mapFacts(subject, source, facts))
dto, err := c.post(ctx, "identity_facts", bson.NewObjectID().Hex(), mapFacts(subject, source, facts), http.StatusOK)
if err != nil {
return identity.Receipt{}, err
}
@@ -48,6 +48,7 @@ func TestSignupWireContractAndCredentialRotation(t *testing.T) {
if strings.Contains(r.URL.Path, "/individuals/") {
require.Equal(t, "registration-request", r.Header.Get("X-Request-ID"))
require.Empty(t, body)
w.WriteHeader(http.StatusCreated)
_, _ = fmt.Fprintf(w, `{"status":"success","data":{"subject_ref":%q,"registered_at":"2026-10-07T10:30:00.123Z"}}`, subject)
} else {
require.JSONEq(t, `{"subject_ref":"`+subject+`","source_submission_ref":"signup-1","facts":{"full_name":"Ada Lovelace","email":"ada@example.com","external_customer_ref":"local-user"}}`, string(body))
@@ -95,6 +96,36 @@ func TestRejectInvalidConfiguration(t *testing.T) {
require.Error(t, err)
}
func TestSignupReceiptsRequireOperationStatus(t *testing.T) {
for _, status := range []int{200, 201, 202, 204, 206, 300, 403, 409, 503} {
t.Run(http.StatusText(status), func(t *testing.T) {
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.WriteHeader(status)
_, _ = fmt.Fprintf(w, `{"status":"success","data":{"subject_ref":%q,"registered_at":"2026-10-07T10:30:00Z","submission_ref":%q,"accepted_at":"2026-10-07T10:30:00Z"}}`, subject, submission)
}))
defer server.Close()
client, err := Open(config(t, server.URL))
require.NoError(t, err)
ref, err := client.Register(context.Background(), "registration-request")
if status == http.StatusCreated {
require.NoError(t, err)
require.Equal(t, subject, ref)
} else {
require.Error(t, err)
require.Empty(t, ref)
}
receipt, err := client.Submit(context.Background(), subject, "key", identity.Facts{})
if status == http.StatusOK {
require.NoError(t, err)
require.Equal(t, submission, receipt.SubmissionRef)
} else {
require.Error(t, err)
require.Empty(t, receipt)
}
})
}
}
func TestUpstreamFailuresNeverProduceReceiptsOrLeakBodies(t *testing.T) {
for _, tc := range []struct {
name string
@@ -113,7 +144,11 @@ func TestUpstreamFailuresNeverProduceReceiptsOrLeakBodies(t *testing.T) {
} {
t.Run(tc.name, func(t *testing.T) {
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.WriteHeader(tc.status)
status := tc.status
if status == http.StatusOK && strings.Contains(r.URL.Path, "/individuals/") {
status = http.StatusCreated
}
w.WriteHeader(status)
_, _ = io.WriteString(w, tc.body)
}))
defer server.Close()