Data and streaming
Atlas has two different data planes in this repository:
- streaming through Amazon MSK
- relational persistence through PostgreSQL RDS for the dashboard backend
Kafka path
| Component | Current role |
|---|---|
| Events ingestion service | publishes Atlas events to Kafka |
| Amazon MSK | shared Kafka backbone |
| Kafka UI | operator view into brokers, topics, and messages |
| MSK Connect S3 sink | optional export path from Kafka topics to S3 |
The events secret template currently seeds:
KAFKA_EVENTS_TOPIC = "atlas.events.raw"KAFKA_DLQ_TOPIC = "atlas.events.dlq"KAFKA_SASL_MECHANISM = "AWS_MSK_IAM"KAFKA_BROKERS = module.msk.bootstrap_brokers_tls
MSK connectivity modes
| Endpoint type | Use |
|---|---|
bootstrap_brokers_tls | internal IAM + TLS access for ECS services and Kafka UI |
bootstrap_brokers_public_tls | public IAM + TLS access for external clients |
| multi-VPC connectivity | enabled in production values for private connectivity patterns |
Optional S3 export path
When the sink is enabled, Atlas provisions:
- one bucket for exported Kafka objects
- one plugin artifact bucket
- one MSK Connect custom plugin
- one MSK Connect connector
- one connector log group
Objects are written under the configured prefix and partitioned with the selected field names.
PostgreSQL path
The dashboard backend has a dedicated PostgreSQL instance with:
- its own subnet groups
- its own parameter group
- its own security group
- an RDS-managed master password secret
The application-facing DATABASE_URL is still owned by the dashboard backend secret, not injected automatically from Terraform outputs.
OpenSpec vs current shipped code
warning
The OpenSpec archive contains history around VPC Lattice and ClickPipes integration, but the current Terraform roots do not instantiate a dedicated VPC Lattice module. The shipped implementation today relies on public IAM + TLS outputs and environment-driven MSK connectivity settings.