CCAT System Integration Documentation

Overview

This documentation covers the CCAT System Integration, including:

• Docker Compose configuration and deployment

• Configuration management with Dynaconf

• Backup and restore procedures

• Monitoring with Grafana, Prometheus, and Loki

• Service orchestration across multiple nodes

Key Components

Infrastructure Services:

• PostgreSQL database with replication

• Redis cache with TLS

• InfluxDB for time-series data

• MinIO object storage

• Grafana dashboards

• Prometheus monitoring

• Loki log aggregation

Deployment Configurations:

• Local development (docker-compose.local.yml)

• Staging environment (docker-compose.staging.yml)

• Production nodes:

  • Database node (docker-compose.production.db.yml)

  • Data processing node (docker-compose.production.data.yml)

  • Input node (docker-compose.production.input.yml)

  • Reuna VM (docker-compose.production.reuna-vm.yml)

Quick Links

• Operations Database

• Data Transfer Documentation

Documentation Contents

Contents:

• Getting Started
  • Installation
  • Prerequisites
  • Quick Start
• Configuration
  • Docker Compose Configuration
  • Dynaconf Settings
  • Environment Variables
• Deployment
  • Overview
  • Main-stack deploy (default group)
  • Docs deploy (docs group)
  • Causal consistency model
  • Local Development
  • Staging Environment
  • Production Deployment
• Monitoring
  • Grafana Dashboards
  • Prometheus Metrics
  • Loki Logging
• Backup and Restore
  • Backup Strategy
  • Automated Backups
  • Restore Procedures
• PAT Rotation and Diagnostics
  • Why a PAT is needed
  • Generating a PAT with minimum permissions
  • Where the secret must be set
  • Checking PAT status
  • Rotating the PAT
  • Recovery after partial failure
  • Recommended rotation cadence

Getting Started

For setup instructions, see the project README and the Getting Started guide.

Related Documentation

• Main Documentation

• Operations Database

• Data Transfer System

Indices and tables

• Index

• Search Page

The system-integration repository contains Docker Compose configurations, Kubernetes manifests, and supporting infrastructure for deploying the complete CCAT data center stack across different environments.

Deployment Environments

Local Development

• docker-compose.local.yml - Local development environment

• docker-compose.dev.yml - Development with live reload

Staging

• docker-compose.staging.yml - Staging environment

• docker-compose.staging.batleth.yml - Batleth staging server

• cologne-staging-k8s/ - Kubernetes configuration for Cologne staging

Production

• docker-compose.production.*.yml - Production deployment configurations

• Separated into database, data transfer, and input components

Infrastructure Components

Core Services

• PostgreSQL: Operations database with replication support

• Redis: Caching and session management

• MinIO: S3-compatible object storage

• InfluxDB: Time-series metrics storage

Monitoring & Observability

• Prometheus: Metrics collection and alerting

• Grafana: Dashboards and visualization

• Loki: Log aggregation

• Promtail: Log collection agent

Supporting Services

• Nginx: Reverse proxy and load balancing

• Celery: Distributed task queue for background jobs

Directory Structure

postgres/

PostgreSQL initialization scripts and configuration

grafana/

Grafana dashboards and provisioning configuration

prometheus/

Prometheus monitoring configuration

loki-config/ & promtail-config/

Log aggregation configuration

jenkins_pipelines/

CI/CD pipeline definitions

tools/

Deployment and management scripts

Deployment Guides

Kubernetes Deployments

The cologne-staging-k8s/ directory contains Kubernetes manifests for deploying on the Cologne cluster:

• Deployment configurations

• Service definitions

• Ingress routing

• Persistent volume claims

See cologne-staging-k8s/README.md for detailed deployment instructions.

Docker Compose Deployments

Docker Compose files are organized by environment and component:

1. Choose your environment: local, staging, or production

2. Select components: database, data transfer, input processing

3. Run: docker-compose -f <config-file> up -d

Initialization Scripts

init_ops_db_entrypoint.sh

Initializes the operations database schema and loads seed data

init-influxdb.sh

Sets up InfluxDB databases and retention policies

init-minio.sh

Configures MinIO buckets and access policies

Monitoring

The monitoring stack provides comprehensive observability:

• System Metrics: CPU, memory, disk, network via Prometheus

• Application Metrics: Custom metrics from Python services

• Logs: Centralized logging with Loki

• Dashboards: Pre-configured Grafana dashboards for all services

Access Grafana at the configured endpoint to view dashboards and alerts.