Configuration
=============

.. verified:: 2025-11-25
   :reviewer: Christof Buchbender

The ops-db package uses `DynaConf <https://www.dynaconf.com/>`_ for flexible, environment-based configuration management. This allows you to configure database connections, credentials, and other settings without modifying code.

Overview
--------

Configuration in ops-db works through three layers:

1. **Default settings** in ``settings.toml`` - Base configuration values
2. **Environment groups** - Override defaults for specific environments (development, production, etc.)
3. **Environment variables** - Override any setting at runtime via ``CCAT_OPS_DB_*`` prefixed variables

The active environment is selected using the ``ENV_FOR_DYNACONF`` environment variable.

How DynaConf Works in ops-db
----------------------------

DynaConf is initialized in ``ccat_ops_db/config/config.py`` with the following settings:

* **envvar_prefix**: ``"CCAT_OPS_DB"`` - All environment variables must be prefixed with this
* **environments**: ``True`` - Enables environment-based configuration groups
* **load_dotenv**: ``True`` - Automatically loads ``.env`` files if present
* **default_env**: ``"default"`` - The default environment if none is specified

This means you can override any setting by setting an environment variable like:

.. code-block:: bash

   export CCAT_OPS_DB_DATABASE_TYPE="postgresql"
   export CCAT_OPS_DB_DATABASE_POSTGRESQL_HOST="my-db-server"

Selecting an Environment
------------------------

The active environment determines which section from ``settings.toml`` is loaded. Set it using the ``ENV_FOR_DYNACONF`` environment variable:

.. code-block:: bash

   # Use the development environment
   export ENV_FOR_DYNACONF=development

   # Use the production environment
   export ENV_FOR_DYNACONF=production

   # Use the default environment (SQLite for local development)
   export ENV_FOR_DYNACONF=default
   # or simply unset it:
   unset ENV_FOR_DYNACONF

Available environments in ``settings.toml``:

* ``default`` - SQLite database for local development
* ``development`` - PostgreSQL with Docker service names
* ``development-ccat`` - PostgreSQL with replica service names
* ``local`` - PostgreSQL for local development
* ``staging-mekleth`` - Staging environment on mekleth server
* ``staging-batleth`` - Staging environment on batleth server
* ``docker-compose`` - PostgreSQL for Docker Compose setups
* ``production`` - Production PostgreSQL server
* ``production-ccat`` - Production with replica services

Environment Variables
---------------------

You can override any setting from ``settings.toml`` using environment variables. The variable name is constructed by:

1. Prefixing with ``CCAT_OPS_DB_``
2. Converting the setting name to uppercase
3. Replacing dots with underscores

Examples:

.. code-block:: bash

   # Override database type
   export CCAT_OPS_DB_DATABASE_TYPE="postgresql"

   # Override PostgreSQL host
   export CCAT_OPS_DB_DATABASE_POSTGRESQL_HOST="db.example.com"

   # Override PostgreSQL port
   export CCAT_OPS_DB_DATABASE_POSTGRESQL_PORT="5433"

   # Override PostgreSQL user
   export CCAT_OPS_DB_DATABASE_POSTGRESQL_USER="myuser"

   # Override PostgreSQL password
   export CCAT_OPS_DB_DATABASE_POSTGRESQL_PASSWORD="mypassword"

   # Override PostgreSQL database name
   export CCAT_OPS_DB_DATABASE_POSTGRESQL_DATABASE="my_database"

   # Override SQLite database path
   export CCAT_OPS_DB_DATABASE_SQLITE_DATABASE="/path/to/database.sqlite"

For nested settings (like ``status.PENDING``), use double underscores:

.. code-block:: bash

   export CCAT_OPS_DB_STATUS__PENDING="waiting"

Database Configuration
----------------------

The database connection is controlled by the following settings:

Database Type
~~~~~~~~~~~~~

Set ``database_type`` to one of: ``sqlite``, ``mysql``, or ``postgresql``.

.. code-block:: bash

   export CCAT_OPS_DB_DATABASE_TYPE="postgresql"

PostgreSQL Settings
~~~~~~~~~~~~~~~~~~~

When using PostgreSQL, configure these settings:

* ``database_postgresql_host`` - Database server hostname or IP
* ``database_postgresql_port`` - Database server port (default: 5432)
* ``database_postgresql_user`` - Database username
* ``database_postgresql_password`` - Database password
* ``database_postgresql_database`` - Database name

Example:

.. code-block:: bash

   export CCAT_OPS_DB_DATABASE_TYPE="postgresql"
   export CCAT_OPS_DB_DATABASE_POSTGRESQL_HOST="localhost"
   export CCAT_OPS_DB_DATABASE_POSTGRESQL_PORT="5432"
   export CCAT_OPS_DB_DATABASE_POSTGRESQL_USER="ccat"
   export CCAT_OPS_DB_DATABASE_POSTGRESQL_PASSWORD="secret"
   export CCAT_OPS_DB_DATABASE_POSTGRESQL_DATABASE="ccat_ops_db"

SQLite Settings
~~~~~~~~~~~~~~~

When using SQLite, configure:

* ``database_sqlite_database`` - Path to the SQLite database file

Example:

.. code-block:: bash

   export CCAT_OPS_DB_DATABASE_TYPE="sqlite"
   export CCAT_OPS_DB_DATABASE_SQLITE_DATABASE="/tmp/my_database.sqlite"

MySQL Settings
~~~~~~~~~~~~~~

When using MySQL, configure:

* ``database_mysql_host`` - Database server hostname or IP
* ``database_mysql_port`` - Database server port (default: 3306)
* ``database_mysql_user`` - Database username
* ``database_mysql_password`` - Database password
* ``database_mysql_database`` - Database name

Example:

.. code-block:: bash

   export CCAT_OPS_DB_DATABASE_TYPE="mysql"
   export CCAT_OPS_DB_DATABASE_MYSQL_HOST="localhost"
   export CCAT_OPS_DB_DATABASE_MYSQL_PORT="3306"
   export CCAT_OPS_DB_DATABASE_MYSQL_USER="ccat"
   export CCAT_OPS_DB_DATABASE_MYSQL_PASSWORD="secret"
   export CCAT_OPS_DB_DATABASE_MYSQL_DATABASE="ccat_ops_db"

Practical Examples
------------------

Local Development with SQLite
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

For quick local development, use the default SQLite configuration:

.. code-block:: bash

   # No configuration needed - uses defaults from settings.toml
   python -c "from ccat_ops_db import init_ccat_ops_db; session, engine = init_ccat_ops_db()"

Or explicitly set the environment:

.. code-block:: bash

   export ENV_FOR_DYNACONF=default
   python your_script.py

Local Development with PostgreSQL
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

To use PostgreSQL locally:

.. code-block:: bash

   export ENV_FOR_DYNACONF=local
   # Or override specific settings:
   export CCAT_OPS_DB_DATABASE_TYPE="postgresql"
   export CCAT_OPS_DB_DATABASE_POSTGRESQL_HOST="localhost"
   export CCAT_OPS_DB_DATABASE_POSTGRESQL_USER="ccat"
   export CCAT_OPS_DB_DATABASE_POSTGRESQL_PASSWORD="mypassword"
   export CCAT_OPS_DB_DATABASE_POSTGRESQL_DATABASE="ccat_ops_db"

   python your_script.py

Connecting to a Remote Database
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Override the host and credentials for a remote database:

.. code-block:: bash

   export ENV_FOR_DYNACONF=production
   export CCAT_OPS_DB_DATABASE_POSTGRESQL_HOST="db.example.com"
   export CCAT_OPS_DB_DATABASE_POSTGRESQL_USER="remote_user"
   export CCAT_OPS_DB_DATABASE_POSTGRESQL_PASSWORD="remote_password"
   export CCAT_OPS_DB_DATABASE_POSTGRESQL_PORT="5432"

   python your_script.py

Using a .env File
~~~~~~~~~~~~~~~~~

Create a ``.env`` file in your project root (DynaConf will automatically load it):

.. code-block:: bash

   # .env file
   ENV_FOR_DYNACONF=development
   CCAT_OPS_DB_DATABASE_POSTGRESQL_HOST=localhost
   CCAT_OPS_DB_DATABASE_POSTGRESQL_USER=ccat
   CCAT_OPS_DB_DATABASE_POSTGRESQL_PASSWORD=secret
   CCAT_OPS_DB_DATABASE_POSTGRESQL_DATABASE=ccat_ops_db

Then run your script:

.. code-block:: bash

   python your_script.py

Switching Environments in a Script
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

You can also override settings programmatically in Python:

.. code-block:: python

   import os
   from ccat_ops_db import init_ccat_ops_db

   # Set environment before importing settings
   os.environ['ENV_FOR_DYNACONF'] = 'development'
   os.environ['CCAT_OPS_DB_DATABASE_POSTGRESQL_HOST'] = 'custom-host'

   # Now initialize - will use the overridden settings
   session, engine = init_ccat_ops_db()

Docker Compose Environment
~~~~~~~~~~~~~~~~~~~~~~~~~~

For Docker Compose setups:

.. code-block:: bash

   export ENV_FOR_DYNACONF=docker-compose
   python your_script.py

This uses the service names from your ``docker-compose.yml`` (e.g., ``postgres``, ``redis``).

How Settings Are Resolved
-------------------------

DynaConf resolves settings in this order (highest priority first):

1. **Environment variables** (``CCAT_OPS_DB_*``) - Highest priority
2. **Environment group** from ``settings.toml`` (e.g., ``[development]``)
3. **Default values** from ``[default]`` section in ``settings.toml``

This means:

* Environment variables always override file-based settings
* Environment groups override default settings
* Default settings are used as fallback

Example resolution:

.. code-block:: bash

   # settings.toml has:
   # [default]
   # database_postgresql_host = "localhost"
   #
   # [development]
   # database_postgresql_host = "postgres"

   # If ENV_FOR_DYNACONF=development and no env vars:
   # → Uses "postgres" from [development]

   # If you set:
   export CCAT_OPS_DB_DATABASE_POSTGRESQL_HOST="custom-host"
   # → Uses "custom-host" regardless of environment

Checking Current Configuration
------------------------------

To see what configuration is active, you can inspect the settings object:

.. code-block:: python

   from ccat_ops_db.config.config import ccat_ops_db_settings

   print(f"Database type: {ccat_ops_db_settings.DATABASE_TYPE}")
   print(f"PostgreSQL host: {ccat_ops_db_settings.DATABASE_POSTGRESQL_HOST}")
   print(f"PostgreSQL user: {ccat_ops_db_settings.DATABASE_POSTGRESQL_USER}")
   print(f"PostgreSQL database: {ccat_ops_db_settings.DATABASE_POSTGRESQL_DATABASE}")

   # Check current environment
   print(f"Current environment: {ccat_ops_db_settings.current_env}")

Common Configuration Patterns
-----------------------------

Development Workflow
~~~~~~~~~~~~~~~~~~~~

For development, create a ``.env`` file with your local settings:

.. code-block:: bash

   ENV_FOR_DYNACONF=local
   CCAT_OPS_DB_DATABASE_POSTGRESQL_HOST=localhost
   CCAT_OPS_DB_DATABASE_POSTGRESQL_USER=dev_user
   CCAT_OPS_DB_DATABASE_POSTGRESQL_PASSWORD=dev_password

Production Deployment
~~~~~~~~~~~~~~~~~~~~~

For production, use environment variables set by your deployment system:

.. code-block:: bash

   export ENV_FOR_DYNACONF=production
   export CCAT_OPS_DB_DATABASE_POSTGRESQL_HOST="${DB_HOST}"
   export CCAT_OPS_DB_DATABASE_POSTGRESQL_USER="${DB_USER}"
   export CCAT_OPS_DB_DATABASE_POSTGRESQL_PASSWORD="${DB_PASSWORD}"

Testing Different Configurations
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Switch between configurations easily:

.. code-block:: bash

   # Test with SQLite
   ENV_FOR_DYNACONF=default python test_script.py

   # Test with local PostgreSQL
   ENV_FOR_DYNACONF=local python test_script.py

   # Test with staging
   ENV_FOR_DYNACONF=staging-mekleth python test_script.py

Troubleshooting
---------------

Configuration Not Taking Effect
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

If your configuration changes aren't being applied:

1. **Check environment variable names**: They must be prefixed with ``CCAT_OPS_DB_`` and use uppercase
2. **Verify ENV_FOR_DYNACONF**: Make sure it's set to the correct environment name
3. **Check for typos**: Setting names are case-sensitive
4. **Restart your Python process**: Settings are loaded at import time

Example debug session:

.. code-block:: bash

   # Check what environment is active
   python -c "from ccat_ops_db.config.config import ccat_ops_db_settings; print(ccat_ops_db_settings.current_env)"

   # Check a specific setting
   python -c "from ccat_ops_db.config.config import ccat_ops_db_settings; print(ccat_ops_db_settings.DATABASE_POSTGRESQL_HOST)"

Connection Errors
~~~~~~~~~~~~~~~~~

If you're getting database connection errors:

1. **Verify credentials**: Check username, password, and database name
2. **Check network access**: Ensure the host is reachable
3. **Verify port**: Default PostgreSQL port is 5432
4. **Check database exists**: The database must exist before connecting

Example connection test:

.. code-block:: python

   from ccat_ops_db import init_ccat_ops_db
   from ccat_ops_db.config.config import ccat_ops_db_settings

   # Print configuration
   print(f"Connecting to: {ccat_ops_db_settings.DATABASE_POSTGRESQL_HOST}:{ccat_ops_db_settings.DATABASE_POSTGRESQL_PORT}")
   print(f"Database: {ccat_ops_db_settings.DATABASE_POSTGRESQL_DATABASE}")
   print(f"User: {ccat_ops_db_settings.DATABASE_POSTGRESQL_USER}")

   # Try to connect
   try:
       session, engine = init_ccat_ops_db()
       print("Connection successful!")
   except Exception as e:
       print(f"Connection failed: {e}")

Related Documentation
---------------------

* :doc:`../api_reference/core` - Database initialization functions
* :doc:`../concepts/overview` - Database concepts and design
* `DynaConf Documentation <https://www.dynaconf.com/>`_ - Complete DynaConf reference