Deployment
==========

.. verified:: 2026-02-23
   :reviewer: Christof Buchbender

Overview
--------

Deployments are driven by a **causal build-state orchestrator** in
``ci/check_builds.py``, triggered via GitHub Actions
(``orchestrate-deploy.yml``) and executed by Jenkins.

Repos are divided into **deploy groups**.  Each group has its own gate
check and its own Jenkins job — a docs rebuild never blocks or is
blocked by a main-stack rebuild.

.. list-table:: Deploy groups
   :header-rows: 1
   :widths: 15 45 40

   * - Group
     - Repos
     - Jenkins jobs
   * - ``default``
     - ops-db, ops-db-api, data-transfer, ops-db-ui
     - ``deploy-staging`` / ``deploy-production``
   * - ``docs``
     - data-center-documentation
     - ``deploy-data-center-documentation-develop`` /
       ``deploy-data-center-documentation-production``


Main-stack deploy (``default`` group)
--------------------------------------

#. A developer pushes to one of the main-stack repos.
#. GitHub Actions builds and pushes the Docker image, then calls
   ``notify-build-complete.yml`` which sends a ``repository_dispatch``
   (type ``build-complete``) to ``system-integration``.
#. ``orchestrate-deploy.yml`` runs (serialised per branch via a
   concurrency group):

   a. **Update state** — records ``sha``, ``image_digest``, and
      ``built_with`` in ``BUILD_STATE_JSON_<BRANCH>``.
   b. **Cascade** — dispatches ``workflow_dispatch`` to immediate
      downstream repos so they rebuild against the new upstream image.
   c. **Cross-group dispatch** — dispatches ``update-submodules.yml``
      in ``data-center-documentation`` for each ``cross_group_trigger``
      entry in ``dependency-graph.yml``.
   d. **Gate check** — resolves the triggering repo's deploy group,
      then verifies every repo in that group is causally consistent
      (``built_with`` matches current upstream digest).  If all green,
      triggers the Jenkins job via its REST API.

#. Jenkins pipeline (``deploy_staging`` / ``deploy_production``):

   - SSHs into ``input-b.{staging.}data.ccat.uni-koeln.de``
   - Runs Alembic migrations (ops-db pipeline only)
   - ``docker compose pull + up -d`` (main stack compose file)
   - Smoke test


Docs deploy (``docs`` group)
------------------------------

The docs stack is **independent** — it has its own compose files
(``docker-compose.docs.input-b.yml`` /
``docker-compose.docs.staging.input-b.yml``) and its own Jenkins jobs.

The trigger chain:

#. Any of the four tracked upstream repos publishes a new image.
#. The orchestrator fires a cross-group ``workflow_dispatch`` to
   ``data-center-documentation``, targeting ``update-submodules.yml``.
#. ``update-submodules.yml`` bumps all tracked submodule pointers to
   their latest HEAD on the dispatched branch, then commits and pushes
   (if anything changed).
#. The push triggers ``docker-docs.yml``, which builds and pushes the
   docs image and calls ``notify-build-complete`` with the submodule
   SHAs as ``built_with_json``.
#. The orchestrator receives the ``build-complete`` event, updates
   state, and runs the gate check for the ``docs`` group.  The gate
   uses ``source_commit`` comparison: ``built_with[dep]`` is compared
   against ``state[dep].sha`` (git commit SHA of each submodule pointer)
   rather than an image digest.
#. If all four submodule SHAs match, the Jenkins docs deploy job is
   triggered.


Causal consistency model
------------------------

Build state is stored in two GitHub repository variables:
``BUILD_STATE_JSON_DEVELOP`` and ``BUILD_STATE_JSON_MAIN``.  Each
per-repo entry has the form:

.. code-block:: json

   {
     "sha":          "<git commit SHA>",
     "image_digest": "sha256:...",
     "ts":           "2026-01-01T00:00:00+00:00",
     "built_with":   { "ops-db": "sha256:..." }
   }

The ``built_with_type`` field in ``ci/dependency-graph.yml`` controls
what the gate check compares:

.. list-table::
   :header-rows: 1
   :widths: 20 80

   * - Type
     - Gate compares ``built_with[dep]`` against
   * - ``runtime``
     - ``state[dep].image_digest``
   * - ``base_image``
     - ``state[dep].image_digest``
   * - ``source_commit``
     - ``state[dep].sha``  (used by docs: submodule pointer vs upstream
       git SHA)


Local Development
-----------------

.. code-block:: bash

   git clone <repository_url>
   cd system-integration
   git submodule update --init --recursive
   cp .env.example .env
   # Edit .env — set POSTGRES_PASSWORD, REDIS_PASSWORD, etc.
   ccat update   # or: make start_main


Staging Environment
-------------------

Staging runs on ``input-b.staging.data.ccat.uni-koeln.de``.

Jenkins ``deploy-staging`` / ``deploy-data-center-documentation-develop``
handle staging deploys automatically when the gate check passes on the
``develop`` branch.

To deploy manually::

   ssh <user>@input-b.staging.data.ccat.uni-koeln.de
   cd /opt/data-center/system-integration
   docker compose -f docker-compose.staging.input-b.yml pull
   docker compose -f docker-compose.staging.input-b.yml up -d


Production Deployment
---------------------

Production runs on ``input-b.data.ccat.uni-koeln.de``.

Jenkins ``deploy-production`` / ``deploy-data-center-documentation-production``
handle production deploys automatically when the gate check passes on
the ``main`` branch.

To deploy manually::

   ssh <user>@input-b.data.ccat.uni-koeln.de
   cd /opt/data-center/system-integration
   # Main stack:
   docker compose -f docker-compose.production.input-b.yml pull
   docker compose -f docker-compose.production.input-b.yml up -d
   # Docs (independent):
   docker compose -f docker-compose.docs.input-b.yml pull
   docker compose -f docker-compose.docs.input-b.yml up -d