Data Flow Examples
==================

Complete data flow examples showing request-to-response patterns for different scenarios.

.. contents:: Table of Contents
   :local:
   :depth: 2

UI Read Request
---------------

Simple query flow without buffering:

.. mermaid::

   sequenceDiagram
       participant UI as Web Frontend
       participant API as FastAPI
       participant Auth as Authentication
       participant Router
       participant DB as Local Database
       
       UI->>API: GET /api/transfer/overview
       API->>Auth: Verify JWT
       Auth-->>API: User authenticated
       API->>Router: Route to handler
       Router->>DB: Query transfers
       DB-->>Router: Transfer data
       Router-->>API: Format response
       API-->>UI: JSON response (< 100ms)

Observatory Write (Buffered)
-----------------------------

Critical operation with transaction buffering:

.. mermaid::

   sequenceDiagram
       participant Script as Observatory Script
       participant API
       participant Builder as Transaction Builder
       participant Manager as Transaction Manager
       participant Redis
       participant BG as Background<br/>Processor
       participant MainDB as Main DB
       participant LSN as LSN Tracker
       participant Replica
       
       Script->>API: POST /executed_obs_units/start
       API->>Builder: Build transaction
       Builder->>Builder: Generate UUID
       Builder-->>API: Transaction
       API->>Manager: Buffer transaction
       Manager->>Redis: LPUSH to buffer
       Manager->>Redis: Cache generated ID
       Redis-->>Manager: OK
       Manager-->>API: Transaction ID
       API-->>Script: 201 Created (< 20ms)
       
       Note over Script: Client continues immediately
       
       loop Background Processing
           BG->>Redis: RPOP from buffer
           BG->>MainDB: Execute transaction
           MainDB-->>BG: Success
           BG->>MainDB: Capture LSN
           MainDB-->>BG: LSN: 0/12345678
           BG->>LSN: Track replication
           LSN->>Replica: Poll replay LSN
           alt Replicated
               Replica-->>LSN: LSN caught up
               LSN-->>BG: Replicated
               BG->>Redis: Cleanup caches
           else Not yet
               Replica-->>LSN: Still behind
               LSN-->>BG: Not replicated
               BG->>Redis: Extend cache TTL
           end
       end

Observatory Read (Smart Query)
-------------------------------

Query merging database + buffer:

.. mermaid::

   sequenceDiagram
       participant Script
       participant API
       participant Smart as Smart Query<br/>Manager
       participant DB as Local Replica
       participant Redis as Redis Buffer
       participant ReadBuf as Read Buffer
       
       Script->>API: GET /executed_obs_units/123
       API->>Smart: search_records(...)
       
       par Parallel Queries
           Smart->>DB: Query database
           Smart->>Redis: Query buffered cache
           Smart->>ReadBuf: Query read buffer
       end
       
       DB-->>Smart: [obs1, obs2]
       Redis-->>Smart: [obs3 (buffered)]
       ReadBuf-->>Smart: [updates to obs3]
       
       Smart->>Smart: Merge (buffer > DB)
       Smart->>Smart: Apply read buffer updates
       Smart->>Smart: Deduplicate by ID
       
       Smart-->>API: [obs1, obs2, obs3 (merged)]
       API-->>Script: JSON response

WebSocket Real-Time Updates
----------------------------

Streaming updates via WebSocket:

.. mermaid::

   sequenceDiagram
       participant UI
       participant API
       participant Redis as Redis<br/>Pub/Sub
       participant DB
       
       UI->>API: Connect WS /api/transfer/ws/overview
       API->>API: Authenticate token
       API->>Redis: Subscribe to "transfer_updates"
       API->>DB: Query initial data
       DB-->>API: Current overview
       API->>UI: Send initial data
       
       Note over UI,DB: Real-time updates
       
       loop On Data Changes
           DB->>Redis: PUBLISH transfer_updates
           Redis-->>API: Update message
           API->>UI: Send update
       end

Complete Observatory Operation
-------------------------------

End-to-end flow from start to finish:

1. **Start observation** (buffered)
2. **Query observation** (smart query, shows buffered)
3. **Finish observation** (update read buffer)
4. **Query again** (smart query, shows updated)
5. **Background processing** executes buffered start
6. **Background processing** executes buffered finish
7. **LSN tracking** confirms replication
8. **Cache cleanup** occurs
9. **Future queries** read from database (normal)

Summary
-------

Key patterns:

* **UI reads**: Direct database queries (fast, simple)
* **Operations writes**: Buffered (reliable, never block)
* **Operations reads**: Smart queries (merge buffer + DB)
* **Real-time**: WebSockets with Redis pub/sub

Next Steps
----------

* :doc:`transaction-buffering/overview` - Buffering details
* :doc:`../tutorials/complex-endpoints/smart-queries-with-buffering` - Tutorial