System architecture

CIEL Lab separates interactive review from slow ingestion and indexing work. The browser talks to a single web entry point; the API coordinates OCL, cached data, relational state, background jobs, and semantic search.

Infrastructure infographic

flowchart LR
    User["CIEL editor<br/>Web browser"] --> Nginx["Nginx<br/>single public entry point"]
    AI["AI client<br/>MCP protocol"] --> Nginx

    Nginx --> Web["React 19 + Vite<br/>Apollo client"]
    Nginx --> API["FastAPI + Strawberry<br/>REST, GraphQL, MCP"]
    Web --> API

    API --> MySQL[("MySQL<br/>users, reports, worklists,<br/>release and app state")]
    API --> Valkey[("Valkey<br/>HEAD mirror, cache,<br/>broker and task results")]
    API --> Qdrant[("Qdrant<br/>vector index")]
    API --> OCL["Open Concept Lab<br/>source of truth"]
    API --> Objects[("S3-compatible storage<br/>release artifacts")]

    Beat["Celery Beat<br/>scheduled jobs"] --> Worker["Celery workers<br/>background queues"]
    API --> Worker
    Worker --> OCL
    Worker --> MySQL
    Worker --> Valkey
    Worker --> Qdrant
    Worker --> Objects

    Embeddings["OpenAI-compatible<br/>embedding service<br/>MedEmbed-small-v0.1"] --> Qdrant
    Worker --> Embeddings
    API --> Embeddings

    Scripts["CIEL scripts<br/>load, transform, validate,<br/>export and publish"] --> TerminologyDB[("Standard terminology<br/>MySQL views")]
    Scripts --> OCL
    TerminologyDB --> Worker

    GitHub["GitHub repository<br/>Markdown + CI"] --> Pages["GitHub Pages<br/>public wiki"]
    R2[("Cloudflare R2<br/>videos and large media")] --> Pages
    User --> Pages

Request path

The React application uses Apollo for GraphQL operations and calls supporting API routes where appropriate. Nginx presents the web application, API, and MCP surfaces under one deployment. MySQL stores durable application state: accounts, reports, worklists, and release records.

API entry points

• GraphQL playground: https://ciellab.filipelopes.med.br/graphql
• REST OpenAPI documentation: https://ciellab.filipelopes.med.br/api/v2/docs

OCL proxy

The REST API includes an OCL proxy for browser-initiated OCL requests. It exists so that an OCL token is never placed in the browser bundle: the user authenticates to CIEL Lab with their session JWT, and the server retrieves that user's encrypted OCL token, adds it to the request, and forwards the request only to the configured OCL API. This preserves each user's OCL permissions, prevents arbitrary-host forwarding, and keeps the plaintext token server-side.

For CIEL concept writes, the proxy also records an audit trail and keeps the HEAD cache and vector index synchronized after a successful upstream response.

OCL is authoritative for terminology content. CIEL Lab does not make Qdrant or Valkey authoritative. It uses them as optimized projections:

• Valkey HEAD mirror supplies a fast snapshot of current CIEL HEAD content. The application can fall back to live OCL reads when the mirror is not available.
• Qdrant vector index supports semantic retrieval over CIEL and indexed external terminologies.
• MySQL terminology views expose a consistent shape for loaders such as SNOMED CT, ICD-11, and RxNorm.

A CIEL refresh currently rebuilds the indexed projection as a whole rather than performing a published incremental reindex. The full rebuild is deliberate for consistency, but it can re-embed content that did not change and must be treated as background operational work rather than an immediate editor action.

Why Valkey and Qdrant coexist

Valkey and Qdrant solve different parts of the same problem. They are both rebuildable projections of terminology data; neither replaces OCL as the source of truth.

Valkey is the application-facing HEAD mirror. It keeps structured CIEL concept data close to the API so the product can retrieve a concept, browse HEAD, apply exact filters, compare a draft with known CIEL content, and support background workflows without repeatedly asking OCL for the same assembled payloads. It is designed for fast key and set lookups, not for ranking text by meaning.

Qdrant is the search-facing index. It turns CIEL names and indexed external terminologies into lexical and semantic search candidates. This supports type-ahead name search, semantic mapping suggestions, and checks against reference terminology codes. It is optimized for finding relevant candidates, not for acting as the complete editable representation of a CIEL concept.

The normal lifecycle is therefore complementary: a HEAD refresh first rebuilds the structured Valkey mirror, then derives the Qdrant index from it. After a concept edit, Valkey keeps local reads responsive while Qdrant is refreshed so future searches reflect the change. Keeping these responsibilities separate avoids turning the search index into a document store or duplicating a full-text and vector index inside the cache.

Embeddings and semantic search

The embedding adapter speaks an OpenAI-compatible API and defaults to abhinand/MedEmbed-small-v0.1. CIEL Lab stores the resulting vectors in the shared Qdrant collection ciellab_medembed_small.

Each vector carries filterable payload metadata, including terminology system, version, code, and CIEL environment. This lets one collection support:

• semantic CIEL searches;
• exact-code lookup;
• version or terminology constraints;
• on-demand external terminology indexes.

Indexing uses stable point identifiers, batched embedding calls, and batched Qdrant upserts. Re-running a job updates existing points instead of creating duplicates.

Semantic similarity is not clinical equivalence

Vector ranking can prefer a broad lexical match over a clinically specific code. Treat results as candidates. Confirm the terminology, version, status, and intended meaning before applying a mapping. This limitation is especially important for post-coordinated or highly specific expressions.

Background jobs

Fast user interactions should not wait for a complete OCL refresh, terminology index, bulk report, or release artifact. The API therefore dispatches work to Celery through Valkey.

The worker task families include:

Task family	What it does
CIEL HEAD refresh	Pulls OCL HEAD, rebuilds the mirror, then updates related projections
Vector indexing	Embeds and indexes CIEL or an external terminology in Qdrant
Dashboard refresh	Recomputes summary metrics and trends
Staged-change refresh	Builds the HEAD-to-release change model
Bulk validation	Executes deterministic QA rules and persists a report
Worklist hydration	Resolves or enriches external terminology candidates
Translation import	Parses model output and creates a reviewable translation batch
OCL apply	Applies approved concept or mapping changes with user authorization
Release artifacts	Produces SQL and Markdown diff artifacts and can publish Liquibase Runner output
Retention	Deletes expired reports and related temporary state

Interactive OCL writes use a separate interactive worker queue so that a save, merge, or other curator action does not wait behind a long HEAD refresh, indexing pass, or bulk report. Long-running work uses the bulk queue. This separation improves responsiveness; it does not make the two task families independent of their shared OCL, database, cache, and index dependencies.

User-started jobs record a durable terminal outcome. A task failure, a revoked task, or a worker that stops without reporting is handled through the same completion path, so the user can receive a result instead of an endless spinner. For a partially applied merge, a rollback_failed outcome still requires manual OCL review before retrying.

Celery Beat schedules the regular refresh chain. Schedule interpretation uses APP_TIMEZONE (the documented default is America/Chicago); confirm the currently deployed schedule before relying on a clock time. Manual refresh actions can enqueue the same expensive work outside that schedule.

QA execution path

The QA catalog separates rules into two execution tiers. Local rules use only the draft and run in the Concept Editor's immediate save-gating path. Deferred rules need the Valkey HEAD mirror, Qdrant, or an OCL-backed reference index; the editor requests them separately and represents a pending or unavailable result rather than treating an unavailable lookup as a clean result.

Bulk Validation uses the same deterministic catalog, but evaluates it across a selected source/version and persists the report in MySQL. Its global execution configuration is stored with application state and passed into the evaluator, so a disabled rule incurs no evaluation work. The HEAD mirror includes secondary indexes for fully specified names, preferred names, and mapping codes to support the editor's cross-concept checks; a mirror built before those indexes is reported as unavailable until refreshed.

Refresh versus reload

These words are intentionally different:

• Reload reads the latest state already held by CIEL Lab.
• Refresh contacts OCL and may rebuild cache, indexes, and derived data.

A refresh can take much longer. Use it only when the source of truth changed and the scheduled update cannot be awaited.

Reliability boundaries

• OCL is the terminology source of truth.
• MySQL is the durable application record.
• Worklists are durable review checkpoints and should survive a moving HEAD snapshot.
• Valkey and Qdrant can be rebuilt.
• A cache reset can temporarily make QA or search look incomplete until the refresh/index chain finishes.
• MCP tools prepare previews; the human-facing app performs the final commit.