Order Reconciliation Solution Architecture

Order Expiry Reconcile Architecture

Scope: Periodic reconciliation for DAY orders at final session end and GTD orders whose expire_date resolves to a contract trading session, when MXV did not deliver a final execution report through the real-time FIX/Kafka pipeline.

1. Context

xpf_exchange_core currently updates order state mainly from MXV FIX messages consumed by order_service:

• internal/service/order/adapter/driving/kafka/mxv_execution_report_consumer.go
• internal/service/order/application/usecase/handle_execution_report.go
• internal/service/order/application/usecase/handle_trade_capture_report.go

That path is correct for real-time processing, but it has one operational gap: if MXV does not publish the expected execution report, a DAY or GTD order can remain in a non-terminal state after the session where it should expire has ended.

The reconcile flow closes that gap by running after the last session of a contract trading date, querying MXV's GET /api/v1/order/list, and applying safe final state corrections to candidate internal orders. For DAY orders, the job trading date is the session date. For GTD orders, the same job also reconciles orders whose expire_date equals that trading date.

2. Goals

• Detect contract/trading-date pairs whose final session has ended plus a grace window.
• Enqueue a single idempotent Asynq reconcile task per {exchange, contract, trading_date}.
• Query MXV only when there are internal DAY or GTD candidate orders.
• Fetch all pages from GET /api/v1/order/list before applying updates.
• Apply state changes inside database transactions.
• Give priority to real-time execution reports when they race with reconcile.
• Keep the implementation aligned with the existing order_service clean architecture and Asynq worker patterns.

3. Non-Goals

• Replacing the existing execution-report consumer.
• Reconstructing fills, deals, or positions from a list endpoint unless MXV confirms the list response carries sufficient fill data.
• Handling TIF orders other than DAY and GTD in the first release.
• Guessing a previous or next trading session when a GTD expire_date cannot resolve to a valid contract session; such orders should be diagnostic/manual-review cases.
• Adding a distributed lock service.
• Moving legacy worker code under internal/worker.

4. Current order_service Architecture Mapping

The reconcile worker should be service-owned by order_service, not a legacy worker. It should extend the current clean-architecture module rather than create a parallel worker under internal/worker.

internal/service/order/
  domain/
    entity/
    event/
    valueobject/
    rule/
  application/
    port/
    usecase/
    dto/          # new for reconcile request/result shapes if needed
  adapter/
    driven/
      external/
      persistence/
    driving/
      kafka/      # existing Kafka command and execution-report runners
      worker/     # new Asynq reconcile driving adapter
  module.go

Current code already follows this dependency direction:

• domain owns pure order state and value objects. Relevant existing code includes entity.Order, valueobject.OrderStatus, and valueobject.TimeInForce. Reconcile should reuse domain methods such as MarkExpired, MarkCanceled, and UnfilledMarginAmount where their preconditions match.
• application/port owns input/output contracts. Existing reusable output ports include Transactor, OrderRepository, MarginPort, OrderEventLogRepository, EventPublisher, and NotificationPublisher.
• application/usecase owns workflow orchestration and transaction boundaries. Reconcile should add dedicated scheduler and worker use cases here.
• adapter/driven/persistence owns pgx/sqlc mechanics and transaction reuse through queriesFromContext. Reconcile-specific job, trading-session, and candidate-lock queries should be implemented here behind application output ports.
• adapter/driven/external owns MXV HTTP clients. GET /api/v1/order/list should be wrapped by a new dedicated client/port instead of extending FIXGateway, which currently models outbound FIX order commands.
• adapter/driving/kafka currently hosts order command and MXV execution-report runners. Reconcile's Asynq scheduler/server/handler should be a new driving adapter under adapter/driving/worker.
• module.go remains the composition root and should wire new ports, use cases, worker scheduler/server, and lifecycle hooks.

Existing Code Reuse And New Code Boundary

Area	Existing code to reuse	New reconcile code needed
Domain state	Order aggregate, OrderStatus, TimeInForce, margin amount helpers, terminal transition methods.	Pure eligibility helpers only if the logic is reusable and does not require DB/MXV/config.
Application ports	Transactor, OrderRepository, MarginPort, OrderEventLogRepository, event and notification publishers.	OrderReconcileJobRepository, TradingSessionRepository, OrderCandidateRepository, MXVOrderListClient, OrderReconcileTaskQueue.
Use cases	Existing execution-report use cases define the real-time source-of-truth behavior and side-effect policy.	EnqueueDueOrderReconcileJobsUseCase and RunOrderExpiryReconcileUseCase.
Persistence	Existing pgx/sqlc repositories and transaction context pattern.	Job ledger queries, trading-session resolution queries, candidate list query, and per-order SELECT ... FOR UPDATE reload/recheck query.
External MXV	Existing MXV auth/request conventions in adapter/driven/external.	Order-list client for GET /api/v1/order/list, pagination, timeout, and response mapping.
Driving adapters	Existing Kafka runners show lifecycle and handler style. Payment/notification workers show Asynq patterns.	Asynq task types, task queue, scheduler, server, handler, Redis option helper, lifecycle registration.
Fx wiring	Current order.Module() provides repositories, external adapters, event publishers, use cases, application services, and runners.	Add new reconcile providers and invokes without importing Fx into domain/application packages.

Do not implement reconcile by fabricating an ExecutionReport and routing it through HandleExecutionReportUseCase. The execution-report flow remains the real-time authoritative path, but reconcile has different inputs, candidate selection, MXV-list safety checks, job status accounting, and retry behavior. Reuse the same domain transition methods and side-effect policy, but keep reconcile orchestration in its own use case.

5. Proposed Components

flowchart LR
    subgraph Core["order_service"]
        SCH[OrderReconcileScheduler]
        Q[OrderReconcileTaskQueue]
        W[OrderReconcileWorker]
        UC1[EnqueueDueOrderReconcileJobsUseCase]
        UC2[RunOrderExpiryReconcileUseCase]
        R1[TradingSessionRepository]
        R2[OrderReconcileJobRepository]
        R3[OrderCandidateRepository]
        MXVC[MXVOrderListClient]
    end

    subgraph Infra["Infrastructure"]
        DB[(PostgreSQL)]
        Redis[(Redis / Asynq)]
        MXV[(MXV API<br/>GET /api/v1/order/list)]
    end

    SCH --> UC1
    UC1 --> R1
    UC1 --> R2
    UC1 --> Q
    Q --> Redis
    Redis --> W
    W --> UC2
    UC2 --> R2
    UC2 --> R3
    UC2 --> MXVC
    R1 --> DB
    R2 --> DB
    R3 --> DB
    MXVC --> MXV

Component Responsibilities

Component	Responsibility
OrderReconcileScheduler	Ticks every 30-60 seconds and asks the scheduler use case to enqueue due jobs.
OrderReconcileTaskQueue	Encodes and enqueues Asynq tasks. Uses deterministic task IDs derived from job key.
OrderReconcileWorker	Decodes job payload and calls the worker use case.
TradingSessionRepository	Resolves the active trading-session profile, rules, and final session end time from the existing trading-session schema.
OrderReconcileJobRepository	Persists a minimal durable job status ledger keyed by job_key.
OrderCandidateRepository	Finds and locks candidate orders for reconciliation.
MXVOrderListClient	Calls GET /api/v1/order/list with timeout, pagination, and response mapping.
RunOrderExpiryReconcileUseCase	Applies business rules, transaction boundaries, and race-condition checks.

6. Data Model

6.1 Existing Trading Session Schema

Trading-session information already exists in internal/sqlc/schema.sql. Do not add a separate trading_calendar_sessions table.

The reconcile scheduler should read these tables:

Table	Role
trading_session_profiles	Defines an exchange/session profile, timezone, validity window, and active flag.
trading_session_rules	Defines recurring sessions by profile_id, season, day of week, session_no, start_time, end_time, and end_day_offset.
trading_session_assignments	Assigns a profile to either a specific contract_code or a commodity_code, with priority and effective date range.
contracts	Supplies contract_code, commodity_code, exchange metadata, and contract active state.

Resolution order:

1. Prefer a direct trading_session_assignments.contract_code = contracts.contract_code assignment.
2. Fall back to trading_session_assignments.commodity_code = contracts.commodity_code.
3. Among multiple valid assignments, choose the highest priority, then the latest effective_from.
4. Only use active assignments, active profiles, and active rules whose validity/effective ranges include the target trading date.
5. Compute concrete session_start_at / session_end_at in Go from trading_date, session_timezone, start_time, end_time, and end_day_offset.
6. The final session end is the maximum computed session_end_at for the selected profile/rules and trading date.

This calculation belongs in the application/usecase layer or a small domain helper, because timezone and season handling are easier to test in Go than in SQL. SQL should return profile/rule rows; Go should turn them into concrete timestamps.

6.2 Reconcile Job And Task Persistence

• PostgreSQL persists the durable reconcile job status ledger.
• Asynq/Redis remains the execution queue, retry engine, timeout owner, and short-term task metadata store.

This keeps operational status queryable even after Redis retention expires, while keeping the persistence model to one job-header table.

The Asynq task payload stays small and deterministic:

{
  "jobKey": "ORDER_EXPIRY:MXV:SIEZ26:2026-07-06",
  "exchange": "MXV",
  "contractCode": "SIEZ26",
  "tradingDate": "2026-07-06",
  "finalSessionEndAt": "2026-07-06T22:00:00+07:00"
}

The deterministic job key is the shared idempotency key between PostgreSQL and Asynq. One job covers both DAY orders for the trading date and GTD orders whose expire_date is that same trading date:

ORDER_EXPIRY:{exchange}:{contract}:{trading_date}

Example:

ORDER_EXPIRY:MXV:SIEZ26:2026-07-06

Use Asynq for execution control:

• asynq.TaskID("order-expiry-reconcile:" + jobKey) gives each contract/date task a deterministic Redis identity.
• asynq.Unique(...) prevents duplicate enqueue attempts during the configured uniqueness window.
• asynq.Retention(...) keeps completed task metadata in Redis long enough for short operational inspection and duplicate suppression while retained.
• asynq.MaxRetry(4) enforces the max retry count.
• asynq.Timeout(...) bounds the slow MXV pagination call.
• A custom RetryDelayFunc should implement the required delays: 1m, 5m, 15m, 30m.

6.3 Minimal Job Status Table

CREATE TABLE IF NOT EXISTS order_reconcile_jobs (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    job_key VARCHAR(160) NOT NULL UNIQUE,
    exchange VARCHAR(32) NOT NULL,
    contract_code VARCHAR(50) NOT NULL,
    trading_date DATE NOT NULL,
    final_session_end_at TIMESTAMPTZ NOT NULL,
    status VARCHAR(24) NOT NULL,
    asynq_task_id VARCHAR(128),
    attempt_count INTEGER NOT NULL DEFAULT 0,
    candidate_count INTEGER NOT NULL DEFAULT 0,
    mxv_total_rows INTEGER NOT NULL DEFAULT 0,
    applied_count INTEGER NOT NULL DEFAULT 0,
    skipped_count INTEGER NOT NULL DEFAULT 0,
    last_error TEXT,
    enqueued_at TIMESTAMPTZ,
    started_at TIMESTAMPTZ,
    finished_at TIMESTAMPTZ,
    created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    CONSTRAINT chk_order_reconcile_jobs_status CHECK (
        status IN ('PENDING', 'ENQUEUED', 'RUNNING', 'SUCCEEDED', 'FAILED', 'SKIPPED')
    )
);

Field reference:

Field	Meaning
id	Internal primary key for the job row. Business logic should use job_key for idempotency.
job_key	Unique business key in the format ORDER_EXPIRY:{exchange}:{contract}:{trading_date}. Shared with Asynq task identity.
exchange	Exchange code, for example MXV.
contract_code	Contract/symbol being reconciled.
trading_date	Trading date resolved by the scheduler for this contract.
final_session_end_at	Concrete timestamp of the final session end for the contract/trading date.
status	Current durable job status. Allowed values are PENDING, ENQUEUED, RUNNING, SUCCEEDED, FAILED, SKIPPED.
asynq_task_id	Asynq task id returned after enqueue. Used for operator traceability, not as the business idempotency key.
attempt_count	Number of worker attempts that reached RUNNING. Incremented when the worker starts processing.
candidate_count	Number of internal candidate orders found before calling MXV.
mxv_total_rows	pagination.totalRows returned by MXV after pagination.
applied_count	Number of candidate orders updated by reconcile.
skipped_count	Number of candidate orders intentionally skipped.
last_error	Latest enqueue, MXV, DB, or retryable processing error for operator diagnosis.
enqueued_at	Time the task was accepted by Asynq.
started_at	Time the first worker attempt started.
finished_at	Time the job reached SUCCEEDED, SKIPPED, or FAILED.
created_at	Row creation timestamp.
updated_at	Last job status/counter update timestamp.

Status semantics:

Status	Meaning
PENDING	DB row exists, but the task has not been accepted by Asynq yet or enqueue failed and should be retried by scheduler.
ENQUEUED	Asynq accepted the task, or a retry is waiting in Asynq after a retryable failure.
RUNNING	A worker started an attempt.
SUCCEEDED	Worker completed candidate scan, MXV pagination, and all safe updates.
SKIPPED	Worker found no internal candidate orders, so it intentionally did not call MXV.
FAILED	Retry budget is exhausted or the job hit a non-recoverable infrastructure/setup error.

State machine:

stateDiagram-v2
    [*] --> PENDING: Scheduler inserts due job
    PENDING --> ENQUEUED: Asynq enqueue accepted
    PENDING --> PENDING: Enqueue failed; scheduler retries later
    ENQUEUED --> RUNNING: Worker starts attempt
    RUNNING --> SKIPPED: No internal candidates
    RUNNING --> SUCCEEDED: Pagination and safe updates completed
    RUNNING --> ENQUEUED: Retryable failure; retries remain
    RUNNING --> FAILED: Final retry exhausted or non-recoverable error
    SKIPPED --> [*]
    SUCCEEDED --> [*]
    FAILED --> [*]

Lifecycle notes:

1. Scheduler computes a due target and inserts PENDING with INSERT ... ON CONFLICT DO NOTHING.
2. If the row already exists, scheduler loads it by job_key.
3. Scheduler enqueues Asynq only when the row is newly inserted or currently PENDING.
4. Existing ENQUEUED, RUNNING, SUCCEEDED, SKIPPED, or FAILED rows are not auto-enqueued
5. If enqueue succeeds, scheduler updates ENQUEUED, asynq_task_id, and enqueued_at.
6. If enqueue fails, keep or return the row to PENDING with last_error; the next scheduler tick can retry enqueue.
7. Worker start updates RUNNING, increments attempt_count, and sets started_at.
8. No candidate orders updates SKIPPED, sets candidate_count=0, and returns nil.
9. Successful processing updates SUCCEEDED, counters, and finished_at.
10. Retryable failure records last_error and returns status to ENQUEUED before returning an error to Asynq.
11. When retry budget is exhausted, update FAILED, last_error, and finished_at.

The DB job row is a status ledger, not the concurrency lock for order state. Per-order idempotency remains protected by row-level transaction checks and terminal-state guards.

7. Candidate Filter

The worker calls MXV only when the internal DB has at least one candidate:

symbol = contract_code
ord_status IN (
  NEW,
  PARTIALLY_FILLED
)
AND (
  -- DAY: FIX tag 59 Day, matching current persisted code.
  (
    time_in_force = '0'
    AND COALESCE(accepted_at, placed_at) within the resolved trading-date session window
  )

  OR

  -- GTD: FIX tag 59 Good Till Date.
  (
    time_in_force = '6'
    AND expire_date = trading_date
    AND COALESCE(accepted_at, placed_at) < candidate_window_end_at
  )
)

To avoid repeatedly reconciling old stuck DAY orders for every later trading date of the same contract, derive a candidate window from the selected trading-session profile:

1. Compute the concrete first session_start_at and final session_end_at for the job's trading_date in session_timezone.
2. Convert that window to timestamps comparable with orders.placed_at and orders.accepted_at.
3. Include only orders whose COALESCE(accepted_at, placed_at) falls within the concrete trading-date session window, optionally widened by a small configured lookback/lookahead tolerance for clock skew and late acceptance writes.

GTD uses the same contract/date job, but its eligibility is anchored by expire_date = trading_date, not by placement inside that trading-date session. A valid GTD order may have been accepted days earlier, so the GTD branch must not apply the DAY lower-bound window. It should still require COALESCE(accepted_at, placed_at) < candidate_window_end_at so an order accepted after the expiry session window is not reconciled by that expired-date job.

If a GTD expire_date cannot be resolved to a concrete final session for the contract/profile, do not guess by moving to a previous or next session. Skip enqueue or skip the order with a diagnostic reason so product/operations can correct the session data or order data.

If upstream terminology uses WORKING, it maps to internal NEW. The current domain has NEW, not WORKING.

Implementation note: the intended candidate SQL uses FIX tag-59 codes ('0' for DAY and '6' for GTD), matching the orders.time_in_force CHAR(1) schema and existing enum code helpers. Before implementing this query, verify the current persistence adapter writes the same representation. If the adapter writes display names such as DAY/GTD, align persistence first or the candidate query will silently miss orders.

PENDING_CANCEL and PENDING_REPLACE are intentionally excluded. They are transient command-in-flight states and are mirrored to the command lifecycle as PROCESSING; expiry reconcile must not override them while cancel/replace handling is still owned by the command runner and execution-report flow.

PENDING_NEW is intentionally excluded unless product confirms that a DAY or GTD order can remain pending-new past its expiry session and still require expiry reconciliation. Excluding it is safer because a missing initial MXV ack may mean the order was never accepted by MXV.

8. MXV API Usage

Endpoint:

GET /api/v1/order/list

Parameters:

Param	Value
sessionDate[gte]	job trading date
sessionDate[lte]	job trading date
status[in]	terminal and active statuses needed for reconciliation
limit	configured page size
offset	page offset

The API has high latency, roughly 8-10 seconds per request. Client timeout should be 30 seconds. Worker concurrency should be 1 or 2 per process.

Pagination rule:

1. Fetch first page with offset=0.
2. Read pagination.totalPages.
3. Fetch every remaining page before applying updates.
4. If totalPages=0 or data=[], return success with a no_mxv_match summary.

9. Status Resolution Policy

MXV execution reports remain the authoritative real-time path. Reconcile is a delayed repair mechanism.

Recommended policy:

MXV list status	Internal candidate	Action
EXPIRED	NEW, PARTIALLY_FILLED	Mark expired and release unfilled margin only when MXV fill fields do not conflict the internal fill state.
CANCELED	NEW, PARTIALLY_FILLED	Mark canceled and release unfilled margin only when MXV fill fields do not conflict the internal fill state.
REJECTED	NEW or equivalent pre-trade state	Mark rejected and release locked margin.
PARTIALLY_FILLED	NEW, PARTIALLY_FILLED	Skip unless response has enough fill data to update quantity, margin, and downstream asset state correctly.
FILLED	NEW, PARTIALLY_FILLED	Skip/manual-review unless response has complete fill economics required by the fill state machine.
Unknown	NEW, PARTIALLY_FILLED	Skip and record diagnostic reason.

Before applying EXPIRED or CANCELED, compare MXV list quantity fields such as filled/cumulative quantity and leaves quantity against the internal order. Apply the terminal correction only when the MXV values are absent or consistent with internal cum_qty and leaves_qty. If MXV indicates an unseen fill, a lower leaves quantity, or any quantity split that would require fill economics the list endpoint does not carry, skip the order and record a manual-review diagnostic instead of releasing margin.

Rationale: order_service already has carefully tested fill logic and margin side effects. A list endpoint should not create fill/deal state unless it carries the same business data as execution report/trade capture events.

10. Workflow

sequenceDiagram
    autonumber
    participant SCH as Order Reconcile Scheduler
    participant DB as PostgreSQL
    participant Q as Asynq Queue
    participant W as Reconcile Worker
    participant MXV as MXV GET /api/v1/order/list
    participant UC as Order Reconcile UseCase
    participant ER as Execution Report Consumer

    SCH->>DB: Find due final_session_end_at + grace <= now
    DB-->>SCH: Due exchange/contract/trading_date rows

    loop Each due contract/date
        SCH->>SCH: Build jobKey + task payload
        SCH->>DB: Insert or load order_reconcile_jobs by jobKey
        alt New or PENDING job
            SCH->>Q: Enqueue task with TaskID/Unique(jobKey)
            alt Task accepted
                Q-->>SCH: task id
                SCH->>DB: Mark job ENQUEUED + asynq_task_id
            else Duplicate task still pending/active/retrying
                Q-->>SCH: duplicate error
                SCH-->>SCH: Skip duplicate
            else Enqueue failed
                Q-->>SCH: enqueue error
                SCH->>DB: Keep job PENDING + last_error
            end
        else Existing terminal/active/failed job
            SCH-->>SCH: Skip enqueue
        end
    end

    Q->>W: Process task payload
    W->>UC: RunOrderExpiryReconcile(payload)
    UC->>DB: Mark job RUNNING + increment attempt_count
    UC->>DB: Load candidate orders

    alt No candidate orders
        UC->>DB: Mark job SKIPPED(candidate_count=0)
        UC-->>W: return nil(no_candidate)
    else Has candidates
        UC->>MXV: GET first page
        MXV-->>UC: data + totalPages + totalRows
        loop Remaining pages
            UC->>MXV: GET next page
            MXV-->>UC: data
        end

        UC->>UC: Match MXV rows to candidates

        loop Each matched candidate
            par Real-time execution report may race
                ER->>DB: Tx: save execution_report + update order + audit
            and Reconcile transaction
                UC->>DB: Tx: SELECT order FOR UPDATE
                UC->>DB: Recheck TIF-specific expiry eligibility, status, updated_at, and latest state
                alt Still eligible
                    UC->>DB: Update order, margin, and order_event_logs
                    DB-->>UC: Commit
                else Already handled or no longer eligible
                    UC->>DB: Record skip reason
                    DB-->>UC: Commit skip
                end
            end
        end

        UC->>DB: Mark job SUCCEEDED + counters
        UC-->>W: return nil(summary logged)
    end

    alt MXV timeout / 5xx / transient DB error
        UC->>DB: Record last_error and keep job ENQUEUED for Asynq retry
        UC-->>W: return error
        W-->>Q: Asynq retries with custom delay
    else retries exhausted
        UC->>DB: Mark job FAILED + last_error + finished_at
        Q-->>Q: Archive task in Redis
    end

11. Race Condition Strategy

No distributed lock is required for correctness if the database transaction is the concurrency boundary.

Per order:

1. Reconcile loads candidate IDs outside the update transaction.
2. For each matched order, reconcile opens a transaction.
3. It reloads the order with row-level locking.
4. It rechecks TIF-specific expiry eligibility and current status.
5. If the execution-report consumer already updated the order, reconcile skips.
6. If the order is still eligible, reconcile applies a terminal transition and audit log in the same transaction.

In code, the transaction boundary should be owned by RunOrderExpiryReconcileUseCase via the existing port.Transactor.WithinTx pattern. The pgx/sqlc mechanics stay in adapter/driven/persistence, where repositories reuse the transaction from context through queriesFromContext.

The current OrderRepository has ordinary FindBy* and Update methods, but no row-locking read. Reconcile therefore needs a new output-port method or dedicated candidate repository method for SELECT ... FOR UPDATE reloads. Do not put row-lock SQL in the use case and do not let the persistence adapter decide the cross-repository business sequence.

This handles the true concurrent case. If a late execution report arrives after reconcile has already committed, the existing execution-report state machine decides whether the late report is a no-op or an invalid transition. Operators should alert on conflicts where a late fill contradicts an expiry/cancel reconcile result.

Important code note: OrderStatus.IsFinal() currently does not include CANCELED, and this reconcile design does not require changing it. Reconcile should use an explicit eligibility check for NEW and PARTIALLY_FILLED candidates plus per-order row-lock rechecks, so it does not depend on the broader execution-report terminal guard semantics.

12. Retry And Backoff

Retry policy:

Attempt after failure	Delay
1	1 minute
2	5 minutes
3	15 minutes
4	30 minutes
More than 4	Asynq archives task in Redis

Let Asynq own retry scheduling and also records durable job status in PostgreSQL. Configure the order reconcile Asynq server with a custom RetryDelayFunc so retry timing matches the required schedule. Handler behavior:

• Return nil for success, no candidates, and non-retryable skip decisions.
• For a retryable failure, update last_error, keep the job ENQUEUED, and return an error so Asynq retries.
• When retry budget is exhausted, update the job to FAILED, then let Asynq archive the task in Redis.
• Use asynq.GetRetryCount(ctx) and asynq.GetMaxRetry(ctx) to decide whether the current failure is the final retry. If retry metadata is unavailable, the server ErrorHandler is the fallback place to mark FAILED.

13. Operational Controls

Runtime controls should be explicit under OrderConfig and loaded from environment overrides with safe defaults. These environment variables are supported overrides, not all required deployment variables:

ORDER_RECONCILE_ENABLED=false
ORDER_RECONCILE_SCHEDULER_INTERVAL_SECONDS=60
ORDER_RECONCILE_GRACE_SECONDS=300
ORDER_RECONCILE_ASYNC_QUEUE_NAME=order-reconcile
ORDER_RECONCILE_ASYNC_CONCURRENCY=1
ORDER_RECONCILE_TASK_TIMEOUT_SECONDS=900
ORDER_RECONCILE_TASK_RETENTION_HOURS=168
ORDER_RECONCILE_TASK_UNIQUE_TTL_HOURS=168
ORDER_RECONCILE_MXV_TIMEOUT_SECONDS=30
ORDER_RECONCILE_MXV_PAGE_LIMIT=100
ORDER_RECONCILE_MAX_RETRY=4
ORDER_RECONCILE_CANDIDATE_WINDOW_TOLERANCE_SECONDS=300

Only dependency configuration is required when ORDER_RECONCILE_ENABLED=true:

• REDIS_URL
• ORDER_MXV_BASE_URL
• either ORDER_MXV_API_KEY or ORDER_MXV_APP_ID + ORDER_MXV_JWT_SECRET

All reconcile tuning values above should have defaults in OrderConfig; validation should reject invalid override values such as empty queue name, non-positive intervals/timeouts/concurrency/page limit, or negative retry/tolerance. A deployment can therefore enable reconcile by setting ORDER_RECONCILE_ENABLED=true plus the dependency credentials, and only override tuning values when the default is inappropriate.

If the service runs more than one worker replica, ORDER_RECONCILE_ASYNC_CONCURRENCY=1 only limits each process. To enforce a global MXV call limit of 1-2 in, run a single reconcile worker replica or add a Redis-backed global limiter.

These settings are new OrderConfig fields. Current order config covers command polling/stale processing, MXV command API credentials/timeouts, and notification topic only. Implementation should add config loading, validation, and .env.sample entries together with the worker wiring.

Keep the first implementation static and process-level. Trading sessions, contract/profile assignments, and contract scope remain DB-backed business data. If operators later need runtime toggles per exchange/contract without deployment, add an application output port such as OrderReconcileControlStore; the use case should read through that port, and driving adapters must not query persistence directly.

14. Observability

Structured logs should include:

• job_key
• exchange
• contract_code
• trading_date
• attempt_count
• candidate_count
• mxv_total_rows
• mxv_pages
• applied_count
• skipped_count
• duration_ms
• error
• asynq_task_id
• job_status

Suggested metrics:

• order_reconcile_jobs_total{status}
• order_reconcile_tasks_total{result}
• order_reconcile_candidates_total
• order_reconcile_applied_total{action}
• order_reconcile_skipped_total{reason}
• order_reconcile_mxv_request_duration_seconds
• order_reconcile_mxv_request_errors_total{class}

15. Key Risks

Risk	Mitigation
MXV API latency causes long jobs	Low concurrency, page limit, 30s timeout, task timeout >= worst-case pages.
Duplicate scheduler instances	UNIQUE(job_key) in DB plus deterministic Asynq TaskID or Unique option.
Execution report and reconcile race	Per-order transaction with row lock and recheck.
Late fill after reconcile expiry	Skip fill synthesis from list endpoint; alert on contradictory late execution report.
MXV list terminal status hides unseen fill quantity	Apply EXPIRED/CANCELED only when MXV filled/cum/leaves fields are absent or consistent with internal cum_qty/leaves_qty; otherwise skip/manual-review.
Old stuck DAY orders repeatedly selected for later trading dates	Restrict DAY candidates by COALESCE(accepted_at, placed_at) inside the resolved trading-date session window, with only a small configured tolerance.
GTD orders placed before the expiry date are missed by the DAY window	Select GTD candidates by expire_date = trading_date and use only the upper-bound COALESCE(accepted_at, placed_at) < candidate_window_end_at.
GTD expire_date cannot resolve to a contract trading session	Do not guess previous/next sessions; skip enqueue or skip the order with a diagnostic/manual-review reason.
Session profile/rule data is wrong	Audit profile/rule changes and expose a due-job preview query that shows resolved final session end time before enqueue.
Redis task metadata expires	DB job status remains durable; Asynq metadata is only execution metadata.
DB job status and Asynq state diverge	Treat DB as the operator ledger and Asynq as execution state. Reconcile stale PENDING/ENQUEUED/RUNNING rows by age-based operational checks.
Global rate limit exceeded with multiple replicas	Run single worker replica or add Redis-backed global limiter.