# Daily Metrics & Historical Snapshot Engine

The Metrics ecosystem is designed to solve a critical business reporting challenge: **The Silent Defaulter Problem**. 

## Business Challenge: The Silent Defaulter
Business stakeholders require a day-by-day record of portfolio health, including current Days Past Due (DPD), Principal Outstanding (POS), and accumulated penalties for every single loan. 
If the system only recalculates these metrics when a payment is made, loans that default and never make a payment will never have their metrics updated. Their DPD would freeze, and business reports would falsely show a healthy portfolio.

To solve this, the BlackSoil engine uses a dual-script architecture: a **Forward Engine** (Night Cron) and a **Correction Engine** (Historical Recalc).

---

## 1. The Forward Engine (Night Cron)
**Endpoint:** `/api/co_lending_daily_metrics_cron`

Every night at midnight, the cron wakes up and calculates a fresh snapshot for **every active loan** in the system, regardless of whether a payment was made. 

### Execution Logic
1. It queries `co_lending_loans` for all active IDs.
2. It executes a massive, highly-optimized SQL aggregation (`HISTORICAL_LOANS_DATA_QUERY`) setting the `:as_on_date` to yesterday's date.
3. The SQL dynamically computes the exact DPD, POS, Interest Outstanding, and Penalties up to that exact date by taking the original dues and subtracting any cash allocated on or before that date.
4. The results are pushed into `co_lending_daily_metrics` utilizing MySQL `ON DUPLICATE KEY UPDATE` upsert logic to ensure idempotency.

---

## 2. The Correction Engine (Historical Recalculation)
**Endpoint:** `/api/co_lending_recalculate_historical_metrics`

While the Night Cron is perfect for forward-moving time, the financial world often involves **backdated corrections**. If a customer paid 5 days ago, but the payment is only uploaded into the system today, the Night Cron's snapshots for the last 5 days are suddenly mathematically incorrect.

### Backdated Payment Trigger
Whenever the Payment Allocation Engine (`co_lending_regular_allocation.py`) processes a payment, it checks the `payment_date`.
If `payment_date < today`, it instantly spawns a background thread (to prevent blocking the API response).

### The "Time Machine" Flow
1. **Force Refresh:** The thread first synchronously invokes the Bounce & Penal engine. This is a critical step to ensure that any penalties that were reversed by the backdated payment are instantly cleared from the `co_lending_emi_payments` table.
2. **Reverse Math:** The thread then loops from the backdated `payment_date` up to `today`.
3. **Overwrite:** For each day in that range, it re-runs the `HISTORICAL_LOANS_DATA_QUERY` and overwrites the stale rows in `co_lending_daily_metrics` with the freshly corrected math.

### Manual Corrections
This API endpoint is also exposed manually. If a developer or operations team member makes a manual database correction (e.g., waiving a fee, deleting a corrupt payment log), they can hit `/api/co_lending_recalculate_historical_metrics` with the affected `loan_ids` and the `from_date` of the correction. The API acts as a snowplow, perfectly rebuilding the historical data up to the current day.

---

## Technical Details: The SQL Projection Logic
The core of this engine relies on `HISTORICAL_LOANS_DATA_QUERY`.

Because the system doesn't maintain granular, timestamped logs of *when* a bounce charge was applied, the SQL query uses a reverse-math projection trick:
1. It looks at the *current* `bounce_charges` value in `co_lending_emi_payments`.
2. It strictly checks if the `emi_date` is `<=` the snapshot `:as_on_date`. If yes, it includes the charge.
3. It subtracts any `bounce_paid` from `co_lending_payment_transactions` that occurred on or before the `:as_on_date`.

This guarantees that the snapshot perfectly reflects the outstanding balance on any given day in the past.