# DA Accrued Revenue Documentation

## Overview
The DA Accrued Revenue script automates the daily tracking of Principal Outstanding (POS) and Accrued Revenue for Blacksoil loans. 
It processes loan data, EMI schedules, and payment transactions to calculate the exact state of each loan on a daily basis and safely stores this historical data in the `da_daily_metrics` database table.

## Business Logic
The core purpose of this script is to determine the precise revenue that has accrued but is not yet due, as well as tracking the daily drop in Principal Outstanding based on received payments.

### Calculation Rules
- **Interest Accrued Not Due:** `Principal Outstanding × ROI (%) × No. of Days / 365`. This represents the interest accrued from the most recent passed EMI date up to the reporting date.
- **Principal Outstanding (POS):** POS is tracked dynamically. It starts at the total loan amount and drops precisely on each EMI date by subtracting any principal payments received during the calendar month of that EMI. If a foreclosure payment is identified, POS drops to 0.
- **Accrual Window:** The system does not consider the full interest component of the next EMI. It solely calculates the interest accrued up to the reporting date (`as_on_date`). For example, if the As-Of Date is August 31st and the next EMI is in September, it accurately calculates the days passed in August.

## Technical Logic

### Data Handling
- **In-Memory Calculation:** To ensure 100% accuracy, the script calculates the daily balances day-by-day from the `first_emi_date` in memory for each loan. This looping is highly performant and takes a fraction of a millisecond per loan.
- **Selective Database Upserts:** To prevent overloading the database, the script uses a `start_date` boundary. It calculates the entire history in memory to find the true balance, but it will **only write/upsert rows to the database** for dates `>= start_date`.
- **Batch Processing:** Database writes are chunked into safe batches of 5,000 records at a time using an `INSERT ... ON DUPLICATE KEY UPDATE` query to avoid MySQL max packet size limits.
- **Timezones & Logging:** It utilizes the system's local timezone for database timestamps (`created_at`, `updated_at`) and actively mirrors all terminal logs to an AWS CloudWatch stream named `da-daily-metrics`.

### Workflow & Integration
The script is deeply integrated with FastAPI, offloading the heavy calculations into background threads using `BackgroundTasks` so the API never times out.

#### 1. Daily Nightly Cron
Designed to run automatically every night to increment the accrued days and DPD metrics.
- **Endpoint:** `POST /api/calculate-blacksoil-da-accrued-revenue`
- **Payload:** `{}`
- **Behavior:** `start_date` and `as_on_date` default to the current date. The script fetches all active loans (where POS > 0) and efficiently inserts exactly **1 row per loan** into the database for the current day.

#### 2. Instant Payment Sync
Directly integrated into the Payment Upload workflow to immediately account for new or backdated payments.
- **Trigger:** When a user successfully uploads a payment file via `POST /api/blacksoil_payments_upload`.
- **Behavior:** The payment script parses the file, calculates the earliest `Payment Date` within it, and collects the unique `loan_ids`. It instantly triggers the accrued revenue calculation in the background, specifically for those affected loans, passing the earliest payment date as the `start_date`. This guarantees 100% accuracy for backdated payments with minimal database writes.

#### 3. Full Historical Backfill
Used for onboarding historical loans from other systems or rebuilding a wiped database table.
- **Endpoint:** `POST /api/calculate-blacksoil-da-accrued-revenue`
- **Payload:** `{"recalculate_history": true}` (Optionally pass an array of specific `loan_ids`)
- **Behavior:** This explicitly bypasses the `start_date` optimization. It aggressively calculates and upserts every single historical day's metric from the inception of the loan (`first_emi_date`) up to the current date.