# Loan Management System Backend

This project is a comprehensive, microservice-oriented backend for a Loan Management System (LMS), built using Python and FastAPI. It is designed to handle various core loan lifecycle processes, from repayment schedule generation to complex financial reporting.

## System Architecture

The system is designed as a set of microservices orchestrated by a primary API Gateway (`app.py`). The gateway, built with FastAPI, receives incoming HTTP requests and routes them to the appropriate business logic module located in the `scripts/` directory. Each script is a self-contained module responsible for a specific task (e.g., payment reconciliation, CAM report generation).

This decoupled architecture allows for independent development, deployment, and scaling of each service. All services interact with a central MySQL database that serves as the single source of truth.

## Project Structure

The project follows a structured layout to maintain separation of concerns and improve maintainability.

```
├── app.py                  # Main FastAPI application with structured routes
├── requirements.txt        # Project dependencies
├── utils.py                # Shared utility functions
├── scripts/                # Core business logic for each microservice
│   ├── mysql_queries.py    # Centralized SQL queries
│   ├── pre_part_payment_v2.py # Enhanced pre-part payment handling
│   ├── Overdue_Reminder.py  # Background overdue notifications
│   ├── ptp_file_processing.py # Promise to Pay (PTP) processing
│   └── ...
├── tool/                   # Tools for specialized tasks (e.g., notebook execution)
├── bureau_report_folder/   # Generated bureau reports
├── dpd_report_folder/      # Generated DPD reports
└── ...
```

## API Endpoints

Each endpoint corresponds to a specific microservice and handles a distinct business operation.

-   **`POST /api/5th_emi_date`**
    -   **Description**: Generates Repayment Schedule for interest & hybrid cases with Broken Period Interest(BPI/Pre-EMI).

    -   **Script**: `bpi_rps_generation.py`

-   **`POST /api/subvention_rps`**
    -   **Description**: Generates an RPS for subvention-based loans.

    -   **Script**: `subvention_rps.py`

-   **`POST /api/payment_allocation_st`**
    -   **Description**: Allocates payment transaction against EMIs in particular order (Pre-EMI -> Interest -> Principal -> Bounce/Penal).

    -   **Script**: `payment_reconciliation.py`

-   **`POST /api/pre_part_payment`**
    -   **Description**: Handles the upload and processing of a pre-part payment file (Legacy).
    -   **Script**: `pre_part_payment.py`

-   **`POST /api/pre_part_payment_v2`**
    -   **Description**: Enhanced version of pre-part payment supporting previews, simulation date adjustments, and EMI/Tenure options.
    -   **Script**: `pre_part_payment_v2.py`

-   **`POST /api/overdue_update`**
    -   **Description**: Updates the overdue amount, total loan outstanding amount etc. for a given list of loan application IDs.
    -   **Script**: `overdue_update.py`

-   **`POST /api/bounce_penal_update`**
    -   **Description**: Calculates & updates bounce & penalties to a given list of loan application IDs.
    -   **Script**: `bounce_penal_update.py`

-   **`POST /api/foreclosure_breakup`**
    -   **Description**: Calculates the foreclosure amount for a specific loan incl. principal due, interest due etc.
    -   **Script**: `foreclosure_breakup.py`

-   **`POST /api/cam_report_script`**
    -   **Description**: Generates a Credit Appraisal Memo (CAM) report for a Vivriti partner.
    -   **Script**: `cam_report_script.py`

-   **`POST /api/utkarsh_cam`**
    -   **Description**: Generates a CAM report for an Utkarsh partner.
    -   **Script**: `utkarsh_cam.py`

-   **`POST /api/settlement_reconcile`**
    -   **Description**: Adds settlement amount and returns the breakup of settled and written off amounts.
    -   **Script**: `settlement_reconcile.py`

-   **`GET /dpd-report`**
    -   **Description**: Generates application-wise DPD reports with extensive filters (lender, merchant, advisor, etc.).
    -   **Script**: `dpd_report_script.py`

-   **`POST /api/update_loan_data`**
    -   **Description**: Calculates & Updates loan metrics like O/S amount, due amounts, dpd etc.
    -   **Script**: `update_loans_data.py`

-   **`POST /api/homeville_rps_update`**
    -   **Description**: Updates Repayment Schedules specifically for Homeville cases.
    -   **Script**: `homeville_rps_update.py`

-   **`POST /api/blacksoil_rps_upload`**
    -   **Description**: Uploads and updates Repayment Schedules for BlackSoil loans from Excel/CSV files.
    -   **Script**: `blacksoil_rps_update.py`

-   **`POST /api/blacksoil_payments_upload`**
    -   **Description**: Uploads payment transactions for BlackSoil loans, splitting them into principal, interest, and foreclosure entries.
    -   **Script**: `blacksoil_payments_update.py`

-   **`POST /api/merchant_collection_report`**
    -   **Description**: Background job to generate and send pending collection reports to merchants with >5 days delay.
    -   **Script**: `merchant_collection_report.py`

-   **`POST /api/co_lending_rps_bulk`**
    -   **Description**: Generates 4 distinct co-lending Repayment Schedules (RPS) in bulk from Excel/CSV uploads with dynamic configurations (partner splits, hurdle rates) fetched from the LMS master config API.
    -   **Script**: `co_lending_rps.py`

-   **`POST /overdue-reminder`**
    -   **Description**: Triggers background tasks for sending overdue reminders to customers.
    -   **Script**: `Overdue_Reminder.py`

-   **`POST /api/process-ptp-excel`**
    -   **Description**: Processes uploaded Excel files for Promise to Pay (PTP) records.
    -   **Script**: `ptp_file_processing.py`

-   **`POST /api/process-pincode-master`**
    -   **Description**: Processes bulk upload of Pincode Master records. It validates columns and data integrity (e.g. types, duplicates), then maps categories, replaces old data, and efficiently bulk inserts records into the `pincode_classifications` table using pandas' `to_sql` method.
    -   **Script**: `pincode_master_processing.py`

## Development Best Practices

To ensure code quality and consistency, developers should adhere to the following guidelines:

-   **Centralize Utility Functions**: Place all common, reusable functions (e.g., date formatting, database connections) in `utils.py`. This avoids code duplication and makes maintenance easier.

-   **Isolate SQL Queries**: All MySQL queries should be stored as functions or variables in `scripts/mysql_queries.py`. Business logic scripts should import queries from this module rather than embedding raw SQL strings. This separates the data access layer from the business logic, making both easier to manage and debug.

## Core Modules & Features

The project is divided into several key functional modules:

-   **Repayment Schedule (RPS) Generation**:
    -   `bpi_rps_generation.py`: Generates Repayment Schedule for interest & hybrid cases with Broken Period Interest(BPI/Pre-EMI). Accepts Net loan amount, tenure, annualised interest & irr(from emi_calculation table), count of morat months and morat_start_date. [JIRA Ticket](https://creditfair.atlassian.net/browse/CRED-5429)
    -   `subvention_rps.py`: Generates an RPS for subvention-based (zero-cost EMI) loans. RPS is created in rps_logs. This ticket was created so that python side payment reconciliation happens for new subvention cases too. [JIRA Ticket](https://creditfair.atlassian.net/browse/CRED-5871)

-   **Payment & Reconciliation**:
    -   `payment_reconciliation.py`: Allocates payment transaction against EMIs in particular order (Pre-EMI -> Interest -> Principal -> Bounce/Penal). [JIRA Ticket](https://creditfair.atlassian.net/browse/CRED-5379)
    -   `pre_part_payment_v2.py`: Handles advanced pre-part payment simulations and updates. Supports "EMI reduction" or "Tenure reduction" strategies and allows previewing changes before database updates. [JIRA Ticket](https://creditfair.atlassian.net/browse/CRED-6003)
    -   `ptp_file_processing.py`: Automates the ingestion of PTP (Promise to Pay) records from external spreadsheets.
    -   `settlement_reconcile.py` & `settlement_reconcile_v2.py`: Manages loan settlements and write-offs, calculating the final breakup for interest, principal, and charges. [JIRA Ticket](https://creditfair.atlassian.net/browse/CRED-6022)

-   **Reminders & Communications**:
    -   `Advance_Reminder.py`: Sends automated SMS and email notifications to customers 1-5 days before their EMI due date.
    -   `Overdue_Reminder.py`: Sends background notifications for overdue payments, including payment links.

-   **Loan Status & Penalty Calculation**:
    -   `overdue_update.py`: Calculates & updates the principal_due, total_principal_due, amount_over_due(Payment Entry page -> Overdue Amount), total_amount_due(Payment Entry page -> Total Loan Outstanding) columns in consumer_application_element table for a given list of loan application IDs. Also updates first_unpaid_emi_date, dpd, dpd_bucket, emi_overdue_count in consumer_application_additional table. Executed daily via cron job.
    -   `bounce_penal_update.py`: Calculates & updates emi wise bounce and penal charges in EMI payments for a given list of loan application IDs. Also updates total_payable, amount_due & payment_status in the same table. Executed daily via cron job.
    -   `foreclosure_breakup.py`: Calculates the final amount required for a loan foreclosure, including outstanding principal, interest, and any applicable charges. Breakup visible on Payment-Entry page -> "Send Foreclosure Link" button. No db update here.
    -   `update_loans_data.py`: Same as above but runs for multiple application_IDs and updates data in DB. Calculates & updated principal_outstanding, interest_outstanding, bounce_charges_due, penal_charges_due, processing_fee_due, down_payment_due, dpd, emi_overdue_count in consumer_application_additional table for multiple application_ids. Executed daily via cron job. Also calculates & updates the principal_due, total_principal_due, amount_over_due(Payment Entry page -> Overdue Amount), total_amount_due(Payment Entry page -> Total Loan Outstanding) columns in consumer_application_element table

-   **Reporting**:
    -   `bureau_report_script.py`: Generates CIBIL Bureau report. [JIRA Ticket](https://creditfair.atlassian.net/browse/CRED-5970)
    -   `cam_report_script.py` & `utkarsh_cam.py`: Creates Credit Appraisal Memo (CAM) reports for off-book lenders Viriti([JIRA Ticket](https://creditfair.atlassian.net/browse/CRED-5454)) & Utkarsh([JIRA Ticket](https://creditfair.atlassian.net/browse/CRED-5642)).
    - `dpd_report_script.py`: Generates DPD report. [JIRA Ticket](https://creditfair.atlassian.net/browse/CRED-5988)

-   **Partner Integrations**:
    -   `homeville_rps_update.py`: Specialized module for handling Homeville-specific repayment schedule updates.
    -   `blacksoil_rps_update.py`: Manages the ingestion and updating of BlackSoil loan repayment schedules via file upload, incorporating soft-delete logic for data integrity. [JIRA Ticket](https://creditfair.atlassian.net/browse/CRED-6212)
    -   `blacksoil_payments_update.py`: Handles the ingestion of BlackSoil payment transactions, splitting row-wise data into component-based entries (principal, interest, foreclosure).

-   **Merchant Risk Analysis**:
    -   The `/merchant-risk` endpoint and associated notebooks are used for performing risk analysis on merchant portfolios.

## Technical Stack

-   **Framework**: FastAPI
-   **Web Server**: Uvicorn
-   **Core Libraries**:
    -   Pandas & NumPy: For data manipulation and numerical calculations.
    -   Pydantic: For data validation and API request modeling.
    -   Aiofiles: For asynchronous file operations.
    -   Pytz: For timezone-aware date/time handling.
-   **Database**: MySQL (via `mysql-connector-python`)
-   **Deployment**: Each script is designed to be containerized using Docker, enabling a microservices architecture.

## Local Development Setup

Follow these steps to set up the project for local development:

1.  **Clone the Repository**
    ```bash
    git clone <your-repository-url>
    cd cf-python-scripts
    ```

2.  **Create and Activate a Virtual Environment**
    ```bash
    # For macOS/Linux
    python3 -m venv venv
    source venv/bin/activate

    # For Windows
    python -m venv venv
    .\venv\Scripts\activate
    ```

3.  **Install Dependencies**
    ```bash
    pip install -r requirements.txt
    ```

4.  **Environment Configuration**
    The application may require database credentials and other secrets. Create a `.env` file in the project root and add the necessary variables. (Note: A `.env.example` file should be created to show the required variables).

5.  **Run the Application**
    The main application with structured routes is `app.py`.
    ```bash
    uvicorn app:app --host 127.0.0.1 --port 5001 --reload
    ```
    The server will be accessible at `http://127.0.0.1:5001`.

## API Documentation

Once the server is running, interactive API documentation (provided by Swagger UI and ReDoc) is available at:

-   **Swagger UI**: `http://127.0.0.1:5001/docs`
-   **ReDoc**: `http://1.2.3.4:5001/redoc`

These interfaces provide detailed information about each endpoint, including request models and response formats.

## Deployment

The project is intended to be deployed using Docker. Each microservice can be containerized individually, and an API gateway can be used to route traffic. A `Dockerfile` and `docker-compose.yml` should be created to manage the build and deployment process.