# BRE Data Point Readiness — Redis Tracker

**CRED-6289**

Fire the BRE jobs when every input they read is present, instead of guessing from events. Emit a notification whenever an application's readiness changes, surface what is still missing on the admin edit screen, and block the Disbursement Checklist Done transition when post-approval details are outstanding.

Scope is **solar merchants only**. Everything in this document is inert for any other merchant category.

## Files

| File | Role |
|------|------|
| [app/Helpers/DatapointHelper.php](../app/Helpers/DatapointHelper.php) | Field lists, labels, DB derivation, dispatch, notification |
| [app/Services/RawRedisConnector.php](../app/Services/RawRedisConnector.php) | Raw RESP-over-TCP Redis client (no predis/phpredis in this runtime) |
| [app/Observers/ConsumerApplicationObserver.php](../app/Observers/ConsumerApplicationObserver.php) | 12 applicant + 6 co-applicant flags |
| [app/Observers/ConsumerApplicationElementObserver.php](../app/Observers/ConsumerApplicationElementObserver.php) | 11 element flags |
| [app/Providers/EventServiceProvider.php](../app/Providers/EventServiceProvider.php) | Observer registration |
| `ConsentController@showConsentPage` ([L34](../app/Http/Controllers/ConsentController.php#L34)) | Renders the consent page (route moved here from `LenderServices@saveConsent`) |
| `ConsentController@saveConsent` ([L192](../app/Http/Controllers/ConsentController.php#L192)) | Writes the `consent_log` row, queues the `mconnectConsent` job |
| `ConsentController@addConsentDocument` ([L300](../app/Http/Controllers/ConsentController.php#L300)) | Consent choke point — the only co-applicant trigger |
| [app/Jobs/AutoDecisionJob.php](../app/Jobs/AutoDecisionJob.php) | Approval job, applicant only |
| [app/Jobs/RunRejectionJob.php](../app/Jobs/RunRejectionJob.php) | Rejection job, both applicant types |
| [app/Jobs/DatapointNotificationJob.php](../app/Jobs/DatapointNotificationJob.php) | Readiness notification → WhatsApp group via Narad |
| `ServiceController@getGroupNameByApplication` ([L1542](../app/Http/Controllers/Service/ServiceController.php#L1542)) | Merchant → WhatsApp group resolution |
| `ConsumerApplicationController@getPendingFieldsForDecision` ([L11443](../app/Http/Controllers/Admin/ConsumerApplicationController.php#L11443)) | Pre-decision admin lines |
| `ConsumerApplicationController@getPostApprovalPendingPoints` ([L11453](../app/Http/Controllers/Admin/ConsumerApplicationController.php#L11453)) | Post-approval checklist + DCD blocker |
| `GradeController@checkEbillNameMatchWithGradeRules` ([L6016](../app/Http/Controllers/Grade/GradeController.php#L6016)) | E-bill name match vs the grade's `e_bill_fuzzy_match` rule |
| [sections/pending-datapoints-alert.blade.php](../resources/views/admin/consumer-application/sections/pending-datapoints-alert.blade.php) | Shared partial for every pending data point line |
| [edit_v1.blade.php](../resources/views/admin/consumer-application/edit_v1.blade.php) | Pending data point lines, e-bill name match line |
| [layouts/footer.blade.php](../resources/views/admin/layouts/footer.blade.php) | Status-change validation popup |

Migrations: `2026_08_16_120154_add_column_source_to_consent_log.php` (`consent_log.source`), `2026_08_26_122431_create_failed_jobs_table.php` (`failed_jobs`, so a dead SQS job is recoverable rather than lost).

---

## Scope gate — who is tracked at all

`DatapointHelper::isEligible($id, $createdAt, $state, $merchantId)` is the single gate, and all three of its conditions must hold:

| Condition | Rule |
|-----------|------|
| Merchant category | `merchant.category_id` equals `SOLAR_CATEGORY_ID` |
| Application state | `in-process`, `waiting for decision`, or `waiting for documents` |
| Application age | `consumer_application.created_at` >= `RELEASE_DATE` (override: `DATAPOINT_RELEASE_DATE`, plain `env()`) |

Anything failing any of them is **completely out of scope** — no flags, no notification, no dispatch, no admin lines.

The release date exists because without it the first post-release save of *any* historical application would cold-start the tracker. Most old applications have every approval field long since filled, so they would immediately claim the guard and dispatch `AutoDecisionJob` — a fresh bureau pull and BRE run on a loan decided months ago, plus a WhatsApp message to the merchant group. `RunRejectionJob` is worse: it only returns early when the state is already `rejected`, so a **disbursed** loan could be moved to `rejected`.

The state list closes the other half of that hole: an application created after go-live that reaches `approved` or `disbursed` and is then edited no longer re-enters the tracker. Note the list is narrower than the one the consent flow checks (`in-process`, `incomplete form`, `form completed`, `waiting for documents`) and narrower than the one the admin edit screen uses for its pre-decision lines (which also includes `form completed`) — an application in `incomplete form` or `form completed` is **not** tracked.

Gated at every entry point: `write()`, `refresh()`, `attemptDispatch()`, `pending()`, `pendingByApplicant()`. `sync()` and `dispatchIfReady()` inherit it.

| Situation | Result |
|-----------|--------|
| Solar merchant, eligible state, created on/after release | in scope |
| Non-solar merchant | skipped everywhere |
| State outside the eligible list | skipped everywhere |
| `created_at` < release date | skipped everywhere |
| `created_at` null/unparseable | skipped (unknown age must not licence a decision) |
| Merchant row missing, or the lookup throws | skipped |
| `RELEASE_DATE` unparseable | feature off, and logged |

`pending()` returns `[]` rather than "everything pending" for an out-of-scope application — otherwise the line would claim a decided application is incomplete.

`ConsumerApplicationObserver` passes `created_at`, `state` and `merchant_id` from the model, so the hot path skips the application lookup. It does **not** skip the merchant lookup, and the per-request memo is currently dead code — see gotcha 13.

> `RELEASE_DATE` ships as `2026-08-19 00:00:00` and will be set to the real go-live timestamp at release.

## The switch

`AUTO_BRE` decides which flow runs:

| Value | Flow |
|-------|------|
| `database` | Old RabbitMQ `EventAutoDecision` path |
| anything else, incl. unset | This Redis/SQS path |

Checked by `DatapointHelper::isRedisFlow()`, which gates **dispatch and notification** (flag writes always happen, so the admin lines work either way).

The old path now survives in exactly one place: `ConsentController@addConsentDocument` still raises `EventAutoDecision` when `isRedisFlow()` is false. The equivalent block in `ConsumerApplicationController` — the one that fired `EventAutoDecision` (or ran `generateSoftBureauReport` + `autoReject` inline) on every application save — has been **removed**, because the observers now cover that trigger. Consequence: with `AUTO_BRE=database` the only remaining auto-decision trigger is a consent event. An application save no longer starts anything on either flow.

> Read with `_env()`, which **decrypts**. The `.env` value must be the `wall_encrypt()` ciphertext of `database` — a plaintext `AUTO_BRE=database` fails the comparison and silently leaves the Redis flow on.

---

## Tiers

A tier is *(field list → job → guard)*, declared in `DatapointHelper::$tiers`.

| Tier | Applicant type | Fields | Job | Guard field |
|------|----------------|--------|-----|-------------|
| Approval | 1 | 24, or 25 with a co-applicant | `AutoDecisionJob` | `autodecision_dispatched_1` |
| Rejection | 1 | 5 | `RunRejectionJob` | `rejection_dispatched_1` |
| Rejection | 2 | 6 | `RunRejectionJob` | `rejection_dispatched_2` |

There is **no co-applicant approval tier**. `required('approval', 2)` returns `[]`, and an empty list means "does not apply" — never "nothing missing".

`required($tier, $applicantType, $datapoints = null)` takes the hash as a third argument so the approval list can depend on whether a co-applicant exists. With `$datapoints` omitted it assumes one does, which is the conservative default for `tracked()` and for candidate resolution.

### Applicant rejection (5)
`name`, `dob`, `pan_card`, `monthly_income`, `consent_log`

### Applicant approval (24)
The 5 above, plus:

| Table | Fields |
|-------|--------|
| `consumer_application` | `loan_amount`, `offer_id`, `company_name`, `current_address_line_1`, `current_address_pincode`, `current_address_state`, `current_address_city`, `employee_status` |
| `consumer_application_element` | `product_name`, `installation_address_line_1`, `installation_address_pincode`, `installation_address_city`, `installation_address_state`, `solar_power_type`, `type_of_customer`, `inverter_monthly_power_bill`, `inverter_KW_installation`, `inverter_gross_project_amount`, `inverter_power_savings` |

Two swaps from the first cut: `profession_type` (element) was replaced by `employee_status` (application), which is the column the form and the BRE actually read, and `product_id` was replaced by `product_name`.

### Co-applicant rejection (6)
All on `consumer_application`:

`co_applicant_name`, `co_applicant_dob`, `co_applicant_pan_card`, `co_applicant_mobile_number`, `co_applicant_monthly_income`, `co_applicant_address_pincode`

Notes:
- **The column is `co_applicant_address_pincode`**, not `co_applicant_pincode` — the latter does not exist. A non-existent name makes `!empty()` permanently false and the tier could never complete.
- `co_applicant_pan_card` / `co_applicant_mobile_number` are encrypted. Fine for a presence flag: `Encryptable` only encrypts non-empty values, so blank stays `NULL`/`''` on both the model path and the raw-query path.
- Consent is **not** in this list — it gates *who may dispatch* instead. See below.

Every name except `consent_log` and `co_applicant_rejection_executed` is a literal column name. That is what lets `buildFromDatabase()` resolve a field with no field→table map. **Verify against the schema before adding one.**

### Co-applicant is optional
`hasCoApplicant()` reads `co_applicant_name`. With no co-applicant: type 2 is never evaluated, no type-2 guard is claimed, `pending` omits key `2` entirely (rather than listing six permanently-missing fields), and the applicant approval tier drops back to 24 fields.

### Labels
`DatapointHelper::$labels` maps every tracked key to the phrasing a merchant or an operator sees — `offer_id` → *Offer*, `inverter_KW_installation` → *System Capacity (kW)*. `label()` falls back to title-casing the raw key, so an unlabelled field is readable rather than missing. `labelled(array)` maps a list.

Applied at both display points: `DatapointNotificationJob` for the WhatsApp message, and `getEditPostConsumerApplication` for the admin lines. Everything else — Redis fields, logs, the API — stays on raw keys.

---

## The co-applicant handshake

`co_applicant_rejection_executed` is a **synthetic** data point: no column backs it. It is added to the applicant approval list whenever the application has a co-applicant, and it has two sources.

**Written** by `RunRejectionJob`, immediately after the co-applicant rejection policy has run:

```php
DatapointHelper::sync($id, ['co_applicant_rejection_executed' => 1]);
```

`sync()` defaults to applicant type 1 and to dispatching, so writing the flag is also what re-evaluates the applicant approval tier.

**Rebuilt** by `buildFromDatabase()` from the dispatch guard, via `coApplicantRejectionExecuted()`:

```php
$guards = $redis->hgetall(self::jobDispatchKey($applicationId));
return empty($guards[self::jobTypeKey(2, 'rejection')]) ? 0 : 1;   // rejection_dispatched_2
```

The guard is the only durable record available. The written flag lives in the data point hash, which has a 30-minute TTL and is rebuilt from scratch by every `refresh()` — so sourcing the rebuild from a column (there is none) or from nothing (the original behaviour, which wrote `0`) left the flag erasable while `rejection_dispatched_2` stayed claimed for 90 days, with nothing left that could ever set it again. `refresh()` now passes its connection down so the guard read shares the socket with the write that follows.

Consequence of using the guard: the flag means **"the co-applicant rejection job has been dispatched"**, not "it has finished". Inside a single `attemptDispatch()` pass that is invisible — the data points are read before anything dispatches, so the snapshot approval is judged against still carries `0`. Across calls it means a `refresh()` landing while the co-applicant job is still queued can let the applicant's approval go out first. A Redis failure during the read degrades to `0`, which withholds an approval rather than issuing one.

What this buys: on a co-applicant application the applicant's approval decision does not go out until the co-applicant has been run through `autoRejectCoApplicantJSONBRE`, so an approval cannot be issued while a disqualifying co-applicant verdict is still unrecorded.

What it costs:

- **Applicant approval is now coupled to co-applicant consent.** The co-applicant rejection tier dispatches only from the consent choke point, so if the co-applicant never consents, the flag never flips and `AutoDecisionJob` never fires for the applicant either.
- **Approval no longer subsumes rejection on a co-applicant application.** The approval tier is blocked on the flag while the applicant rejection tier (a strict subset of the other 24 fields) is not, so `RunRejectionJob` for applicant type 1 goes out first and `AutoDecisionJob` follows later — which runs the applicant rejection policy a second time. On a single-applicant application the original subsumption still holds.
- **It inherits the guard's 90-day lifetime.** Once `rejection_dispatched_2` expires, a rebuild reports `0` again. That is beyond the life of any live application, and it is the same expiry every other guard already has.

`attemptDispatch()` `krsort`s its candidate map so applicant type 2 is evaluated before type 1, which is the order the handshake wants within a single call. In practice every live call site passes an explicit applicant type, so the map never has more than one key and the sort is defensive only. The method's docblock still describes the old ascending order and is stale.

---

## Redis keys

### `loan:{id}:datapoints` — readiness flags
TTL **1800 s** (`TTL_SECONDS`), rolled forward on every write. Values are `"1"`/`"0"` only, never business data. Disposable — `refresh()` rebuilds it from the DB.

### `loan:{id}:dispatch_guard` — what has been sent
TTL **7 776 000 s / 90 days** (`JOB_TTL_SECONDS`), set once at first claim.

Fields: `autodecision_dispatched_{1,2}`, `rejection_dispatched_{1,2}`.

Guards live in a **separate key** because the two facts have opposite durability needs: losing a flag costs nothing, losing a guard means sending a job twice. The datapoints key's short TTL and the backfill `DEL` must not touch guards.

---

## Flow

```
Eloquent save (application / element)        Consent document captured
            │                                          │
            ▼                                          ▼
  own attrs → 0|1 flags                    DatapointHelper::refresh($id)
  DatapointHelper::write()                 → rebuild EVERY flag from the DB
            │                                          │
            ├──────────────► write() ◄─────────────────┤
            │        isEligible? (solar / state / age) │
            │        HGETALL (before)                  │
            │        HMSET + EXPIRE                    │
            │        notifyOnChange()                  │
            │          cold?    → handle() inline      │
            │          flipped? → SQS                  │
            │                                          │
            ▼                                          ▼
  maybeDispatch(..., 1)                    dispatchIfReady($id, $applicant_type)
  APPLICANT ONLY                           applicant OR co-applicant
            │                                          │
            └──────────────► attemptDispatch ◄─────────┘
                    1. isRedisFlow? isEligible? candidates?
                    2. HGETALL guards → drop claimed
                    3. HGETALL datapoints
                       → drop type 2 if no co-applicant
                    4. per (type, tier), co-applicant first:
                       all fields present?
                    5. HSETNX guard, then dispatch
                       (+ claim subjob guards, same type)
                                    │
                                    ▼
                                AWS SQS
                    ┌───────────────┼───────────────┐
                    ▼               ▼               ▼
            AutoDecisionJob   RunRejectionJob   RunRejectionJob
                 (type 1)        (type 1)          (type 2)
              bureau + BRE    bureau + reject   bureau + log only
                                                        │
                                                        ▼
                                          sync(co_applicant_rejection_executed)
                                              → re-evaluates applicant approval
```

### Why the co-applicant only dispatches from the consent choke point

`generateSoftBureauReport()` refuses to pull for applicant type 2 without co-applicant consent (`co_app_consent_logs` file or `is_co_app_consent = 1`).

The six co-applicant fields typically land at form-completion time, long before consent. If the observers dispatched type 2, the job would arrive, fail the pull, and its guard would already be claimed — silent for 90 days.

So the observers pass an explicit type `1`, and the co-applicant tier is dispatched only from `addConsentDocument`, where consent exists by definition. `saveConsent()` sets `is_co_app_consent` *before* the consent row is written, so the ordering holds.

There is deliberately **no `ConsentLog` observer** — the consent document choke point covers it.

### The consent path, end to end

The consent page route now points at the controller that owns the flow:

```
GET /consent/{application_id}/{applicant_type?}   →  ConsentController@showConsentPage
```

(previously `Admin\LenderServices@saveConsent`). `showConsentPage` decodes the Hashid, short-circuits to the "already logged" view when both the consent flag and the consent file exist, and otherwise renders `front.consent_page`.

Submitting it runs `saveConsent`, which verifies the mobile-hash in the link, requires a geolocation (unless the application came from the WhatsApp agent), sets `is_consent` / `is_co_app_consent`, scrubs the consent entry from `mandatory_document_list`, and writes the `consent_log` row with the new **`source = 'admin'`** column. It then queues a `JobProvider` row:

```php
['from' => 'mconnectConsent', 'is_available' => 0, 'payload' => {application_id, source, applicant_type, status}]
```

`is_available` was `1` on insert, which is the value `Jobs::start()` treats as already handled — the job was therefore never picked up. It is now `0`, so the row is actually consumed, and `status` / `source` are read from the payload instead of being hardcoded to `agree` / `mconnect`.

That job calls `LenderServices@addConsentDocument` → `ConsentController@addConsentDocument`, which renders and uploads the consent PDF, calls `DatapointHelper::refresh()`, and makes the dispatch decision. The `$source` parameter was dropped from both signatures along with the `|| $source == 'mconnect'` escape hatch that let an mconnect-sourced call re-run the whole block on an application whose consent flag was already set.

> `addConstantDocument` is spelled `addConsentDocument` everywhere now (`ConsentController`, `LenderServices`, `Jobs`). Docs [10](10.consent-phase-2.md) and [12](12.bureau-bre-flow-modification-on-waiting-for-documents.md) were updated for the rename.

The auto-decision precondition inside `addConsentDocument` also changed: `type_of_customer == 1 && solar_power_type == 1` was replaced by `merchant->category_id == SOLAR_CATEGORY_ID`. Solar is now identified by the merchant's category rather than by two element fields that only happened to correlate with it.

---

## Notification

`write()` reads the hash before writing, because Redis has no "set and return the previous value" and `HMSET` counts fields *created*, not changed — so a `1 → 0` flip is otherwise invisible. The same read detects a cold start free (an empty hash cannot exist in Redis).

| Trigger | Condition | Delivery |
|---------|-----------|----------|
| `cold_start` | key was absent | direct `handle()` call, in-process |
| `changed` | key existed, a flag flipped | SQS |

Cold start wins when both apply. Nothing flipped → no notification.

What counts as a flip: `1→0` yes, `0→1` yes, absent→`1` yes (reported as `old: null`), **absent→`0` no**, unchanged no.

### Payload

```
application_id    int
trigger           'cold_start' | 'changed'
has_co_applicant  bool
changed           ['offer_id', ...]
changes           ['offer_id' => ['old' => 0, 'new' => 1]]
flags             all 31 tracked points, ints
pending           [1 => ['approval' => [...], 'rejection' => [...]],
                   2 => ['rejection' => [...]]]      // key 2 only if co-applicant
ready             [1 => ['approval' => false, ...], 2 => ['rejection' => false]]
```

A snapshot taken at trigger time, not re-derived on delivery. Use `pendingByApplicant($id, true)` for live state.

### Delivery — WhatsApp group (Narad)

`DatapointNotificationJob::notify()` logs the snapshot to CloudWatch under `DatapointNotificationQueueLog`, then messages the merchant's WhatsApp group.

1. **Flatten.** All pending fields across every tier *and* both applicant types are collapsed into one deduplicated list, then run through `DatapointHelper::labelled()`. The message is deliberately generalised — no approval/rejection split, no applicant/co-applicant split. The merchant needs to know what is missing, not which rule tier wants it.
2. **Nothing pending → no message.** A fully complete application is silent.
3. **Resolve the group** via `ServiceController@getGroupNameByApplication($id)` (made `public` for this, and constructed with a `Request` since the controller requires one). That method: looks up `merchant_id`; drops out if `WHATSAPP_BOT_MERCHANT_WHITELIST` is non-empty and does not contain it; loads `groups.json` from S3 (`GROUPS_JSON_PATH`); matches group name → merchant. `null` anywhere → no message.
4. **POST** to `NARAD_HOST` + `/_admin/send-group-status` with `CF_ACCESS_KEY` / `CF_SECRET_KEY` headers, 30 s timeout. cURL errors are logged as `DatapointNotificationError` and swallowed.

Message body:

```
Customer Name: <name>
Mobile Number: <mobile_number>
Application ID: <id>
Pending Data Points for Auto-Decision:
- Offer
- Co-Applicant Date of Birth
```

`mobile_number` is an encrypted column read through the model, so it is decrypted here — **the message puts a customer name and phone number into a merchant WhatsApp group.**

**`WHATSAPP_BOT_MERCHANT_WHITELIST` is the launch control.** Set it to one pilot merchant to bound the blast radius. Note it gates only the *message* — dispatch is not whitelisted, so BRE jobs still fire for every eligible application.

Three constraints on this channel, all live:

1. **Must not write to an observed model** — the cold-start call runs inside `Observer::saved()` and would re-enter `write()` and recurse. Currently satisfied (reads only).
2. **Must tolerate running twice** — SQS redelivers, and `trigger` is not a guard. Currently **not** deduplicated; see gotcha 7.
3. **Must be cheap** — the cold-start path blocks the caller's request. Currently **violated**: a 30 s cURL runs inline. See gotcha 12.

---

## Jobs

**`AutoDecisionJob`** serves **applicant type 1 only**. The type-2 branch — which pulled the co-applicant bureau and returned without deciding — was removed, because the co-applicant now has its own tier and its own job. The job still forces `Input::merge(['applicant_type' => 1])` so a worker process cannot leak a co-applicant value from a previous job into the hard pull.

**`RunRejectionJob`** serves both applicant types off `$param['applicant_type']`. Both pull the bureau first via `generateSoftBureauReport($id, 'autoBRE', $type, false)` — `$runBre = false` matters, or the controller runs the rejection policy a second time itself.

| Type | Policy | On a `true` verdict |
|------|--------|---------------------|
| 1 | `autoRejectJSONBRE` | `state = 'rejected'` + `StatusLogs` row under user 107 |
| 2 | `autoRejectCoApplicantJSONBRE` | **Verdict recorded only, state untouched.** The policy writes the `auto_reject_logs` row (`applicant_type = 2`) and the operator comment itself. Matches `GradeController@autoApproveInitCoapplicant`. Then signals `co_applicant_rejection_executed`. |

Both return early when the application is already `rejected`. Type 2 also returns early if `co_applicant_name` is empty — note that an early return skips the `sync()`, so the applicant approval tier stays blocked.

### Queue plumbing

`failed_jobs` now exists, so a job that exhausts `$tries = 3` is persisted instead of vanishing. `AppServiceProvider` registers a `Queue::before` listener that prints the SQS job id and raw body to STDOUT when `APP_ENV` is `staging` — staging-only, because the body contains the payload.

---

## `DatapointHelper` API

| Method | Purpose |
|--------|---------|
| `$tiers` | Registry: `tier => ['job', 'guard', 'subjob']`. Array order is dispatch priority. |
| `$labels` / `label($field)` / `labelled($fields)` | Raw key → human-readable label. |
| `isRedisFlow()` | Is `AUTO_BRE` anything other than `database`? |
| `isEligible($id, $createdAt, $state, $merchantId)` | Solar category + eligible state + created on/after release. Pass what you already have to skip lookups. |
| `required($tier, $applicantType, $datapoints = null)` | Field list. `[]` means the tier does not apply to that type. Pass `$datapoints` so the approval list can drop `co_applicant_rejection_executed` on a single-applicant application. |
| `tracked()` | Union of every list — 31 fields. |
| `hasCoApplicant($datapoints)` | Reads `co_applicant_name`. |
| `datapointKey($id)` / `jobDispatchKey($id)` | The two hash keys. |
| `jobTypeKey($applicantType, $tier)` | A guard field name. |
| `buildFromDatabase($id, $redis = null)` | Authoritative DB → flags, plus the one synthetic flag read off the dispatch guard. `[]` if the application is missing. Pass a connection to reuse it. |
| `write($redis, $id, $fields)` | **The one write path.** `HGETALL` → `HMSET` → `EXPIRE` → notification. Never `HMSET` yourself. |
| `refresh($id)` | `buildFromDatabase()` → `write()`. Repairs drift. Does not dispatch. |
| `sync($id, $fields, $type, $dispatch)` | For a writer that knows only its own fields. |
| `dispatchIfReady($id, $applicantType = null)` | `null` = applicant plus co-applicant if present. |
| `pending($id, $rebuild, $applicantType = 1)` | Flat `['approval'=>[], 'rejection'=>[]]` for one type. |
| `pendingByApplicant($id, $rebuild)` | Keyed by applicant type. Powers the admin lines and the notification. |
| `maybeDispatch($id, $redis, $key, $ownFields, $applicantType)` | Observer path — reuses the open connection, short-circuits on own fields. |
| `log($context)` | CloudWatch line. `public` so the observers and controllers log in the same shape. |

Everything except `maybeDispatch()`/`write()` opens its own connection and **never throws** — a Redis problem must not break a save.

### Flag derivation
- `consent_log` — an agreed (`status = 1`), non-deleted `consent_log` row with **`applicant_type = 1`**. The applicant-type filter matters: without it a co-applicant's consent satisfied the applicant's tier.
- `co_applicant_rejection_executed` — synthetic, see above. Seeded from the `rejection_dispatched_2` dispatch guard rather than from a column, so it survives a rebuild.
- Everything else — the data point name *is* the column name; resolved against `consumer_application`, then `consumer_application_element`. No match while an element row exists logs an error (that means the column exists nowhere).

---

## Admin edit screen

### Pre-decision data points

Shown when the merchant is solar **and** the state is `in-process`, `waiting for decision`, `waiting for documents` or `form completed`. Sourced from `getPendingFieldsForDecision($id)` → `pendingByApplicant($id, true)`, so an expired hash is rebuilt by the page load.

All four lines render through `sections/pending-datapoints-alert.blade.php`, which replaced the dismissible Bootstrap alerts with a single bold-label line carrying a count and a `<code>` value — the same shape as the other status lines on the page. It accepts either an array of labels or a comma-separated string.

| Line | Source |
|------|--------|
| Pending Data Points | shown instead of the two below when the rejection and approval lists are identical |
| Pending Rejection Data Points | `[1]['rejection']` |
| Pending Approval Data Points | `[1]['approval']` |
| Pending Co-Applicant Rejection Data Points | `[2]['rejection']`, absent without a co-applicant |
| Pending Post Approval Data Points | see below |

The dedupe matters because the rejection list is a subset of the approval list: before the state is far along, both lists are frequently the same five fields, and printing them twice was noise.

### Post-approval data points

A separate, non-Redis checklist read straight off the loaded model by `getPostApprovalPendingPoints($application)`. Shown for solar merchants in state `approved`, `cf approved` or `disbursement query`.

| Table | Fields |
|-------|--------|
| `consumer_application` | `marital_status`, `email`, `gender`, `permanent_address_pincode` |
| `consumer_application_element` | `type_of_customer_sub_category`, `applicant_religion`, `inverter_model_description`, `panel`, `inverter`, `inverter_subsidy_amount`, `discom_service_provider`, `type_of_customer`, `father_name` |

Returns a comma-separated, title-cased string of whatever is empty.

### The DCD blocker

The same list blocks the move to **Disbursement Checklist Done**. In `checkValidation`, for `lender_id == 20` and a solar merchant:

```php
$pendingDatapoints = $this->getPostApprovalPendingPoints($application);
if (!empty($pendingDatapoints)) {
    return response(['main' => ['post_approval_datapoints_pending' => false, 'pending_points' => $pendingDatapoints]], 200);
}
```

It runs **before** the existing loan-agreement / hypothecation / TFR checks, so an operator sees the missing details first. The selector also changed from `$application->purpose == SOLAR_CATEGORY_ID` to `$application->merchant->category_id == SOLAR_CATEGORY_ID` (hence `merchant` was added to the eager-load list in the fallback query), and the lender from `2` to `20`.

Three related changes in the validation popup (`footer.blade.php`):

1. **New key** `post_approval_datapoints_pending`, `mandatory: true`, message *"Post approval data points pending. Please fill the following details - &lt;list&gt;"*. `getValidationError` reads `pending_points` out of `response.main` (falling back to the top level) — defensively, because `response.data` on this endpoint is an HTML blob.
2. **The `mandatory` flag was inverted.** `if (typeof validation.mandatory == 'undefined')` set `mandatory = true`, i.e. exactly backwards: a soft warning blocked, and a genuine `mandatory: true` blocker got the "Next" button and let the status change through. Now `!= 'undefined'`. Every pre-existing `mandatory: true` key in that file — nine of them — was bypassable before this fix.
3. **`aadhaar_missing` was removed.** The blocker that stopped a case reaching DCD without an Aadhaar number is gone, along with its validation message. Aadhaar is no longer a DCD precondition.

### E-bill name match

`GradeController@checkEbillNameMatchWithGradeRules($application_id)` compares the Karza name-match percentages stored on the latest electricity bill against the grade's configured threshold, and the edit page prints the verdict as a pass/fail line.

- **Input** — the most recent non-deleted `consumer_application_files` row of type `electricity_bill` or `electricity_bill_validation_report` with a non-null `ebill_name_verify`, e.g. `{"applicant_match": 81.76, "co_applicant_match": 75}`.
- **Threshold** — `e_bill_fuzzy_match` from the first active (`status = 1`) `ApprovalRule` on the merchant's grade that defines it numerically. The value is identical across active policies, so the first hit wins.
- **Verdict** — passes when the applicant match clears the threshold, or when it falls short but the co-applicant match clears it.
- **Returns** a fixed-shape array: `status` (`true` / `false` / `null` when it could not be evaluated), `threshold`, `applicant_match`, `co_applicant_match`, `applicant_pass`, `co_applicant_pass`, `file_id`, `reason`. Exceptions are logged as `checkEbillNameMatchWithGradeRules failed` and collapse to `status: null`.
- **Display** — green tick or red cross with the rule and each percentage when `status` is non-null; a grey info line carrying `reason` otherwise (no e-bill, unreadable JSON, no grade mapped, rule not configured).

This line is computed for every application on the edit screen, not just solar ones — it simply resolves to a `reason` line where the inputs do not exist.

---

## Config

Redis (`RawRedisConnector`, plain `env()`): `RAW_REDIS_HOST`, `RAW_REDIS_PORT`, `RAW_REDIS_SCHEME` (`tcp`/`tls`), `RAW_REDIS_PASSWORD`, `RAW_REDIS_DB`, `RAW_REDIS_TIMEOUT`, `REDIS_READ_TIMEOUT`, `RAW_REDIS_MAX_RETRIES`, `RAW_REDIS_RETRY_BACKOFF_MS`.

Queue: `QUEUE_DRIVER=sqs` with `SQS_KEY`, `SQS_SECRET`, `SQS_PREFIX`, `SQS_QUEUE`, `SQS_REGION`. Jobs set `$tries = 3`.

WhatsApp: `NARAD_HOST`, `NARAD_CF_ACCESS_KEY`, `NARAD_CF_SECRET_KEY`, `GROUPS_JSON_PATH`, `WHATSAPP_BOT_MERCHANT_WHITELIST` (all `_env()`).

| Variable | Accessor | Purpose |
|----------|----------|---------|
| `AUTO_BRE` | `_env()` — **encrypted** | `database` = old flow, else this one |
| `SOLAR_CATEGORY_ID` | `_env()` — **encrypted** | Merchant category that puts an application in scope |
| `DATAPOINT_RELEASE_DATE` | `env()` — plain | Overrides `RELEASE_DATE`; applications older than this are skipped |
| `APP_ENV` | `_env()` | `staging` turns on SQS job body logging |

> This codebase has two env accessors: `env()`, and `_env()` ([CustomHelpers.php:289](../app/Helpers/CustomHelpers.php#L289)) which **decrypts** via `wall_decrypt()`. Store the value to match the accessor — an encrypted value read with `env()` comes back as ciphertext, and a plaintext value read with `_env()` comes back as garbage.

---

## Gotchas

1. **`co_applicant_rejection_executed` reports dispatch, not completion.** It is rebuilt from `rejection_dispatched_2`, which is claimed when the co-applicant rejection job reaches SQS — so between that claim and the job finishing, a `refresh()` reports the flag as `1`. If the applicant's remaining 24 fields are already in, the applicant's approval can go out while the co-applicant verdict is still unrecorded. The window is one job's latency wide. Narrowing it means finding a durable *completion* record instead: the `auto_reject_logs` row for `applicant_type = 2` that `autoRejectCoApplicantJSONBRE` writes is the obvious candidate, at the cost of a query per rebuild.
2. **Query-builder writes bypass observers.** `DB::table(...)->update()`, `Model::where(...)->update()`, `Model::insert()`, raw SQL — none fire Eloquent events. Call `refresh()` at such a write site.
3. **Two `ConsentLog` writers are blind.** `ApiController` (`digio_kyc`) and `ConsumerApplicationController` (`self_attested_kyc`) both insert `status = 1` rows without going through the consent document choke point. Since there is no `ConsentLog` observer, `consent_log` self-heals only on the next `refresh()`.
4. **Dispatch is not transaction-aware.** `saved` fires on `save()`, not on commit. Inside `DB::transaction()` a worker can pick the job up before the commit. Laravel 5.5 has no `$afterCommit`.
5. **Exactly-once holds for 90 days**, then the guard expires. Every job needs its own defensive guard (already-rejected, existing credit report) — SQS redelivers regardless.
6. **Co-applicant fields completing *after* consent do not re-trigger.** The tier is only dispatched from the consent choke point, so if the six fields land later, nothing re-evaluates until another consent event. Accepted trade-off for not spending the guard on a pre-consent pull. With the handshake in place this also stalls the applicant's approval.
7. **`cold_start` fires often, and each one sends a WhatsApp message.** The 30-minute TTL means any save after a lull is a cold start, and `changed` fires on every flag flip — so an actively-edited application can message the merchant group repeatedly with a shrinking pending list. `trigger` describes the cache, not a lifecycle stage; it is **not** an idempotency guard. There is currently no dedupe. Options: key a Redis `SETNX` on `(application_id, flags)` inside `notify()`, or only message when the pending set actually shrank.
8. **Redis is a hard dependency for dispatch.** `refresh()`/`dispatchIfReady()` swallow errors and return empty/false, so an outage means no auto-dispatch. Nothing alerts — monitor the `DatapointHelper:` / `Redis-Cache` log lines.
9. **`!empty()` semantics.** `0`, `"0"`, `""`, `null` all read as missing. `co_applicant_monthly_income` is an integer, so a literal 0 reads as missing.
10. **One TCP connection per call.** No pooling; the destructor closes the socket.
11. **Stale fields linger.** `HMSET` only overwrites what it is given, so old hashes may carry renamed flags — `product_id` and `profession_type` among them, since both were replaced. `flagsFrom()` keeps them out of the notification. `DEL` the keys to clean up.
12. **The cold-start notification blocks the caller.** It is a direct `handle()` call, not an SQS push, and it makes a 30-second-timeout cURL to Narad — inside whatever performed the save, including an admin page load (`pending($id, true)` rebuilds an expired hash). Fix by having the cold-start path queue like `changed` does, or by pushing the HTTP call to its own job.
13. **The eligibility memo is dead code.** `self::$eligibility[$cacheKey]` is still written but no longer read — the early return went away when the cache key gained the state. So every `isEligible()` call runs the `merchant` lookup, and the ones with no arguments run the `consumer_application` lookup too. A single observer save reaches it three times (`write()`, `attemptDispatch()`, and the observer's own guarded call), costing roughly five queries where it used to cost one.

---

## Extending

**Add a data point** — confirm it is a real column (`grep -rn "'your_column'" database/migrations | grep -v dropColumn`), add it to the tier array, add a label to `$labels`, add the flag to the owning observer's `$fieldsData`. `buildFromDatabase()`, `refresh()`, `pending()` and the admin lines derive from `tracked()` and need no change.

Backfill: `redis-cli --scan --pattern 'loan:*:datapoints' | xargs -r -n 100 redis-cli DEL`. Safe — guards are untouched — but every one of those applications then reports a `cold_start`.

**Add a writer** — non-Eloquent source: `refresh($id)` then `dispatchIfReady($id)` once your own gates pass. New Eloquent model: copy `ConsumerApplicationElementObserver`, register it, guard `application_id`, wrap in `try/catch`, write through `DatapointHelper::write()`, pass your flags as `$ownFields`.

**Add a tier** — declare the field list, teach `required($tier, $applicantType, $datapoints)` to return it, register it in `$tiers` with a guard prefix no other tier uses, decide its position (priority) and `subjob` list, and make the job accept `['id' => ..., 'applicant_type' => ...]`.

**Co-applicant approval** — define a field list, return it from `required('approval', 2)`, and decide whether `AutoDecisionJob` can serve type 2 (it is applicant-only today).

**Add a post-approval data point** — it is a plain array in `getPostApprovalPendingPoints()`, split by table. Adding one there changes both the edit-screen line and the DCD blocker, since they share the method.

---

## Verifying

```bash
php -l app/Helpers/DatapointHelper.php
php -l app/Jobs/RunRejectionJob.php
php -l app/Jobs/DatapointNotificationJob.php
php -l app/Http/Controllers/Grade/GradeController.php

# nothing bypasses write() — expect only the one inside write()
grep -rn "hmset" app/ | grep -v RawRedisConnector.php

# every tracked field is a real column, and has an observer writer
php artisan tinker
>>> count(App\Helpers\DatapointHelper::buildFromDatabase(<id>));   // expect 31
>>> App\Helpers\DatapointHelper::pendingByApplicant(<id>, true);
```

```bash
redis-cli HGETALL loan:<id>:datapoints
redis-cli TTL     loan:<id>:datapoints          # ~1800
redis-cli DEL     loan:<id>:datapoints          # replay derivation (does NOT re-arm dispatch)

redis-cli HGETALL loan:<id>:dispatch_guard      # what has been sent
redis-cli TTL     loan:<id>:dispatch_guard      # ~7776000
redis-cli HDEL    loan:<id>:dispatch_guard rejection_dispatched_2   # re-arm the co-applicant tier
```

Cases worth checking:

- **Single applicant, both tiers complete on one save** → exactly one SQS message (`AutoDecisionJob`), and *both* `autodecision_dispatched_1` and `rejection_dispatched_1` set (proves subsumption fired instead of a duplicate policy run).
- **Rejection tier alone** — 5 fields in, `offer_id` blank → one `RunRejectionJob`, no `autodecision_dispatched_1`.
- **Co-applicant handshake** — all 24 applicant approval fields in, co-applicant present, then submit consent with `applicant_type = 2` → `RunRejectionJob` with `applicant_type: 2`, `rejection_dispatched_2` set, verdict in `RunRejectionQueueLog`, `state` unchanged. `co_applicant_rejection_executed` becomes `1` and `AutoDecisionJob` follows. Expect the applicant rejection tier to have gone out separately beforehand.
- **The handshake survives a rebuild** — after the above, `DEL loan:<id>:datapoints` and reload the edit page (or capture another consent document). `co_applicant_rejection_executed` must come back `1` and *Co-Applicant Rejection Check Pending* must **not** reappear in the approval line. Then `HDEL loan:<id>:dispatch_guard rejection_dispatched_2`, `DEL` the datapoints key and rebuild: the flag drops back to `0` — that is the re-arm path.
- **Saving the application with all 6 co-applicant fields filled dispatches nothing for type 2** — the observer is applicant-only. This is the point of the design.
- **No co-applicant** → `pendingByApplicant()` has no key `2`; no `rejection_dispatched_2` is ever claimed; the approval line never mentions the co-applicant rejection check.
- **Notification** — `DEL` the datapoints key then save: one `cold_start` entry, no SQS message. Then flip a tracked column: one queued `DatapointNotificationJob`, `trigger: changed`. Then save again without changing anything: no entry. Labels, not raw keys, in the message body.
- **WhatsApp** — with the merchant in `WHATSAPP_BOT_MERCHANT_WHITELIST` and present in `groups.json`, confirm one group message listing the flattened pending fields. Then: remove the merchant from the whitelist → no message, no `DatapointNotificationError`. Then complete every field → no message (nothing pending). Point `NARAD_HOST` at a black hole → the save still completes and `DatapointNotificationError` is logged.
- **Scope gate** — a non-solar merchant's application: **no** datapoints key, no guard, no SQS message, no WhatsApp message, no admin lines. Same for an application in `approved`, and for one created before `RELEASE_DATE`. Setting `DATAPOINT_RELEASE_DATE` into the past should re-enable the last one.
- **Exactly-once survives the datapoints key** — `DEL loan:<id>:datapoints`, refresh, confirm no new SQS message.
- **`AUTO_BRE=database`** (encrypted value) → no SQS dispatch, no notification; `EventAutoDecision` fires from the consent flow only, and **not** from an application save. Admin lines still populate.
- **Consent path** — open `/consent/<hashid>`, submit with location allowed: `consent_log` row with `source = 'admin'`, a `JobProvider` row `from = mconnectConsent` with `is_available = 0`, and after the job runs, the consent PDF on S3 and the data points refreshed. Re-opening the link shows "already logged".
- **DCD blocker** — lender 20, solar merchant, clear `gender` on an `approved` application → the validation popup shows *Post approval data points pending* with `Gender` in the list and the "Next" button does **not** appear. Fill it → the usual loan-agreement / hypothecation / TFR checks resume. Also confirm a case with no Aadhaar number now reaches DCD.
- **E-bill name match** — an application with `ebill_name_verify` on its latest electricity bill and `e_bill_fuzzy_match` set on the active policy shows the pass/fail line with both percentages. Unset the rule → grey line, *"E-Bill Fuzzy Match rule is not configured on the active policy"*. No e-bill → *"E-bill name match not available for the application"*.

---

## Related

- [9.autoDCD.md](9.autoDCD.md) — the auto-decision flow the approval tier feeds
- [10.consent-phase-2.md](10.consent-phase-2.md) — consent capture
- [12.bureau-bre-flow-modification-on-waiting-for-documents.md](12.bureau-bre-flow-modification-on-waiting-for-documents.md)
- [13.table-of-offers-decisioning.md](13.table-of-offers-decisioning.md)