Companion note to Financial Advisory LLM Model for Modernizing Financial Services and Innovative Solutions for Financial Literacy in India.
Ask any large public model “how much tax do I owe on ₹18 lakh in Chennai?” and something strange happens. The reply is in the right currency, sometimes. The tax brackets are American, usually. Somewhere in there is a mention of a 401(k). Occasionally a Roth IRA turns up like a tourist who wandered into the wrong shop.
The model isn’t stupid. Its training distribution is 90% American. When you ask about Indian finance, you’re asking it to work in the long tail — and it does what a model on the long tail always does. It confabulates plausibly from the fat part of the distribution.
Here’s what “plausibly wrong” looks like at three different salary levels.
Same question, two very different answers
A user in Chennai types the same question at a generic assistant and at our Indian-context model. Move the salary slider — watch the answers diverge.
How this demo works
The tax numbers are computed live using FY 2024-25 Indian slab rates for the two regimes: the old regime with ₹1.5L 80C + ₹50k standard deduction assumed, and the new regime with the higher standard deduction and no exemptions. The generic-model panel is a stylized composite of the common failure modes we observed in generic assistants — USD framing, US-tax reasoning, or generic "consult an advisor" hedging.
Move the salary slider. Toggle between old and new regime. Watch the left panel keep quoting US brackets and the right panel actually compute the correct Indian tax under whichever regime you picked. The difference isn’t fluency — both models sound confident. The difference is whether they’re operating in the right country.
What we actually built
A three-part stack, each part solving a specific failure mode of the generic model.
1. A small, dense instruction dataset
Not big. A few thousand carefully written examples covering the core Indian personal-finance surface:
- Sections 80C, 80D — tax-saving instruments and health-insurance deductions.
- Old vs new regime — the choice, the trade-offs, when each is optimal.
- HRA, LTA, NPS — salary components that a US-tuned model has literally never heard of at scale.
- Capital gains — LTCG vs STCG, indexation, ₹1L exempt threshold on equity.
- PPF, EPF, ELSS, SIPs, ULIPs — the actual products Indian users buy.
Every example was hand-checked for correctness by someone who files an ITR themselves. That last part is unglamorous and non-negotiable — you cannot bootstrap financial correctness from GPT-generated examples, because the base model gets it wrong and you’d be baking those errors into your fine-tuned model.
2. Retrieval over a dated, sourced corpus
The Income Tax Act changes. Every budget shifts something. If your model paraphrases a rule from its training weights, it will confidently quote the 2022 rule to a 2025 user.
The retrieval corpus is small and dated — sections of the ITA, SEBI circulars, RBI notifications, all timestamped. The model quotes law from retrieved passages instead of paraphrasing from weights. This is boring engineering that matters way more than it should.
3. Refusal on advice that would need a SEBI-registered person
This is the one I care about most. The model is not licensed. It never will be. If a user asks “should I put my ₹5L bonus into ELSS or debt funds?”, the model:
- Explains what each vehicle is.
- Shows the tax treatment of each.
- Refuses to say which one they should pick.
That refusal is trained. It’s easy to make an LLM helpful. It’s much harder to make it helpfully draw a line and stop.
The evaluation is where I sweated
Standard NLP benchmarks are useless here. A model that gets 90% on Indian-Finance-QA might be catastrophically wrong on the 10% where it hallucinates a tax rule.
We ran three eval axes:
- Numerical correctness. Given a scenario (salary, deductions, regime), does the model compute the right tax to within a rupee? This one is mechanical and easy to grade.
- Legal grounding. Does every rule the model quotes correspond to a real, current section? Human graders with tax knowledge. Slow, painful, necessary.
- Refusal calibration. On advice questions, does the model actually refuse? A model that refuses too much is useless; a model that refuses too little is dangerous.
The number I’d shout from the rooftop: numerical accuracy went from ~54% (base model, prompted) to ~91% (fine-tuned + retrieval). Legal grounding is harder to summarize but roughly doubled.
The thing I’m least confident about
Financial literacy is downstream of trust. A model that gives correct answers to 91% of questions is a great product. It is not, by itself, a movement in financial literacy in India.
The users I most want to reach — first-time earners, gig workers, people whose parents didn’t have brokerage accounts — need an interface that meets them where they are. That’s product design, vernacular localization, and a lot of user research. The paper is about the model. The literacy question is about everything around the model, and we barely touched it.
What I’d do next
- Vernacular support. Hindi, Tamil, Marathi, at minimum. Not translation of the English output — retrieval and generation directly in the language.
- Voice interface. A large fraction of the target user base uses WhatsApp voice notes for everything else. The finance model should meet them there.
- Longitudinal case-based eval. Instead of one-shot Q&A, evaluate the model over a multi-turn conversation that resembles a real tax-planning session. That’s where hedging and consistency really get tested.
Full paper: Research Square.