Impact of Model Design on Financial Trust

Here’s a paradox of modern AI that many organizations deploying Large Language Models (LLMs) are discovering: a model can ace a university-level math exam but still struggle with seemingly simple arithmetic. As LLMs become more integrated into daily operations, we've observed that users are naturally attempting to leverage them for complex calculations, like figuring out payment schedules or financial projections. The potential for instant, helpful answers is immense. However, we know that LLMs, for all their conceptual brilliance, can not perform calculations in a fully reliable way. This inherent limitation presents a significant challenge for any financial institution where accuracy is paramount. At Scotiabank, we put this to the test. Through experimentation with hundreds of real-world financial calculation questions, we observed that: • Quite often, the AI's answers were remarkably close, within a dollar. • Crucially, in other cases, the answers were off by tens of dollars. For a bank, "close enough" is never good enough when it impacts a client's financial situation. A minor discrepancy, while seemingly small, can erode the most important thing we build: trust. This led us to a crucial decision regarding our internal AI tools. Rather than risk providing potentially inaccurate financial calculations, we've implemented safeguards. Our LLMs are now prompted to decline direct calculation requests, instead guiding users to established formulas. This story highlights a core principle of our AI strategy, and what we believe should be an industry standard: trust and accuracy above all else. It's not about limiting technology; it’s about deploying it responsibly, with a clear understanding of its current capabilities and inherent risks. The long-term solution for such challenges is already in sight: augmenting LLMs with dedicated, reliable tools for precise calculations, moving us toward a more robust, agentic model where AI can seamlessly leverage external, verified systems. This work is underway across the industry, and we will only roll it out when it’s proven to be precise and reliable.

Years ago, I sent a forecast to the exec team that showed a clean runway into next year. Burn was steady. Pipeline looked strong. Everyone exhaled. Then, in the board meeting, someone asked a simple question: “Why does revenue drop off in November?” I hadn’t noticed. Neither had my analyst. Turns out a single formula was still pointing to an old source tab—with stale assumptions and broken links. The forecast was wrong. Not dramatically wrong. But wrong enough that trust took a hit. And rebuilding trust with your data? That’s expensive. It taught me two things that still shape how I model today: => 1. Bad data breaks more than your model. It breaks confidence. Momentum. Decision-making. Finance doesn’t get a lot of second chances—especially when the numbers are off. => 2. Every model needs a data audit, not just a review. Check your links. Trace your sources. Validate your assumptions. If your inputs are dirty, the whole model is compromised—no matter how clean it looks on the surface. Now I treat models like operating systems. They don’t just need updates—they need maintenance. And sometimes, a reboot. If you've ever gotten burned by a wrong number in the right cell, I see you. You're not alone. But it might be time to put process behind the polish.

How to Lose an Investor in 7 Minutes: Step 1️⃣: Show them a financial model with sky-high returns… and zero margin for error. Step 2️⃣: Use rosy rent growth projections and “forget” rising expenses. Step 3️⃣: Skip the sensitivity analysis—and when they ask about it, say: “Uh… what’s that?” That’s it. Game over. Investors aren’t impressed by colorful graphs. They want to see math that holds up under pressure. Your model isn’t just a sales tool—it’s your first credibility test. Let me give you a real example: I worked with a first-time fund manager who had a great deal under contract. But his financial model? Full of holes: ❌ Unrealistic assumptions ❌ No stress testing ❌ No scenario planning So we rebuilt it from the ground up: ✅ Conservative assumptions on both revenue and expenses ✅ Sensitivity analysis that showed real-world downside protection ✅ Transparent returns grounded in reality—not fantasy The result? 📈 Investors leaned in. 💬 Questions turned into conviction. 💰 Trust turned into wires. Because nothing kills a raise faster than bad math. #CapitalRaising #FundManager #FinancialModeling #InvestorTrust #RealEstateInvesting #PrivateEquity #FundamentalsMatter #RaiseCapitalRight

What Investors Really Want in Your Financial Model Hint: It’s not a 1,000-line spreadsheet. After building investor decks, negotiating credit lines, and raising $20M+, I’ve learned one truth, complex models don’t win deals. Clear ones do. Yet founders still repeat the same mistakes: → Overcomplicating with endless line items.  More detail doesn’t build trust, it breeds confusion. → Painting a perfect picture.  Projections that only go up and to the right ignore reality. Investors care about a few key numbers that tell the real story: → Cash Runway How long can you operate without funding? Investors fund growth—not survival. → Breakeven Point When do you become self-sustaining? This answer signals maturity. → Use of Funds Where is the money going—and why? “Growth” isn’t enough—name the levers. To stand out, focus on these principles: Prioritize Drivers → Identify the 3-5 metrics that move your business (CAC, churn, LTV). Plan for Scenarios → Show Base, Best, and Worst Cases. This isn’t doubt—it’s preparedness. Simplify Ruthlessly → If an investor can’t grasp your model in minutes, it’s too complex. A great financial model isn’t just math—it’s a trust signal. It proves you’re a strategist, not just an operator. P.S I’m putting together a newsletter for finance leaders, operators, and founders who want clear, actionable insights on growth and finance. If that’s you, keep an eye out, launching soon.

📢 New Research Insights! "Design Choices and Machine Learning in Stock Return Prediction" 📈 Keep reading!🔻 👉 A comprehensive study analyzes 1,056 machine learning models for stock return prediction, systematically evaluating design choices across algorithms, target variables, feature selection, and training methods. The research documents how variations in model specifications lead to measurably different outcomes in financial markets. 👉 The analysis reveals that non-standard error from design choices exceeds standard error by 59%, quantifying the methodological impact on results. The effect manifests in monthly returns ranging from 0.13% to 1.98% based on design specifications. This variation is comparable to or exceeds findings in related studies on non-standard errors in finance. 👉 The study presents evidence-based design considerations for implementation. Empirical results indicate that ensemble ML models produce more consistent results than individual algorithms. Market-adjusted returns as targets generate higher raw returns, while CAPM risk-adjusted returns yield improved risk-adjusted metrics. The data suggests continuous targets outperform binary classification approaches. 👉 The research identifies three conditions associated with superior performance of non-linear ML models compared to linear alternatives: the use of market-adjusted returns as targets, implementation of continuous target variables, and application of expanding training windows. These findings contribute to the ongoing discussion of model selection criteria. 👉 This study develops a systematic framework for evaluating design choices in machine learning stock prediction, addressing gaps in methodological standardization. The framework aims to enhance reproducibility and comparison across studies in the field. ----------------------- → Join 3000+ Asset Pricing & Quant Finance enthusiasts who receive top new research ideas weekly in their email: bit.ly/3suSS6e ----------------------- Link to the paper: https://lnkd.in/gzz5ysyw

Have you ever wondered how lenders decide if a borrower is worth the risk? Let me take you behind the scenes of a process that’s as much about precision as it is about trust. Imagine: You’re sitting with a financial model that’s more than just numbers on a spreadsheet it’s a living story about a company’s future. This story guides decisions that could mean millions in funding or none at all. In lending, The stakes are high. Unlike equity investors, lenders don’t benefit if a company does exceptionally well. Our focus is simple yet critical: Will we get the principal and interest back? To figure this out, We don’t just look at one possible future. We create several. Here’s how: -Base Case: This is the borrower’s forecast. Optimistic, but plausible. -Downside 1 and 2: These are the “what ifs. What if sales drop? What if margins shrink? These are the scenarios that keep lenders up at night. But it’s not just about imagining worst-case scenarios it’s about preparing for them. Let’s say a company forecasts 5% sales growth annually. What happens if that growth dips to 2%? Or zero? Will they still have the cash flow to pay down their debt? Why does this matter? Because working capital, taxes, and capital expenses don’t stop just because sales do. The model accounts for it all: -Receivable days. How long does it take for customers to pay up? -Inventory cycles. Are products sitting on shelves too long? -Debt terms. What happens when interest rates rise? One particularly powerful tool in this analysis is a toggle feature. With it, we can flip between scenarios in seconds, testing the model’s resilience to real-world shocks. It’s like a stress test for the future. Imagine you’re planning a road trip. Your Base Case assumes clear skies, smooth roads, and perfect gas mileage. But then, You hit traffic, the weather turns, and you need a pit stop. A good financial model doesn’t just get you there in perfect conditions it ensures you’ll make it even when things go wrong. As I worked through this example, it hit me how much lenders act like navigators. We’re not here to control the company’s journey but to make sure their ship doesn’t sink. And that requires more than formulas it requires empathy, foresight, and a deep understanding of the businesses we partner with. What do you think? If you’ve built or used financial models like this, I’d love to hear your insights. How do you balance optimism with caution in your projections? Let’s dive into this in the comments.

The Alan Turing Institute: The Impact of Large Language Models #LLMs in #Finance: Towards Trustworthy Adoption As large language models (LLMs) evolve AI capabilities for interpreting complex, and often unstructured linguistic text and for their ability to generate engaging human-like language responses, The Alan Turing Institute’s Fair Prosperity Partnership is exploring emerging opportunities for safe, trustworthy adoption within the financial services sector. The Impact of large language Models in Finance: Towards Trustworthy Adoption captures collective insights revealed within a study conducted with the support of colleagues from HSBC, Accenture and the UK’s Financial Conduct Authority (FCA). The work included an extensive literature survey on the impact of LLMs in banking and insights shared by 43 participants who attended a face-to-face workshop examining questions about the likelihood, significance, and timing of the integration of LLMs into financial services. Workshop participants were from major high street banks, regulators, investment banks, insurers, consultancies, payment service providers, and other stakeholders, with the majority revealing that they have begun to employ LLMs for varied internal processes, and to actively assess their potential for market-facing activity. They illustrated a granularity of understanding that is inherently emerging with these deployments which could lead to the development of purpose-specific, auditable models that mitigate many risks stemming from the current lack of ability to predict or explain, and thereby rely on LLM outcomes. Overall, discussions delved into significant potential to tackle persistent concerns and develop strategies that could advance safe adoption of LLMs generally. They also elevated global considerations to be navigated, for example, unfair concentration of services in large organisations with the data to support LLM development or competitive advantage in countries with a favourable regulatory landscape. Recommendations included support for a sector-wide and cross-sector analysis of current use that can elevate best practices, and exploration into opportunities emerging with open-source models specialised in financial tasks, such as FinMA and FinGPT. Link: https://lnkd.in/eynqFm7E

🛡️🏛️ The Hidden and Growing Risks of Third-Party AI Models 🏛️🛡️ ⚡Why Vendor's Model Validation is a Growing Concern? The Federal Reserve's SR 11-7 guidance mandates that financial institutions validate all models, whether built in-house or procured from 3rd party vendors. However, in practice, vendor model validation presents unique challenges for Model Risk Management (MRM) teams, particularly due to their "black box" nature. Many vendors restrict access to their AI models, citing intellectual property concerns. But is this truly about protecting proprietary technology, or is it an excuse to mask flaws and governance gaps? ⚠️ Lack of transparency leaves institutions unable to assess risks fully. ⚡The Growing Challenge with Generative AI (GenAI) Models GenAI models have exacerbated these challenges, with critical aspects often overlooked: 1️⃣ Assumptions & Limitations: Understanding foundational assumptions is crucial for assessing a model’s applicability and reliability. 2️⃣ Data Inputs & Parameters: Knowing input sources and parameter settings is key to evaluating robustness and relevance. 3️⃣ Explainability: Clear explanations of model design and analytics help stakeholders trust and effectively use the model. 👉 Open-source initiatives like Meta’s Llama 3 represent major steps toward transparency. By making model weights publicly available, Meta has enabled greater scrutiny, collaboration, and ultimately, more trustworthy AI. 💡 How Risk Teams Can Strengthen Vendor Model Validation? 🔹 Develop Specialized Expertise – AI model validation requires domain-specific knowledge. If in-house expertise is lacking, consider training teams or engaging third-party validators. 🔹 Enforce SR 11-7 Compliance in Vendor Contracts – Require transparency on model components, design, intended use, assumptions, and limitations to ensure alignment with risk policies. 🔹 Document Model Use – Maintain internal documentation covering inputs, outputs, key assumptions, and vendor-provided details to support audits and compliance. 🔹 Validate Independently – Review vendor testing results and conduct additional testing where feasible to verify performance and identify risks. 🔹 Assess Data Sources – Scrutinize input data quality, completeness, and appropriateness, particularly for LLMs, to mitigate data transparency and copyright concerns. 💡 Final Thoughts The financial industry is undergoing a transformative period with the rapid adoption of AI models, driven by promises of efficiency gains. However, this progression must align with robust governance standards. ⚠️ Major commercial vendors often prioritize performance, sometimes at the expense of transparency and comprehensive real-world testing. 👉 It is incumbent upon risk teams to implement appropriate guardrails and advocate for a more transparent and open approach to model validation, ensuring that innovation does not compromise integrity and reliability.