Tech-Driven Performance Reviews

New! We’ve published a new set of automated evaluations and benchmarks for RAG - a critical component of Gen AI used by most successful customers today. Sweet. Retrieval-Augmented Generation lets you take general-purpose foundation models - like those from Anthropic, Meta, and Mistral - and “ground” their responses in specific target areas or domains using information which the models haven’t seen before (maybe confidential, private info, new or real-time data, etc). This lets gen AI apps generate responses which are targeted to that domain with better accuracy, context, reasoning, and depth of knowledge than the model provides off the shelf. In this new paper, we describe a way to evaluate task-specific RAG approaches such that they can be benchmarked and compared against real-world uses, automatically. It’s an entirely novel approach, and one we think will help customers tune and improve their AI apps much more quickly, and efficiently. Driving up accuracy, while driving down the time it takes to build a reliable, coherent system. 🔎 The evaluation is tailored to a particular knowledge domain or subject area. For example, the paper describes tasks related to DevOps troubleshooting, scientific research (ArXiv abstracts), technical Q&A (StackExchange), and financial reporting (SEC filings). 📝 Each task is defined by a specific corpus of documents relevant to that domain. The evaluation questions are generated from and grounded in this corpus. 📊 The evaluation assesses the RAG system's ability to perform specific functions within that domain, such as answering questions, solving problems, or providing relevant information based on the given corpus. 🌎 The tasks are designed to mirror real-world scenarios and questions that might be encountered when using a RAG system in practical applications within that domain. 🔬 Unlike general language model benchmarks, these task-specific evaluations focus on the RAG system's performance in retrieving and applying information from the given corpus to answer domain-specific questions. ✍️ The approach allows for creating evaluations for any task that can be defined by a corpus of relevant documents, making it adaptable to a wide range of specific use cases and industries. Really interesting work from the Amazon science team, and a new totem of evaluation for customers choosing and tuning their RAG systems. Very cool. Paper linked below.

OpenAI CPO: Evals are becoming a core skill for PMs. PM in 2025 is changing fast. PMs need to learn brand new skills: 1. AI Evals (https://lnkd.in/eGbzWMxf) 2. AI PRDs (https://lnkd.in/eMu59p_z) 3. AI Strategy (https://lnkd.in/egemMhMF) 4. AI Discovery (https://lnkd.in/e7Q6mMpc) 5. AI Prototyping (https://lnkd.in/eJujDhBV) And evals is amongst the deepest topics. There's 3 steps to them: 1. Observing (https://lnkd.in/e3eQBdMp) 2. Analyzing Errors (https://lnkd.in/eEG83W5D) 3. Building LLM Judges (https://lnkd.in/ez3stJRm) - - - - - - Here's your simple guide to evals in 5 minutes: (Repost this before anything else ♻️) 𝟭. 𝗕𝗼𝗼𝘁𝘀𝘁𝗿𝗮𝗽 𝗬𝗼𝘂𝗿 𝗗𝗮𝘁𝗮𝘀𝗲𝘁 Start with 100 diverse traces of your LLM pipeline. Use real data if you can, or systematic synthetic data generation across key dimensions if you can't. Quality over quantity here: aggressive filtering beats volume. 𝟮. 𝗔𝗻𝗮𝗹𝘆𝘇𝗲 𝗧𝗵𝗿𝗼𝘂𝗴𝗵 𝗢𝗽𝗲𝗻 𝗖𝗼𝗱𝗶𝗻𝗴 Read every trace carefully and label failure modes without preconceptions. Look for the first upstream failure in each trace. Continue until you hit theoretical saturation, when new traces reveal no fundamentally new error types. 𝟯. 𝗦𝘁𝗿𝘂𝗰𝘁𝘂𝗿𝗲 𝗬𝗼𝘂𝗿 𝗙𝗮𝗶𝗹𝘂𝗿𝗲 𝗠𝗼𝗱𝗲𝘀 Group similar failures into coherent, binary categories through axial coding. Focus on Gulf of Generalization failures (where clear instructions are misapplied) rather than Gulf of Specification issues (ambiguous prompts you can fix easily). 𝟰. 𝗕𝘂𝗶𝗹𝗱 𝗔𝘂𝘁𝗼𝗺𝗮𝘁𝗲𝗱 𝗘𝘃𝗮𝗹𝘂𝗮𝘁𝗼𝗿𝘀 Create dedicated evaluators for each failure mode. Use code-based checks when possible (regex, schema validation, execution tests). For subjective judgments, build LLM-as-Judge evaluators with clear Pass/Fail criteria, few-shot examples, and structured JSON outputs. 𝟱. 𝗗𝗲𝗽𝗹𝗼𝘆 𝘁𝗵𝗲 𝗜𝗺𝗽𝗿𝗼𝘃𝗲𝗺𝗲𝗻𝘁 𝗙𝗹𝘆𝘄𝗵𝗲𝗲𝗹 Integrate evals into CI/CD, monitor production with bias-corrected success rates, and cycle through Analyze→ Measure→ Improve continuously. New failure modes in production feed back into your evaluation artifacts. Evals are now a core skill for AI PMs. This is your map. - - - - - I learned this from Hamel Husain and Shreya Shankar. Get 35% off their course: https://lnkd.in/e5DSNJtM 📌 Want our step-by-step guide to evals? Comment 'steps' + DM me. Repost to cut the line. ➕ Follow Aakash Gupta to stay on top of AI x PM.

Explaining the Evaluation method LLM-as-a-Judge (LLMaaJ). Token-based metrics like BLEU or ROUGE are still useful for structured tasks like translation or summarization. But for open-ended answers, RAG copilots, or complex enterprise prompts, they often miss the bigger picture. That’s where LLMaaJ changes the game. 𝗪𝗵𝗮𝘁 𝗶𝘀 𝗶𝘁? You use a powerful LLM as an evaluator, not a generator. It’s given: - The original question - The generated answer - And the retrieved context or gold answer 𝗧𝗵𝗲𝗻 𝗶𝘁 𝗮𝘀𝘀𝗲𝘀𝘀𝗲𝘀: ✅ Faithfulness to the source ✅ Factual accuracy ✅ Semantic alignment—even if phrased differently 𝗪𝗵𝘆 𝘁𝗵𝗶𝘀 𝗺𝗮𝘁𝘁𝗲𝗿𝘀: LLMaaJ captures what traditional metrics can’t. It understands paraphrasing. It flags hallucinations. It mirrors human judgment, which is critical when deploying GenAI systems in the enterprise. 𝗖𝗼𝗺𝗺𝗼𝗻 𝗟𝗟𝗠𝗮𝗮𝗝-𝗯𝗮𝘀𝗲𝗱 𝗺𝗲𝘁𝗿𝗶𝗰𝘀: - Answer correctness - Answer faithfulness - Coherence, tone, and even reasoning quality 📌 If you’re building enterprise-grade copilots or RAG workflows, LLMaaJ is how you scale QA beyond manual reviews. To put LLMaaJ into practice, check out EvalAssist; a new tool from IBM Research. It offers a web-based UI to streamline LLM evaluations: - Refine your criteria iteratively using Unitxt - Generate structured evaluations - Export as Jupyter notebooks to scale effortlessly A powerful way to bring LLM-as-a-Judge into your QA stack. - Get Started guide: https://lnkd.in/g4QP3-Ue - Demo Site: https://lnkd.in/gUSrV65s - Github Repo: https://lnkd.in/gPVEQRtv - Whitepapers: https://lnkd.in/gnHi6SeW

𝐃𝐞𝐬𝐩𝐢𝐭𝐞 𝐭𝐡𝐞 𝐬𝐤𝐞𝐩𝐭𝐢𝐜𝐬, 𝐥𝐞𝐚𝐝𝐞𝐫𝐬 𝐦𝐮𝐬𝐭 𝐞𝐦𝐛𝐫𝐚𝐜𝐞 𝐭𝐰𝐨 𝐆𝐞𝐧𝐀𝐈-𝐝𝐫𝐢𝐯𝐞𝐧 𝐬𝐭𝐫𝐚𝐭𝐞𝐠𝐢𝐞𝐬 𝐭𝐨 𝐞𝐥𝐞𝐯𝐚𝐭𝐞 𝐞𝐦𝐩𝐥𝐨𝐲𝐞𝐞 𝐏𝐞𝐫𝐟𝐨𝐫𝐦𝐚𝐧𝐜𝐞 𝐑𝐞𝐯𝐢𝐞𝐰𝐬 ! 🤔 HR leaders dipping their toes into GenAI for performance management often hit pause—new tech brings compliance questions, and 52% of HRBPs admit their organizations aren’t yet ready to implement AI in reviews. 🔍 The disconnect? GenAI tools tend to be rigid—embedded deep inside Performance Management systems with limited flexibility, not offered as agile add-ons. 💡 It’s time for talent management leaders to rethink their approach. How can we harness emerging GenAI capabilities to help CHROs deliver efficient, data-driven, and unbiased performance reviews? 🚀 By embedding customizable GenAI features into PM platforms, we unlock: ⏱️ Manager time savings 📊 Deeper, objective insights 🎨 Tailored workflows that fit your culture 🙏 Yet only 44% of employees feel at ease letting AI run feedback sessions solo. Trust, transparency, and the right human touch remain non-negotiable, according to a new interesting research published by Gartner using data 📊 from their internal clients. Researchers highlight a pivotal choice for talent leaders ready to harness GenAI in performance management: ➡️ 𝐀𝐩𝐩𝐫𝐨𝐚𝐜𝐡 𝟏— Evaluate add-on GenAI tools that could be applied to support performance management outcomes. ➡️ 𝐀𝐩𝐩𝐫𝐨𝐚𝐜𝐡 𝟐 — Focus on process maturity to improve the potential of embedded GenAI features. Researchers revealed that add-on GenAI capabilities that could address common pain points include: 1️⃣ Overlay text evaluation for bias detection 2️⃣ Summarization of feedback and performance data points from multiple HR systems 3️⃣ Manager coaching support 4️⃣ Skills-based development planning ☝️ 𝙈𝙮 𝙥𝙚𝙧𝙨𝙤𝙣𝙖𝙡 𝙫𝙞𝙚𝙬: I’m energized by these findings—GenAI is poised to transform performance reviews with speed, data-rich insights, and workflows that can bend to our culture. But here’s the real deal: only 44% of employees trust AI to go solo. Numbers and algorithms can’t capture ambition, context, or the “why” behind someone’s performance. The future of reviews must be hybrid. Let AI flag patterns and surface objective measures—then let managers bring empathy, storytelling, and developmental nuance. That human touch isn’t a “nice-to-have,” it’s non-negotiable. Thank you 🙏 Gartner for HR researchers team for these insightful findings: Laura Gardiner ✍️ How are you striking the balance between GenAI innovation and employee confidence in your performance process? ———————————— ♻️ Share to empower HR professionals and elevate excellence in 2025! 💡 Follow Nicolas BEHBAHANI for more insights on HR, People Analytics & the future of work! #Talent #PerformanceReview #GenAI #PerformanceManagement

Evaluating LLMs is not like testing traditional software. Traditional systems are deterministic → pass/fail. LLMs are probabilistic → same input, different outputs, shifting behaviors over time. That makes model selection and monitoring one of the hardest engineering problems today. This is where Eval Protocol (EP) developed by Fireworks AI is so powerful. It’s an open-source framework for building an internal model leaderboard, where you can define, run, and track evals that actually reflect your business needs. → Simulated Users – generate synthetic but realistic user interactions to stress-test models under lifelike conditions. → evaluation_test – pytest-compatible evals (pointwise, groupwise, all) so you can treat model behavior like unit tests in CI/CD. → MCP Extensions – evaluate agents that use tools, multi-step reasoning, or multi-turn dialogue via Model Context Protocol. → UI Review – a dashboard to visualize eval results, compare across models, and catch regressions before they ship. Instead of relying on generic benchmarks, EP lets you encode your own success criteria and continuously measure models against them. If you’re serious about scaling LLMs in production, this is worth a look: evalprotocol.io

User Feedback Loops: the missing piece in AI success? AI is only as good as the data it learns from -- but what happens after deployment? Many businesses focus on building AI products but miss a critical step: ensuring their outputs continue to improve with real-world use. Without a structured feedback loop, AI risks stagnating, delivering outdated insights, or losing relevance quickly. Instead of treating AI as a one-and-done solution, companies need workflows that continuously refine and adapt based on actual usage. That means capturing how users interact with AI outputs, where it succeeds, and where it fails. At Human Managed, we’ve embedded real-time feedback loops into our products, allowing customers to rate and review AI-generated intelligence. Users can flag insights as: 🔘Irrelevant 🔘Inaccurate 🔘Not Useful 🔘Others Every input is fed back into our system to fine-tune recommendations, improve accuracy, and enhance relevance over time. This is more than a quality check -- it’s a competitive advantage. - for CEOs & Product Leaders: AI-powered services that evolve with user behavior create stickier, high-retention experiences. - for Data Leaders: Dynamic feedback loops ensure AI systems stay aligned with shifting business realities. - for Cybersecurity & Compliance Teams: User validation enhances AI-driven threat detection, reducing false positives and improving response accuracy. An AI model that never learns from its users is already outdated. The best AI isn’t just trained -- it continuously evolves.

She was one of our brightest talents Smart. Committed. A quiet force that lifted the whole team And then... she resigned No warning. No second thoughts. Just… gone. We were stunned. She had everything: a promising future, fair pay, great feedback. So we asked her why. Her words hit like a punch: "I didn’t feel seen. I didn’t feel like we mattered." That moment changed everything. Because the truth is, we missed the signs: - Her engagement score had dropped - Her internal applications went nowhere - She kept going the extra mile with no recognition We had the data. We just didn’t use it wisely. Today, we have no excuse. AI and predictive analytics give us a head start. They help us spot patterns before they become problems: - Who might be silently disengaging? - Where are we overlooking skills and potential? - Are we creating an inclusive space where everyone feels they belong? This isn’t about replacing human connection, it’s about deepening it. When we pair data with empathy, we lead smarter, faster, and more human. Because great HR doesn’t just prevent risks. It unlocks possibility. If we reinforce our data and tools, we can spend even more time on what matters most: making sure people remain at the heart of our organizations. #Talents #PredictiveHR #DataDrivenLeadership #EmployeeExperience #humanresources

That’s the thing about feedback—you can’t just ask for it once and call it a day. I learned this the hard way. Early on, I’d send out surveys after product launches, thinking I was doing enough. But here’s what happened: responses trickled in, and the insights felt either outdated or too general by the time we acted on them. It hit me: feedback isn’t a one-time event—it’s an ongoing process, and that’s where feedback loops come into play. A feedback loop is a system where you consistently collect, analyze, and act on customer insights. It’s not just about gathering input but creating an ongoing dialogue that shapes your product, service, or messaging architecture in real-time. When done right, feedback loops build emotional resonance with your audience. They show customers you’re not just listening—you’re evolving based on what they need. How can you build effective feedback loops? → Embed feedback opportunities into the customer journey: Don’t wait until the end of a cycle to ask for input. Include feedback points within key moments—like after onboarding, post-purchase, or following customer support interactions. These micro-moments keep the loop alive and relevant. → Leverage multiple channels for input: People share feedback differently. Use a mix of surveys, live chat, community polls, and social media listening to capture diverse perspectives. This enriches your feedback loop with varied insights. → Automate small, actionable nudges: Implement automated follow-ups asking users to rate their experience or suggest improvements. This not only gathers real-time data but also fosters a culture of continuous improvement. But here’s the challenge—feedback loops can easily become overwhelming. When you’re swimming in data, it’s tough to decide what to act on, and there’s always the risk of analysis paralysis. Here’s how you manage it: → Define the building blocks of useful feedback: Prioritize feedback that aligns with your brand’s goals or messaging architecture. Not every suggestion needs action—focus on trends that impact customer experience or growth. → Close the loop publicly: When customers see their input being acted upon, they feel heard. Announce product improvements or service changes driven by customer feedback. It builds trust and strengthens emotional resonance. → Involve your team in the loop: Feedback isn’t just for customer support or marketing—it’s a company-wide asset. Use feedback loops to align cross-functional teams, ensuring insights flow seamlessly between product, marketing, and operations. When feedback becomes a living system, it shifts from being a reactive task to a proactive strategy. It’s not just about gathering opinions—it’s about creating a continuous conversation that shapes your brand in real-time. And as we’ve learned, that’s where real value lies—building something dynamic, adaptive, and truly connected to your audience. #storytelling #marketing #customermarketing

Evaluating Retrieval-Augmented Generation (RAG) systems has long been a challenge, given the complexity and subjectivity of long-form responses. A recent collaborative research paper from institutions including the University of Waterloo, Microsoft, and Snowflake presents a promising solution: the AutoNuggetizer framework. This innovative approach leverages Large Language Models (LLMs) to automate the "nugget evaluation methodology," initially proposed by TREC in 2003 for assessing responses to complex questions. Here's a technical breakdown of how it works under the hood: 1. Nugget Creation:   - Initially, LLMs automatically extract "nuggets," or atomic pieces of essential information, from a set of related documents.   - Nuggets are classified as "vital" (must-have) or "okay" (nice-to-have) based on their importance in a comprehensive response.   - An iterative prompt-based approach using GPT-4o ensures the nuggets are diverse and cover different informational facets. 2. Nugget Assignment:   - LLMs then automatically evaluate each system-generated response, assigning nuggets as "support," "partial support," or "no support."   - This semantic evaluation allows the model to recognize supported facts even without direct lexical matching. 3. Evaluation and Correlation:   - Automated evaluation scores strongly correlated with manual evaluations, particularly at the system-run level, suggesting this methodology could scale efficiently for broad usage.   - Interestingly, the automation of nugget assignment alone significantly increased alignment with manual evaluations, highlighting its potential as a cost-effective evaluation approach. Through rigorous validation against human annotations, the AutoNuggetizer framework demonstrates a practical balance between automation and evaluation quality, providing a scalable, accurate method to advance RAG system evaluation. The research underscores not just the potential of automating complex evaluations, but also opens avenues for future improvements in RAG systems.

‘AI Evals’ explained: what they are and how to write them 📚 Like traditional PMs write product specs, AI PMs are supposed to write AI evals. In traditional products, you ship a feature → track usage → optimize based on feedback. But with AI-powered features, your model’s output is the product. So you need to test it before launch with structured evaluations. That’s where AI Evals come in. 𝗪𝗵𝗮𝘁 𝗶𝘀 𝗮𝗻 𝗔𝗜 𝗘𝘃𝗮𝗹? An AI Evaluation is a test suite that helps you measure how well your AI model performs on specific tasks using real or synthetic data. 𝗬𝗼𝘂 𝘂𝘀𝗲 𝗶𝘁 𝘁𝗼: 🔹Compare models (e.g., GPT-4 vs Claude on your use case) 🔹Validate prompt chains or agents 🔹Detect failure cases 🔹Track quality regressions over time Think of it as writing unit tests but for LLM outputs. 𝗛𝗼𝘄 𝘁𝗼 𝘄𝗿𝗶𝘁𝗲 𝗮 𝗴𝗼𝗼𝗱 𝗔𝗜 𝗘𝘃𝗮𝗹 𝗮𝘀 𝗮 𝗣𝗠? A simple structure structure of AI eval would include 2 aspects: 1. Component 2. What do they define Run this over 100s of test cases → analyze failure patterns → tune prompts or switch models. Below table explains it clearly with examples 👇 𝗧𝗵𝗲𝘆 𝗮𝗿𝗲 𝘃𝗲𝗿𝘆 𝗶𝗺𝗽𝗼𝗿𝘁𝗮𝗻𝘁 𝗳𝗼𝗿 𝗣𝗠𝘀 𝗯𝘂𝗶𝗹𝗱𝗶𝗻𝗴 𝗔𝗜 𝗳𝗲𝗮𝘁𝘂𝗿𝗲𝘀, 𝗮𝘀 𝗶𝘁: ✅ Helps PMs validate AI product quality before launch ✅ Forces clarity on what “good” output looks like ✅ Saves time vs launching and learning from real users’ frustration ✅ Helps track quality when prompts/models change over time Even if you're not an AI PM yet, AI Evals are becoming crucial to understand with the shift coming up in digital products. P.S. Let me know if you want me to create a detailed guide around AI evals!