Automated Evaluation Tools

Explaining the Evaluation method LLM-as-a-Judge (LLMaaJ). Token-based metrics like BLEU or ROUGE are still useful for structured tasks like translation or summarization. But for open-ended answers, RAG copilots, or complex enterprise prompts, they often miss the bigger picture. That’s where LLMaaJ changes the game. 𝗪𝗵𝗮𝘁 𝗶𝘀 𝗶𝘁? You use a powerful LLM as an evaluator, not a generator. It’s given: - The original question - The generated answer - And the retrieved context or gold answer 𝗧𝗵𝗲𝗻 𝗶𝘁 𝗮𝘀𝘀𝗲𝘀𝘀𝗲𝘀: ✅ Faithfulness to the source ✅ Factual accuracy ✅ Semantic alignment—even if phrased differently 𝗪𝗵𝘆 𝘁𝗵𝗶𝘀 𝗺𝗮𝘁𝘁𝗲𝗿𝘀: LLMaaJ captures what traditional metrics can’t. It understands paraphrasing. It flags hallucinations. It mirrors human judgment, which is critical when deploying GenAI systems in the enterprise. 𝗖𝗼𝗺𝗺𝗼𝗻 𝗟𝗟𝗠𝗮𝗮𝗝-𝗯𝗮𝘀𝗲𝗱 𝗺𝗲𝘁𝗿𝗶𝗰𝘀: - Answer correctness - Answer faithfulness - Coherence, tone, and even reasoning quality 📌 If you’re building enterprise-grade copilots or RAG workflows, LLMaaJ is how you scale QA beyond manual reviews. To put LLMaaJ into practice, check out EvalAssist; a new tool from IBM Research. It offers a web-based UI to streamline LLM evaluations: - Refine your criteria iteratively using Unitxt - Generate structured evaluations - Export as Jupyter notebooks to scale effortlessly A powerful way to bring LLM-as-a-Judge into your QA stack. - Get Started guide: https://lnkd.in/g4QP3-Ue - Demo Site: https://lnkd.in/gUSrV65s - Github Repo: https://lnkd.in/gPVEQRtv - Whitepapers: https://lnkd.in/gnHi6SeW

Evaluating LLMs is not like testing traditional software. Traditional systems are deterministic → pass/fail. LLMs are probabilistic → same input, different outputs, shifting behaviors over time. That makes model selection and monitoring one of the hardest engineering problems today. This is where Eval Protocol (EP) developed by Fireworks AI is so powerful. It’s an open-source framework for building an internal model leaderboard, where you can define, run, and track evals that actually reflect your business needs. → Simulated Users – generate synthetic but realistic user interactions to stress-test models under lifelike conditions. → evaluation_test – pytest-compatible evals (pointwise, groupwise, all) so you can treat model behavior like unit tests in CI/CD. → MCP Extensions – evaluate agents that use tools, multi-step reasoning, or multi-turn dialogue via Model Context Protocol. → UI Review – a dashboard to visualize eval results, compare across models, and catch regressions before they ship. Instead of relying on generic benchmarks, EP lets you encode your own success criteria and continuously measure models against them. If you’re serious about scaling LLMs in production, this is worth a look: evalprotocol.io

Evaluating LLMs accurately/reliably is difficult, but we can usually automate the evaluation process with another (more powerful) LLM... Automatic metrics: Previously, generative text models were most commonly evaluated using automatic metrics like ROUGE and BLEU, which simply compare how well a model’s output matches a human-written target resopnse. In particular, BLEU score was commonly used to evaluatate machine translation models, while ROUGE was most often used for evaluating summarization models. Serious limitations: With modern LLMs, researchers began to notice that automatic metrics did a poor job of comprehensively capturing the quality of an LLM’s generations. Oftentimes, ROUGE scores were poorly correlated with human preferences—higher scores don’t seem to indicate a better generation/summary [1]. This problem is largely due to the open-ended nature of most tasks solved with LLMs. There can be many good responses to a prompt. LLM-as-a-judge [2] leverages a powerful LLM (e.g., GPT-4) to evaluate the quality of an LLM’s output. To evaluate an LLM with another LLM, there are three basic structures or strategies that we can employ: (1) Pairwise comparison: The LLM is shown a question with two responses and asked to choose the better response (or declare a tie). This approach was heavily utilized by models like Alpaca/Vicuna to evaluate model performance relative to proprietary LLMS like ChatGPT. (2) Single-answer grading: The LLM is shown a response with a single answer and asked to provide a score for the answer. This strategy is less reliable than pairwise comparison due to the need to assign an absolute score to the response. However, authors in [2] observe that GPT-4 can nonetheless assign relatively reliable/meaningful scores to responses. (3) Reference-guided grading: The LLM is provided a reference answer to the problem when being asked to grade a response. This strategy is useful for complex problems (e.g., reasoning or math) in which even GPT-4 may struggle with generating a correct answer. In these cases, having direct access to a correct response may aid the grading process. “LLM-as-a-judge offers two key benefits: scalability and explainability. It reduces the need for human involvement, enabling scalable benchmarks and fast iterations.” - from [2] Using MT-bench, authors in [2] evaluate the level of agreement between LLM-as-a-judge and humans (58 expert human annotators), where we see that there is a high level of agreement between these strategies. Such a finding caused this evaluation strategy to become incredibly popular for LLMs—it is currently the most widely-used and effective alternative to human evaluation. However, LLM-as-a-judge does suffer from notable limititations (e.g., position bias, verbosity bias, self-enhancement bias, etc.) that should be considered when interpretting data.

New! We’ve published a new set of automated evaluations and benchmarks for RAG - a critical component of Gen AI used by most successful customers today. Sweet. Retrieval-Augmented Generation lets you take general-purpose foundation models - like those from Anthropic, Meta, and Mistral - and “ground” their responses in specific target areas or domains using information which the models haven’t seen before (maybe confidential, private info, new or real-time data, etc). This lets gen AI apps generate responses which are targeted to that domain with better accuracy, context, reasoning, and depth of knowledge than the model provides off the shelf. In this new paper, we describe a way to evaluate task-specific RAG approaches such that they can be benchmarked and compared against real-world uses, automatically. It’s an entirely novel approach, and one we think will help customers tune and improve their AI apps much more quickly, and efficiently. Driving up accuracy, while driving down the time it takes to build a reliable, coherent system. 🔎 The evaluation is tailored to a particular knowledge domain or subject area. For example, the paper describes tasks related to DevOps troubleshooting, scientific research (ArXiv abstracts), technical Q&A (StackExchange), and financial reporting (SEC filings). 📝 Each task is defined by a specific corpus of documents relevant to that domain. The evaluation questions are generated from and grounded in this corpus. 📊 The evaluation assesses the RAG system's ability to perform specific functions within that domain, such as answering questions, solving problems, or providing relevant information based on the given corpus. 🌎 The tasks are designed to mirror real-world scenarios and questions that might be encountered when using a RAG system in practical applications within that domain. 🔬 Unlike general language model benchmarks, these task-specific evaluations focus on the RAG system's performance in retrieving and applying information from the given corpus to answer domain-specific questions. ✍️ The approach allows for creating evaluations for any task that can be defined by a corpus of relevant documents, making it adaptable to a wide range of specific use cases and industries. Really interesting work from the Amazon science team, and a new totem of evaluation for customers choosing and tuning their RAG systems. Very cool. Paper linked below.

Evaluating Retrieval-Augmented Generation (RAG) systems has long been a challenge, given the complexity and subjectivity of long-form responses. A recent collaborative research paper from institutions including the University of Waterloo, Microsoft, and Snowflake presents a promising solution: the AutoNuggetizer framework. This innovative approach leverages Large Language Models (LLMs) to automate the "nugget evaluation methodology," initially proposed by TREC in 2003 for assessing responses to complex questions. Here's a technical breakdown of how it works under the hood: 1. Nugget Creation:   - Initially, LLMs automatically extract "nuggets," or atomic pieces of essential information, from a set of related documents.   - Nuggets are classified as "vital" (must-have) or "okay" (nice-to-have) based on their importance in a comprehensive response.   - An iterative prompt-based approach using GPT-4o ensures the nuggets are diverse and cover different informational facets. 2. Nugget Assignment:   - LLMs then automatically evaluate each system-generated response, assigning nuggets as "support," "partial support," or "no support."   - This semantic evaluation allows the model to recognize supported facts even without direct lexical matching. 3. Evaluation and Correlation:   - Automated evaluation scores strongly correlated with manual evaluations, particularly at the system-run level, suggesting this methodology could scale efficiently for broad usage.   - Interestingly, the automation of nugget assignment alone significantly increased alignment with manual evaluations, highlighting its potential as a cost-effective evaluation approach. Through rigorous validation against human annotations, the AutoNuggetizer framework demonstrates a practical balance between automation and evaluation quality, providing a scalable, accurate method to advance RAG system evaluation. The research underscores not just the potential of automating complex evaluations, but also opens avenues for future improvements in RAG systems.

Use MLflow for efficient LLM evaluations: automate processes, standardize experiments, and achieve reproducible results with comprehensive tracking and versatile metrics. Managing Large Language Model (LLM) experiments can be complex. Juggling numerous prompts, refining parameters, and tracking best results can be tedious and time-consuming. MLflow's LLM evaluation tools provide a powerful and efficient solution, featuring: - Comprehensive tracking: Log prompts, parameters, and outputs seamlessly for effortless review and comparison. - Versatile evaluation: Support diverse LLM types, models, and even Python callables. - Predefined metrics: Simplify tasks with built-in metrics for common LLM tasks such as question answering and summarization. - Custom metrics: Craft unique metrics tailored to your specific needs. LLM-as-judge metrics allow you to develop highly-specific custom metrics tailored to your use case. - Static dataset evaluation: Evaluate saved model outputs without rerunning the model. - Integrated results: Gain clear insights through comprehensive results viewable directly in code or in the MLflow UI. Some of the main benefits of using MLflow evaluations are:  ⏳ Automation: Save time and effort compared to manual processes.  📏 Standardization: Ensure consistent evaluation across experiments.  🔁 Reproducible results: Easily share and compare findings with colleagues.  💡 Focus on innovation: Spend less time managing, more time exploring new prompts and solutions.   Check out the first comment below for technical tutorials and guides on using MLflow for LLM Evalutions.#mlflow #llm #llmops #mlops #ai

As the AI landscape evolves, so does the challenge of effectively evaluating Large Language Models (LLMs). I've been exploring various frameworks, metrics, and approaches that span from statistical to model-based evaluations. Here's a categorical overview: 🛠️ 𝗘𝘃𝗮𝗹𝘂𝗮𝘁𝗶𝗼𝗻 𝗙𝗿𝗮𝗺𝗲𝘄𝗼𝗿𝗸𝘀: 1. Cloud Provider Platforms (e.g., AWS Bedrock, Azure AI Studio, Vertex AI Studio) 2. LLM-specific Tools (e.g., DeepEval, LangSmith, Helm, Weights & Biases, TruLens, Parea AI, Prompt Flow, EleutherAI, Deepchecks, MLflow LLM Evaluation, Evidently AI, OpenAI Evals, Hugging Face Evaluate) 3. Benchmarking Tools (e.g., BIG-bench, (Super)GLUE, MMLU, HumanEval) 📈 𝗞𝗲𝘆 𝗘𝘃𝗮𝗹𝘂𝗮𝘁𝗶𝗼𝗻 𝗠𝗲𝘁𝗿𝗶𝗰𝘀: 1. Text Generation & Translation (e.g., BLEU, ROUGE, BERTScore, METEOR, MoverScore, BLEURT) 2. LLM-specific (e.g., GPTScore, SelfCheckGPT, GEval, EvalGen) 3. Question-Answering (e.g., QAG Score, SQuAD2.0) 4. Natural Language Inference (e.g., MENLI, AUC-ROC, MCC, Precision-Recall AUC, Confusion Matrix, Cohen's Kappa, Cross-entropy Loss) 5. Sentiment Analysis (e.g., Precision, Recall, F-measure, Accuracy) 6. Named Entity Recognition (e.g., F1 score, F-beta score) 7. Contextual Word Embedding & Similarity (e.g., Cosine similarity, (Damerau-)Levenshtein Distance, Euclidean distance, Hamming distance, Jaccard similarity, Jaro(-Winkler) similarity, N-gram similarity, Overlap similarity, Smith-Waterman similarity, Sørensen-Dice similarity, Tversky similarity) IMO, these "objective" metrics should be balanced with human evaluation for a comprehensive assessment, which would include the subjective eye-test for relevance, fluency, coherence, diversity, and simply someone "trying to break it." 🤔 What are your thoughts on LLM evaluation? Any frameworks or metrics you'd add to this list? Would you like me to explain any changes or provide further suggestions? #AIEvaluation #LLM #MachineLearning #DataScience

OpenAI CPO: Evals are becoming a core skill for PMs. PM in 2025 is changing fast. PMs need to learn brand new skills: 1. AI Evals (https://lnkd.in/eGbzWMxf) 2. AI PRDs (https://lnkd.in/eMu59p_z) 3. AI Strategy (https://lnkd.in/egemMhMF) 4. AI Discovery (https://lnkd.in/e7Q6mMpc) 5. AI Prototyping (https://lnkd.in/eJujDhBV) And evals is amongst the deepest topics. There's 3 steps to them: 1. Observing (https://lnkd.in/e3eQBdMp) 2. Analyzing Errors (https://lnkd.in/eEG83W5D) 3. Building LLM Judges (https://lnkd.in/ez3stJRm) - - - - - - Here's your simple guide to evals in 5 minutes: (Repost this before anything else ♻️) 𝟭. 𝗕𝗼𝗼𝘁𝘀𝘁𝗿𝗮𝗽 𝗬𝗼𝘂𝗿 𝗗𝗮𝘁𝗮𝘀𝗲𝘁 Start with 100 diverse traces of your LLM pipeline. Use real data if you can, or systematic synthetic data generation across key dimensions if you can't. Quality over quantity here: aggressive filtering beats volume. 𝟮. 𝗔𝗻𝗮𝗹𝘆𝘇𝗲 𝗧𝗵𝗿𝗼𝘂𝗴𝗵 𝗢𝗽𝗲𝗻 𝗖𝗼𝗱𝗶𝗻𝗴 Read every trace carefully and label failure modes without preconceptions. Look for the first upstream failure in each trace. Continue until you hit theoretical saturation, when new traces reveal no fundamentally new error types. 𝟯. 𝗦𝘁𝗿𝘂𝗰𝘁𝘂𝗿𝗲 𝗬𝗼𝘂𝗿 𝗙𝗮𝗶𝗹𝘂𝗿𝗲 𝗠𝗼𝗱𝗲𝘀 Group similar failures into coherent, binary categories through axial coding. Focus on Gulf of Generalization failures (where clear instructions are misapplied) rather than Gulf of Specification issues (ambiguous prompts you can fix easily). 𝟰. 𝗕𝘂𝗶𝗹𝗱 𝗔𝘂𝘁𝗼𝗺𝗮𝘁𝗲𝗱 𝗘𝘃𝗮𝗹𝘂𝗮𝘁𝗼𝗿𝘀 Create dedicated evaluators for each failure mode. Use code-based checks when possible (regex, schema validation, execution tests). For subjective judgments, build LLM-as-Judge evaluators with clear Pass/Fail criteria, few-shot examples, and structured JSON outputs. 𝟱. 𝗗𝗲𝗽𝗹𝗼𝘆 𝘁𝗵𝗲 𝗜𝗺𝗽𝗿𝗼𝘃𝗲𝗺𝗲𝗻𝘁 𝗙𝗹𝘆𝘄𝗵𝗲𝗲𝗹 Integrate evals into CI/CD, monitor production with bias-corrected success rates, and cycle through Analyze→ Measure→ Improve continuously. New failure modes in production feed back into your evaluation artifacts. Evals are now a core skill for AI PMs. This is your map. - - - - - I learned this from Hamel Husain and Shreya Shankar. Get 35% off their course: https://lnkd.in/e5DSNJtM 📌 Want our step-by-step guide to evals? Comment 'steps' + DM me. Repost to cut the line. ➕ Follow Aakash Gupta to stay on top of AI x PM.

‘AI Evals’ explained: what they are and how to write them 📚 Like traditional PMs write product specs, AI PMs are supposed to write AI evals. In traditional products, you ship a feature → track usage → optimize based on feedback. But with AI-powered features, your model’s output is the product. So you need to test it before launch with structured evaluations. That’s where AI Evals come in. 𝗪𝗵𝗮𝘁 𝗶𝘀 𝗮𝗻 𝗔𝗜 𝗘𝘃𝗮𝗹? An AI Evaluation is a test suite that helps you measure how well your AI model performs on specific tasks using real or synthetic data. 𝗬𝗼𝘂 𝘂𝘀𝗲 𝗶𝘁 𝘁𝗼: 🔹Compare models (e.g., GPT-4 vs Claude on your use case) 🔹Validate prompt chains or agents 🔹Detect failure cases 🔹Track quality regressions over time Think of it as writing unit tests but for LLM outputs. 𝗛𝗼𝘄 𝘁𝗼 𝘄𝗿𝗶𝘁𝗲 𝗮 𝗴𝗼𝗼𝗱 𝗔𝗜 𝗘𝘃𝗮𝗹 𝗮𝘀 𝗮 𝗣𝗠? A simple structure structure of AI eval would include 2 aspects: 1. Component 2. What do they define Run this over 100s of test cases → analyze failure patterns → tune prompts or switch models. Below table explains it clearly with examples 👇 𝗧𝗵𝗲𝘆 𝗮𝗿𝗲 𝘃𝗲𝗿𝘆 𝗶𝗺𝗽𝗼𝗿𝘁𝗮𝗻𝘁 𝗳𝗼𝗿 𝗣𝗠𝘀 𝗯𝘂𝗶𝗹𝗱𝗶𝗻𝗴 𝗔𝗜 𝗳𝗲𝗮𝘁𝘂𝗿𝗲𝘀, 𝗮𝘀 𝗶𝘁: ✅ Helps PMs validate AI product quality before launch ✅ Forces clarity on what “good” output looks like ✅ Saves time vs launching and learning from real users’ frustration ✅ Helps track quality when prompts/models change over time Even if you're not an AI PM yet, AI Evals are becoming crucial to understand with the shift coming up in digital products. P.S. Let me know if you want me to create a detailed guide around AI evals!

Three ways to evaluate AI Agents LLM evaluation is complex; agent evaluation is even more complicated. The biggest problem is that AI Agents execute a sequence of actions. They need to understand, plan and eventually execute. All these three stages might contain multiple rounds or steps inside. From another perspective, we can always check the work's result by evaluating the artefacts that have been produced. The first way is to check the result of the AI Agent's execution. This is what was covered in the MLE-bench paper. The paper reviews ML agents automating machine learning engineering (MLE) tasks. The idea was brought from the Kaggle competition. In a nutshell, MLE-bench works as an offline Kaggle competition environment. Target AI Agent receive a task, which can be one of 75 competitions from Kaggle, and then the agent generates a submission, which is a CSV file. The submission is evaluated by grading code, which is unique for each task, to calculate a raw score. Based on grading, the agent wins gold, silver or bronze medals. MLE-bench focuses on ML tasks, so the expected result is a working model, not the output from AI Agent. Apart from this rule, there is also a risk of plagiarism. To detect rule-breaking agents and plagiarism, the MLE bench analyses the execution logs of AI Agents and uses the Dolod tool to detect code duplicates. The second way is to analyse workflow generation. Instead of evaluating artefacts as in the first approach, we can focus on checking how AI agents think or plan. Planning in AI Agents can be complex, so we need steps which the AI Agent wants to execute and dependencies among them in the form of a graph (DAG). The WorFBewnch paper consists of a dataset with scenarios and workflow evaluation code for matching generated graphs. The framework sends the AI Agent a task with tools to use and expects to receive the predicted node chain and graph. The node chain is a list of actions the agent plans to use, and the graph defines the execution flow. The graph can represented in a text form in a simple edge notation: (START, 1) (1, 2) (2, END). The third approach is Agent-as-a-Judge for other AI Agents. The paper which Meta proposed is based on the idea of LLM-as-a-Judge. The biggest difference between LLM- and Agent-as-a-Judge is that the latter is an agent, which means it's not just a prompt that rates two responses; it's a system of components. The paper proposes an Agent consisting of 8 components: graph, locate, search, retrieve, read, ask, planning and memory. All these components help the Agent-as-a-Judge evaluate coding agents, which is the paper's main focus. The Agent-as-a-Judge starts from the input: an initial task and requirements for the result. The planning component prepares the plan and executes it step by step. The goal of the plan is to check each requirement and find evidence. Agent outperforms LLM Judge. Alignment is comparable to Human-as-a-Judge 90% vs 94%, where LLM gives only 60%.