Understanding DeepEval: A Practical Guide for Evaluating Large Language Models
The rapid evolution of Large Language Models (LLMs) has transformed the way we build intelligent applications, but with this growth comes an equally important challenge — how do we measure their true effectiveness? Traditional evaluation methods often fall short in capturing the diverse capabilities and limitations of LLMs. From reasoning and accuracy to coherence, bias, and ethical alignment, a robust evaluation framework is essential to ensure that these models are reliable and suitable for real-world use.
This is where DeepEval comes in. DeepEval is an open-source framework built to streamline LLM testing by offering a comprehensive suite of metrics, synthetic dataset generation, real-time evaluation, and seamless integration with popular testing frameworks like Pytest. By enabling easy customization, DeepEval empowers researchers and developers to benchmark models against tasks like MMLU, apply advanced metrics such as G-eval, and rigorously validate outputs for relevance and reliability.
In this tutorial, you’ll learn how to set up DeepEval, create a relevance test inspired by Pytest, evaluate LLM outputs using the G-eval metric, and run MMLU benchmarking on the TinyLlama model. By the end, you’ll have a clear workflow to systematically test and improve the performance of your LLM-powered applications.
Getting Started
Table of contents
- What is DeepEval
- Key Features
- Getting started with evaluation of LLM models using DeepEval
- Installing the dependencies
- Querying the Model & Measuring Different Metrics
- Example 1: Answer Relevancy Metric
- Example 2: G-Eval Metric
- Example 3: Prompt Alignment Metric
- Example 4: Json Correctness Metric
- Example 5: Summarization Metric
- Example 6: LLM Integration
- Example 7: Hallucinations
- Example 8: Faithfulness Metric
- Example 9: Chatbot Evaluation
- Example 10: LLM Tracing
- Example 11: MCP Interactions
- MMLU benchmarking with DeepEval for custom LLMs
What is DeepEval
DeepEval is an open-source evaluation framework designed for testing Large Language Models (LLMs) across multiple dimensions such as reasoning, accuracy, coherence, relevance, and ethical alignment. Unlike simple benchmarks, DeepEval goes beyond by offering custom metrics, real-time evaluation, synthetic dataset generation, and seamless integration with testing pipelines. It allows researchers and developers to systematically measure and monitor LLM performance both in experimentation and production.
Key Features
- Extensive Metric Suite
Provides 14+ research-backed metrics for LLM evaluation.
Includes advanced metrics like G-Eval (chain-of-thought reasoning), Faithfulness (accuracy & reliability), Toxicity, Answer Relevancy, and Conversational Metrics such as knowledge retention and conversation completeness.
- Custom Metric Development
Allows users to define their own evaluation metrics tailored to specific use cases. - Integration with LLMs
Compatible with any LLM (including OpenAI models).
Supports benchmarking against popular datasets like MMLU and HumanEval. - Real-Time Monitoring & Benchmarking
Enables continuous monitoring of LLMs in production.
Provides robust benchmarking capabilities to assess models efficiently. - Simplified Testing with Pytest Integration
Built with a Pytest-like architecture, making it easy to write unit tests for LLM outputs in just a few lines of code. - Batch Evaluation Support
Supports large-scale evaluations with batch processing, saving time when benchmarking custom LLMs.
Getting started with evaluation of LLM models using DeepEval
In this session, we will explore how to evaluate Large Language Models (LLMs) with DeepEval. By default, DeepEval supports OpenAI, and we’ll be using the OpenAI GPT-4o-mini model. However, you can use any other LLM of your choice.
We will also use uv, a modern and fast Python package manager (instead of pip), to set up our environment and handle dependencies.
Installing uv
uv simplifies dependency management, virtual environments, and running scripts.
- For Windows:
powershell -ExecutionPolicy ByPass -c "irm https://astral.sh/uv/install.ps1 | iex"
set Path=C:\Users\Codem\.local\bin;%Path%- For Linux / Mac:
curl -LsSf https://astral.sh/uv/install.sh | shRefer to the official website for detailed installation instructions.
Installing the dependencies
- Initialize a uv project by executing the following command.
uv init deepeval_demo
cd deepeval_demo- Create and activate a virtual environment by executing the following command.
uv venv
source .venv/bin/activate # for linux
.venv\Scripts\activate # for windows- Install
deepeval,langchain-openai,fastmcp,transformers,accelerate,bitsandbytes,datasets,pandasandpython-dotenvusing uv.
uv add deepeval langchain-openai fastmcp transformers accelerate bitsandbytes datasets pandas python-dotenvSetting up the credentials
- Create a file named
.env. This file will store your environment variables, including the OpenAI key. - Open the
.envfile and add the following code to specify your OpenAI API key and Neo4j credentials.
OPENAI_API_KEY=sk-proj-C1K1hKug99wXxtj...Querying the Model & Measuring Different Metrics
Now that the environment is set up, let’s start querying our LLM and measure the quality of its responses using different metrics.
Example 1: Answer Relevancy Metric
The Answer Relevancy Metric evaluates how relevant the model’s response is compared to the retrieval context. This is useful in RAG (Retrieval-Augmented Generation) systems or whenever you want to ensure the response aligns with supporting facts.
Create a file named test_relevancy.py and add the following code to it.
from deepeval import assert_test
from deepeval.test_case import LLMTestCase
from deepeval.metrics import AnswerRelevancyMetric
from dotenv import load_dotenv
load_dotenv()
def test_relevancy():
# Define the metric with a threshold
relevancy_metric = AnswerRelevancyMetric(threshold=0.7, model="gpt-4o-mini")
# Case 1: Partially relevant answer
test_case_1 = LLMTestCase(
input="Can I return these shoes after 30 days?",
actual_output="Yes, you can return them. We offer a 30-day full refund. Do you have your original receipt?",
retrieval_context=[
"All customers are eligible for a 30-day full refund at no extra cost.",
"Returns are only accepted within 30 days of purchase.",
],
)
# Case 2: Fully relevant answer
test_case_2 = LLMTestCase(
input="Can I return these shoes after 30 days?",
actual_output="Unfortunately, returns are only accepted within 30 days of purchase.",
retrieval_context=[
"All customers are eligible for a 30-day full refund at no extra cost.",
"Returns are only accepted within 30 days of purchase.",
],
)
# Run evaluation
assert_test(test_case_1, [relevancy_metric])
assert_test(test_case_2, [relevancy_metric])Code explanation:
- Metric Defined →
AnswerRelevancyMetric(threshold=0.7)sets the relevancy cutoff. - Test Cases Created → Each
LLMTestCaseincludes:
The user query (input), LLM output (actual_output) and retrieval context used for evaluation. - Assertion →
assert_test()automatically checks if the relevancy score passes the threshold.
In Test Case 1: the answer contradicts the context slightly (“Yes, you can return after 30 days” vs. rule of within 30 days).
In Test Case 2: the answer is fully aligned with the context.
Executing the test
Use the following command to run the test.
deepeval test run test_relevancy.py
Example 2: G-Eval Metric
G-Eval is an LLM evaluation framework that leverages chain-of-thought (CoT) reasoning to assess model outputs based on custom criteria. Unlike fixed metrics, G-Eval is highly flexible and can evaluate nearly any aspect of a response — such as factual accuracy, omissions, clarity, or adherence to instructions.
It works in two steps:
- Generate Evaluation Steps — Uses CoT reasoning to break down the evaluation based on the given criteria.
- Determine Final Score — Applies those steps to score the LLM’s output.
If evaluation steps are manually provided, G-Eval skips step one and directly uses them to calculate the score.
Create a file named test_geval_example.py and add the following code to it.
from deepeval import assert_test
from deepeval.test_case import LLMTestCase, LLMTestCaseParams
from deepeval.metrics import GEval
from dotenv import load_dotenv
load_dotenv()
correctness_metric = GEval(
name="Correctness",
model="gpt-4o-mini",
evaluation_params=[
LLMTestCaseParams.EXPECTED_OUTPUT,
LLMTestCaseParams.ACTUAL_OUTPUT],
evaluation_steps=[
"Check whether the facts in 'actual output' contradicts any facts in 'expected output'",
"You should also lightly penalize omission of detail, and focus on the main idea",
"Vague language, or contradicting OPINIONS, are OK"
],
)
first_test_case = LLMTestCase(input="What are the main causes of deforestation?",
actual_output="The main causes of deforestation include agricultural expansion, logging, infrastructure development, and urbanization.",
expected_output="The main causes of deforestation include agricultural expansion, logging, infrastructure development, and urbanization.")
second_test_case = LLMTestCase(input="Define the term 'artificial intelligence'.",
actual_output="Artificial intelligence is the simulation of human intelligence by machines.",
expected_output="Artificial intelligence refers to the simulation of human intelligence in machines that are programmed to think and learn like humans, including tasks such as problem-solving, decision-making, and language understanding.")
third_test_case = LLMTestCase(input="List the primary colors.",
actual_output="The primary colors are green, orange, and purple.",
expected_output="The primary colors are red, blue, and yellow.")
test_cases = [first_test_case, second_test_case, third_test_case]
for test_case in test_cases:
assert_test(test_case, [correctness_metric])Code explanation
- Define a G-Eval metric named “Correctness” using
gpt-4o-mini. - Specify evaluation parameters: compare expected vs actual output.
- Add custom evaluation steps: check contradictions, penalize omissions, allow vague language/opinions.
- Create three LLM test cases with queries, actual outputs, and expected outputs.
- Run all test cases using
assert_test()with the correctness metric.
Executing the test
Use the following command to run the test.
deepeval test run test_geval_example.py
Example 3: Prompt Alignment Metric
The prompt alignment metric evaluates whether an LLM’s generated output aligns with the instructions defined in the prompt template. It ensures that the model not only provides a relevant response to the query but also adheres to any specified formatting, style, or structural requirements.
Create a file named test_prompt_alignment.py and add the following code to it.
from deepeval import evaluate
from deepeval.metrics import PromptAlignmentMetric
from deepeval.test_case import LLMTestCase
from langchain_core.prompts import ChatPromptTemplate
from langchain_openai import ChatOpenAI
from dotenv import load_dotenv
load_dotenv()
template = """Question: {question}
Answer: Answer in Upper case."""
prompt = ChatPromptTemplate.from_template(template)
model = ChatOpenAI(model="gpt-4o-mini")
chain = prompt | model
query = "What is capital of India?"
input_data = {"question": query}
# Invoke the chain with input data and display the response
actual_output = chain.invoke(input_data).content
print("actual_output:", actual_output)
# Measuring prompt alignment
metric = PromptAlignmentMetric(
prompt_instructions=["Reply in all uppercase"],
model="gpt-4o-mini",
include_reason=True
)
test_case = LLMTestCase(
input=query,
actual_output=actual_output
)
metric.measure(test_case)
print("metric.score:", metric.score)
print("metric.reason:", metric.reason)
# or evaluate test cases in bulk
evaluate([test_case], [metric])Run the test using deepval test run test_prompt_alignment.py command to check if the model output aligns with the prompt instructions.
Example 4: Json Correctness Metric
The JSON Correctness Metric evaluates whether an LLM’s generated output follows the correct JSON schema. Unlike other metrics that rely on an LLM for assessment, this metric simply checks the provided expected_schema and verifies if the actual_output can be successfully validated against it.
Create a file named test_json_correctness.py and add the following code to it.
from deepeval import evaluate
from deepeval.metrics import JsonCorrectnessMetric
from deepeval.test_case import LLMTestCase
from langchain_core.prompts import ChatPromptTemplate
from langchain_openai import ChatOpenAI
from pydantic import BaseModel
from dotenv import load_dotenv
load_dotenv()
class ExampleSchema(BaseModel):
name: str
# Querying the model
template = """Question: {question}
Answer: Let's think step by step."""
prompt = ChatPromptTemplate.from_template(template)
model = ChatOpenAI(model="gpt-4o-mini")
chain = prompt | model
query ="Output me a random Json with the 'name' key"
input_data = {"question": query}
# Invoke the chain with input data and display the response
actual_output = chain.invoke(input_data).content
print("actual_output:", actual_output)
# Measuring Json correctness
metric = JsonCorrectnessMetric(
expected_schema=ExampleSchema,
model="gpt-4o-mini",
include_reason=True
)
test_case = LLMTestCase(
input=query,
actual_output=actual_output
)
metric.measure(test_case)
print("metric.score:", metric.score)
print("metric.reason:", metric.reason)
# or evaluate test cases in bulk
evaluate([test_case], [metric])Run the test using deepval test run test_json_correctness.py command to check if the model output aligns with the instructions.
Example 5: Summarization Metric
The Summarization Metric evaluates whether an LLM generates factually accurate summaries that include the essential details from the original text. Its score is calculated using two components: the alignment_score, which checks if the summary avoids hallucinations or contradictions, and the coverage_score, which measures whether the summary captures all the necessary information from the source text.
Create a file named test_summarization.py and add the following code to it.
from deepeval import evaluate
from deepeval.metrics import SummarizationMetric
from deepeval.test_case import LLMTestCase
from langchain_core.prompts import ChatPromptTemplate
from langchain_openai import ChatOpenAI
from pydantic import BaseModel
from dotenv import load_dotenv
load_dotenv()
class ExampleSchema(BaseModel):
name: str
# This is the original text to be summarized
text = """
Rice is the staple food of Bengal. Bhortas (lit-"mashed") are a really common type of food used as an additive too rice. there are several types of Bhortas such as Ilish bhorta shutki bhorta, begoon bhorta and more. Fish and other seafood are also important because Bengal is a reverrine region.
Some fishes like puti (Puntius species) are fermented. Fish curry is prepared with fish alone or in combination with vegetables.Shutki maach is made using the age-old method of preservation where the food item is dried in the sun and air, thus removing the water content. This allows for preservation that can make the fish last for months, even years in Bangladesh
"""
template = """Question: {question}
Answer: Let's think step by step."""
prompt = ChatPromptTemplate.from_template(template)
model = ChatOpenAI(model="gpt-4o-mini")
chain = prompt | model
query ="Summarize the text for me %s"%(text)
input_data = {"question": query}
# Invoke the chain with input data and display the response in Markdown format
actual_output = chain.invoke(input_data).content
print("actual_output:", actual_output)
test_case = LLMTestCase(input=text, actual_output=actual_output)
metric = SummarizationMetric(
threshold=0.7,
model="gpt-4o-mini",
)
metric.measure(test_case)
print("metric.score:", metric.score)
print("metric.reason:", metric.reason)
# or evaluate test cases in bulk
evaluate([test_case], [metric])Run the test using deepval test run test_summarization.py command to check if the summary is generated correctly.
Example 6: LLM Integration
DeepEval is capable of integrating with any LLM and evaluating its performance across different metrics. In this example, we use LangChain’s ChatOpenAI with DeepEval to test the relevancy of a response. The model (gpt-4o-mini) is queried with “What is the capital of India?”, and the output is evaluated using the Answer Relevancy Metric.
Create a file named test_openai_llm.py and add the following code to it.
from langchain_openai import ChatOpenAI
from deepeval.metrics import AnswerRelevancyMetric
from deepeval.test_case import LLMTestCase
from dotenv import load_dotenv
load_dotenv()
# Initialize the model
chat = ChatOpenAI(model="gpt-4o-mini",temperature=0.7)
# Get response
query = "What is the capital of India?"
response = chat.invoke(query).content
print(f"User: {query} \nAssistant: {response}")
metric = AnswerRelevancyMetric(
threshold=0.7,
model="gpt-4o-mini",
include_reason=True
)
test_case = LLMTestCase(
input=query,
actual_output=response
)
metric.measure(test_case)
print(metric.score)
print(metric.reason)Run the test using deepval test run test_openai_llm.py command to check the evaluation metrics.
Example 7: Hallucinations
DeepEval provides a Hallucination Metric that helps detect and score these cases. It compares the model’s actual output against the given context to check whether the response faithfully sticks to the facts provided. If the output strays beyond or invents unsupported details, the score decreases. This way, DeepEval helps ensure your LLM outputs remain accurate and grounded in the input context.
Create a file named test_hallucinations.py and add the following code to it.
from deepeval import evaluate
from deepeval.metrics import HallucinationMetric
from deepeval.test_case import LLMTestCase
from dotenv import load_dotenv
load_dotenv()
# Replace this with the actual documents that you are passing as input to your LLM.
context=["A man with blond-hair, and a brown shirt drinking out of a public water fountain."]
# Replace this with the actual output from your LLM application
actual_output="A blond drinking water in public."
test_case = LLMTestCase(
input="What was the blond doing?",
actual_output=actual_output,
context=context
)
metric = HallucinationMetric(threshold=0.5)
# To run metric as a standalone
# metric.measure(test_case)
# print(metric.score, metric.reason)
evaluate(test_cases=[test_case], metrics=[metric])Run the test using deepval test run test_hallucinations.py command to check the evaluation metrics.
Example 8: Faithfulness Metric
The FaithfulnessMetric in DeepEval evaluates whether an LLM’s output is accurately grounded in the provided context or source material. It measures if the response remains consistent with the facts without introducing fabricated or contradictory information. This metric is particularly useful for applications like question-answering or retrieval-augmented generation, where maintaining factual consistency is critical. A higher faithfulness score indicates that the model’s output reliably reflects the input context.
Create a file named test_faithfulness.py and add the following code to it.
from deepeval import evaluate
from deepeval.test_case import LLMTestCase
from deepeval.metrics import FaithfulnessMetric
from dotenv import load_dotenv
load_dotenv()
# Replace this with the actual output from your LLM application
actual_output = "We offer a 30-day full refund at no extra cost."
# Replace this with the actual retrieved context from your RAG pipeline
retrieval_context = ["All customers are eligible for a 30 day full refund at no extra cost."]
metric = FaithfulnessMetric(
threshold=0.7,
model="gpt-4o-mini",
include_reason=True
)
test_case = LLMTestCase(
input="What if these shoes don't fit?",
actual_output=actual_output,
retrieval_context=retrieval_context
)
# To run metric as a standalone
# metric.measure(test_case)
# print(metric.score, metric.reason)
evaluate(test_cases=[test_case], metrics=[metric])Run the test using deepval test run test_faithfulness.pycommand to check the evaluation metrics.
Example 9: Chatbot Evaluation
Chatbot Evaluation differs from standard single-turn evaluations because conversations occur over multiple turns. This requires the chatbot to maintain context awareness throughout the interaction, rather than simply providing accurate responses in isolation.
In DeepEval, chatbots are assessed through multi-turn interactions, which must be structured as test cases following OpenAI’s message format. Evaluating multi-turn conversations is challenging, as each AI response depends on the preceding user input and all prior turns in the conversation, making the evaluation inherently context-dependent.
Create a file named test_chatbots.py and add the following code to it.
from deepeval.test_case import ConversationalTestCase, Turn
from deepeval.metrics import TurnRelevancyMetric, KnowledgeRetentionMetric
from deepeval import evaluate
from dotenv import load_dotenv
load_dotenv()
test_case = ConversationalTestCase(
turns=[
Turn(role="user", content="Hello, how are you?"),
Turn(role="assistant", content="I'm doing well, thank you!"),
Turn(role="user", content="How can I help you today?"),
Turn(role="assistant", content="I'd like to buy a ticket to a Coldplay concert."),
]
)
evaluate(test_cases=[test_case], metrics=[TurnRelevancyMetric(), KnowledgeRetentionMetric()])Run the test using deepval test run test_chatbots.pycommand to check the evaluation results.
Example 10: LLM Tracing
LLM Tracing allows you to monitor the full execution of your application from start to finish. In DeepEval, the @observe decorator enables tracing and evaluation of any LLM interaction, regardless of the application’s complexity. By identifying individual components of your LLM workflow—such as functions that perform specific tasks or are invoked selectively—you can apply the @observe decorator to track their behavior. This provides detailed insights into how each part of your LLM application operates, making it easier to debug, optimize, and evaluate performance.
Tracing requires Confident AI credentials to see the traces. Create a basic account on Confident AI and add the given credentials in the .env file.
Confident AI is a comprehensive platform designed to evaluate and enhance the performance of large language models (LLMs). It leverages its open-source evaluation framework, DeepEval, to provide robust testing, benchmarking, and monitoring capabilities for LLM applications. Confident AI emphasizes observability, allowing teams to trace LLM interactions, conduct A/B testing, and gather real-time performance insights.
Create a file named test_llm_tracing.py and add the following code to it.
from openai import OpenAI
from deepeval.tracing import observe
from dotenv import load_dotenv
load_dotenv()
client = OpenAI()
@observe()
def llm_app(query: str) -> str:
return client.chat.completions.create(
model="gpt-4o-mini",
messages=[
{"role": "user", "content": query}
]
).choices[0].message.content
return
# Call app to send trace to Confident AI
llm_app("Write me a poem.")Run the test using deepval test run test_llm_tracing.pycommand to see the tracing results.
Example 11: MCP Interactions
The MCP Use Metric evaluates how effectively an MCP-based LLM agent utilizes the MCP servers it has access to. It leverages an LLM-as-a-judge approach to assess both the MCP primitives invoked and the arguments generated by the LLM application. This metric can be applied to a single-turn LLMTestCase containing MCP parameters, providing insights into the agent’s efficiency and correctness in interacting with the MCP environment.
- Create a basic MCP server using FastMCP. For that, create a file named
mcp_echo_server.pyand add the following code to it.
import asyncio
from fastmcp import FastMCP
# Initialize FastMCP server
mcp = FastMCP("Simple Echo Server")
@mcp.tool()
def echo(message: str) -> str:
"""Echo back the provided message."""
return message
def main():
"""Run the server."""
mcp.run()
if __name__ == "__main__":
main()- Run the server using the command
python mcp_echo_server.py - Create a test case to evalute the MCP. For that, create a file named
test_mcp.pyand add the following code to it.
from deepeval import evaluate
from deepeval.metrics import MCPUseMetric
from deepeval.test_case import LLMTestCase, MCPServer
from dotenv import load_dotenv
load_dotenv()
test_case = LLMTestCase(
input="Hello", # Your input here
actual_output="Hello", # Your LLM app's final output here
mcp_servers=[MCPServer(server_name="Simple Echo Server")] # Your MCP server's data
# MCP primitives used (if any)
)
metric = MCPUseMetric()
# To run metric as a standalone
# metric.measure(convo_test_case)
# print(metric.score, metric.reason)
evaluate([test_case], [metric])Run the test using deepval test run test_mcp.pycommand to see the mcp evaluation results.
MMLU benchmarking with DeepEval for custom LLMs
MMLU (Massive Multitask Language Understanding) is a benchmark commonly used to evaluate large language models through multiple-choice questions. Covering 57 subjects ranging from math and history to law and ethics, it provides a thorough assessment of an LLM’s knowledge and reasoning skills. Its wide subject coverage and carefully designed questions have made MMLU a gold standard for measuring model performance.
In this guide, we will evaluate our custom LLM (TinyLlama-1.1B) on the MMLU dataset. Each entry in the dataset consists of an input prompt and multiple-choice answers (A, B, C, D). Model performance is measured by calculating the percentage of questions answered correctly.
Creating the Custom LLM Model Class
Define a custom class called TinyLlamaModel that extends DeepEvalBaseLLM to generate responses using the language model and tokenizer. The objective is to produce short outputs (two tokens) for a given prompt, while efficiently managing device allocation and handling preprocessing for both the input prompt and the generated output.
Create a file named test_custom_model.py and add the following code to it.
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
from deepeval.models.base_model import DeepEvalBaseLLM
from deepeval.benchmarks import MMLU
from deepeval.benchmarks.tasks import MMLUTask
from typing import List
class TinyLlamaModel(DeepEvalBaseLLM):
def __init__(self, model, tokenizer):
self.model = model
self.tokenizer = tokenizer
self.device = "cuda" if torch.cuda.is_available() else "cpu"
def load_model(self):
return self.model
def generate(self, prompt: str) -> str:
# Clean the prompt for multiple choice
prompt = prompt.replace("Output 'A', 'B', 'C', or 'D'. Full answer not needed.", "")
# Format the prompt for TinyLlama
formatted_prompt = f"### Instruction: Answer with just the letter (A, B, C, or D)\n\n### Question: {prompt}\n\n### Answer:"
model_inputs = self.tokenizer([formatted_prompt], return_tensors="pt").to(self.device)
generated_ids = self.model.generate(
**model_inputs,
max_new_tokens=3,
do_sample=False,
temperature=0.1,
pad_token_id=self.tokenizer.eos_token_id,
repetition_penalty=1.1
)
# Extract only the new tokens
generated_tokens = generated_ids[0][model_inputs['input_ids'].shape[1]:]
clean_output = self.tokenizer.decode(generated_tokens, skip_special_tokens=True).strip()
# Extract just the letter (A, B, C, or D)
for char in clean_output:
if char in ['A', 'B', 'C', 'D']:
return char
return clean_output[:1] # Fallback: return first character
async def a_generate(self, prompt: str) -> str:
return self.generate(prompt)
def batch_generate(self, prompts: List[str]) -> List[str]:
"""Batch generate method required by MMLU benchmark"""
results = []
for prompt in prompts:
try:
result = self.generate(prompt)
results.append(result)
except Exception as e:
print(f"Error generating for prompt: {e}")
results.append("") # Fallback empty response
return results
async def a_batch_generate(self, prompts: List[str]) -> List[str]:
return self.batch_generate(prompts)
def get_model_name(self):
return "TinyLlama-1.1B-Chat"Loading the Model and Tokenizer
Create two functions to load the LLM model and tokenizer directly from local storage. The model will be loaded in 8-bit precision, and the tokenizer will be initialized with appropriate padding and special token configurations.
def load_model(model_name: str):
# Use light quantization for TinyLlama
quant_config = BitsAndBytesConfig(
load_in_4bit=True,
bnb_4bit_quant_type="nf4",
bnb_4bit_compute_dtype=torch.float16,
)
try:
model = AutoModelForCausalLM.from_pretrained(
model_name,
quantization_config=quant_config,
device_map="auto",
dtype=torch.float16,
trust_remote_code=True
)
return model
except Exception as e:
print(f"Error loading quantized model, trying without quantization: {e}")
# Fallback without quantization
return AutoModelForCausalLM.from_pretrained(
model_name,
device_map="auto",
dtype=torch.float16,
trust_remote_code=True
)
def load_tokenizer(model_name):
tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
tokenizer.pad_token = tokenizer.eos_token
tokenizer.padding_side = "left"
return tokenizerBuilding the custom LLM
We will use the Hugging Face model TinyLlama/TinyLlama-1.1B-Chat-v1.0 to directly load both the model and tokenizer. These will then be passed into the custom LLM class to create an LLM response generator.
After loading the TinyLlama 1.1B model and tokenizer from Hugging Face and wrapping them in our custom class, we can test the response generation. The code first runs a single test prompt related to abstract algebra to verify that the model produces an output. It then performs batch generation with multiple prompts, such as arithmetic and geography questions, to validate that the model can handle multiple inputs efficiently. The results are printed for both single and batch generations, ensuring that our custom model class works as expected.
# Load TinyLlama 1.1B model from Hugging Face
tinyllama_model_name = "TinyLlama/TinyLlama-1.1B-Chat-v1.0"
print("Loading tokenizer...")
tokenizer = load_tokenizer(tinyllama_model_name)
print("Loading model...")
model = load_model(tinyllama_model_name)
print("Creating custom model...")
custom_model = TinyLlamaModel(model, tokenizer)
# Test model generation
print("\nTesting model generation:")
prompt = """
The following are multiple choice questions (with answers) about abstract algebra.
Find all c in Z_3 such that Z_3[x]/(x^2 + c) is a field.
A. 0
B. 1
C. 2
D. 3
Answer:"""
test_output = custom_model.generate(prompt)
print(f"Generated output: '{test_output}'")
# Test batch generation
print("\nTesting batch generation...")
test_prompts = [
prompt,
"What is 2+2? A. 1 B. 2 C. 3 D. 4 Answer:",
"Capital of France? A. London B. Berlin C. Paris D. Rome Answer:"
]
batch_outputs = custom_model.batch_generate(test_prompts)
for i, output in enumerate(batch_outputs):
print(f"Batch output {i+1}: '{output}'")Running the MMLU Benchmark
Finally, we will load the MMLU benchmark, define the tasks, and run the evaluation on the custom model. The results of each task can be reviewed using benchmark.task_scores, while benchmark.predictions provides detailed outputs showing which samples were answered correctly and which were not. This allows for a more granular analysis of the model’s performance.
# Run MMLU benchmark with very light settings
print("\nRunning MMLU benchmark...")
benchmark = MMLU(
tasks=[MMLUTask.HIGH_SCHOOL_COMPUTER_SCIENCE], # Only one task
n_shots=2 # Reduced shots for smaller model
)
try:
benchmark.evaluate(model=custom_model, batch_size=1) # Batch size 1
print("\nBenchmark Results:")
print(f"Task Scores: {benchmark.task_scores}")
print(f"Overall Score: {benchmark.overall_score}")
# Print detailed predictions
print("\nSample Predictions:")
for i, (input_text, prediction) in enumerate(list(benchmark.predictions.items())[:3]):
print(f"Prediction {i+1}:")
print(f"Input: {input_text[:100]}...")
print(f"Prediction: {prediction}")
print("---")
except Exception as e:
print(f"Error during benchmark evaluation: {e}")
print("Trying with even smaller settings...")
# Fallback: try with minimal settings
benchmark = MMLU(
tasks=[MMLUTask.HIGH_SCHOOL_COMPUTER_SCIENCE],
n_shots=1
)
benchmark.evaluate(model=custom_model, batch_size=1)
print(f"Fallback overall score: {benchmark.overall_score}")Final Code
The complete implementation will look as follows:
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
from deepeval.models.base_model import DeepEvalBaseLLM
from deepeval.benchmarks import MMLU
from deepeval.benchmarks.tasks import MMLUTask
from typing import List
class TinyLlamaModel(DeepEvalBaseLLM):
def __init__(self, model, tokenizer):
self.model = model
self.tokenizer = tokenizer
self.device = "cuda" if torch.cuda.is_available() else "cpu"
def load_model(self):
return self.model
def generate(self, prompt: str) -> str:
# Clean the prompt for multiple choice
prompt = prompt.replace("Output 'A', 'B', 'C', or 'D'. Full answer not needed.", "")
# Format the prompt for TinyLlama
formatted_prompt = f"### Instruction: Answer with just the letter (A, B, C, or D)\n\n### Question: {prompt}\n\n### Answer:"
model_inputs = self.tokenizer([formatted_prompt], return_tensors="pt").to(self.device)
generated_ids = self.model.generate(
**model_inputs,
max_new_tokens=3,
do_sample=False,
temperature=0.1,
pad_token_id=self.tokenizer.eos_token_id,
repetition_penalty=1.1
)
# Extract only the new tokens
generated_tokens = generated_ids[0][model_inputs['input_ids'].shape[1]:]
clean_output = self.tokenizer.decode(generated_tokens, skip_special_tokens=True).strip()
# Extract just the letter (A, B, C, or D)
for char in clean_output:
if char in ['A', 'B', 'C', 'D']:
return char
return clean_output[:1] # Fallback: return first character
async def a_generate(self, prompt: str) -> str:
return self.generate(prompt)
def batch_generate(self, prompts: List[str]) -> List[str]:
"""Batch generate method required by MMLU benchmark"""
results = []
for prompt in prompts:
try:
result = self.generate(prompt)
results.append(result)
except Exception as e:
print(f"Error generating for prompt: {e}")
results.append("") # Fallback empty response
return results
async def a_batch_generate(self, prompts: List[str]) -> List[str]:
return self.batch_generate(prompts)
def get_model_name(self):
return "TinyLlama-1.1B-Chat"
def load_model(model_name: str):
# Use light quantization for TinyLlama
quant_config = BitsAndBytesConfig(
load_in_4bit=True,
bnb_4bit_quant_type="nf4",
bnb_4bit_compute_dtype=torch.float16,
)
try:
model = AutoModelForCausalLM.from_pretrained(
model_name,
quantization_config=quant_config,
device_map="auto",
dtype=torch.float16,
trust_remote_code=True
)
return model
except Exception as e:
print(f"Error loading quantized model, trying without quantization: {e}")
# Fallback without quantization
return AutoModelForCausalLM.from_pretrained(
model_name,
device_map="auto",
dtype=torch.float16,
trust_remote_code=True
)
def load_tokenizer(model_name):
tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
tokenizer.pad_token = tokenizer.eos_token
tokenizer.padding_side = "left"
return tokenizer
# Load TinyLlama 1.1B model from Hugging Face
tinyllama_model_name = "TinyLlama/TinyLlama-1.1B-Chat-v1.0"
print("Loading tokenizer...")
tokenizer = load_tokenizer(tinyllama_model_name)
print("Loading model...")
model = load_model(tinyllama_model_name)
print("Creating custom model...")
custom_model = TinyLlamaModel(model, tokenizer)
# Test model generation
print("\nTesting model generation:")
prompt = """
The following are multiple choice questions (with answers) about abstract algebra.
Find all c in Z_3 such that Z_3[x]/(x^2 + c) is a field.
A. 0
B. 1
C. 2
D. 3
Answer:"""
test_output = custom_model.generate(prompt)
print(f"Generated output: '{test_output}'")
# Test batch generation
print("\nTesting batch generation...")
test_prompts = [
prompt,
"What is 2+2? A. 1 B. 2 C. 3 D. 4 Answer:",
"Capital of France? A. London B. Berlin C. Paris D. Rome Answer:"
]
batch_outputs = custom_model.batch_generate(test_prompts)
for i, output in enumerate(batch_outputs):
print(f"Batch output {i+1}: '{output}'")
# Run MMLU benchmark with very light settings
print("\nRunning MMLU benchmark...")
benchmark = MMLU(
tasks=[MMLUTask.HIGH_SCHOOL_COMPUTER_SCIENCE], # Only one task
n_shots=2 # Reduced shots for smaller model
)
try:
benchmark.evaluate(model=custom_model, batch_size=1) # Batch size 1
print("\nBenchmark Results:")
print(f"Task Scores: {benchmark.task_scores}")
print(f"Overall Score: {benchmark.overall_score}")
# Print detailed predictions
print("\nSample Predictions:")
for i, (input_text, prediction) in enumerate(list(benchmark.predictions.items())[:3]):
print(f"Prediction {i+1}:")
print(f"Input: {input_text[:100]}...")
print(f"Prediction: {prediction}")
print("---")
except Exception as e:
print(f"Error during benchmark evaluation: {e}")
print("Trying with even smaller settings...")
# Fallback: try with minimal settings
benchmark = MMLU(
tasks=[MMLUTask.HIGH_SCHOOL_COMPUTER_SCIENCE],
n_shots=1
)
benchmark.evaluate(model=custom_model, batch_size=1)
print(f"Fallback overall score: {benchmark.overall_score}")Executing the test
Executing the custom LLM model benchmark test using the above code by running the following command.
deepeval test run test_custom_model.py
Thanks for reading this article !!
Thanks Gowri M Bhatt for reviewing the content.
If you enjoyed this article, please click on the clap button 👏 and share to help others find it!
The full source code for this tutorial can be found here,