Paolo Di Francesco

Paolo Di Francesco

MLflow contributor · MLOps & agentic AI at AWS

Track LLM model evaluation using Amazon SageMaker managed MLflow and FMEval

less than 1 minute read

In this post, we show how to use FMEval and Amazon SageMaker to programmatically evaluate LLMs. FMEval is an open source LLM evaluation library, designed to provide data scientists and machine learning (ML) engineers with a code-first experience to evaluate LLMs for various aspects, including accuracy, toxicity, fairness, robustness, and efficiency.

Full text here, and GitHub repository here

Architecture of SageMaker managed MLflow: the MLflow tracking server's compute and metadata live in the SageMaker service account, while the artifacts are stored in an S3 bucket in the customer account

We demonstrate how to combine FMEval with Amazon SageMaker managed MLflow to track and compare LLM evaluation results, enabling systematic model selection and governance for your generative AI workflows.

Why I wrote this. LLM evaluation results tend to live in ad-hoc spreadsheets and notebook outputs, which makes model selection unreviewable. The point of this post is that evaluation metrics — accuracy, toxicity, fairness, robustness — deserve the same treatment as training metrics: logged, versioned, and comparable in an experiment tracker.

You May Also Enjoy

Govern models with MLflow and Amazon SageMaker AI Model Registry sync: Part 2

Larger organizations rarely stop at one account: they run multiple development accounts, a central governance function, and — in regulated environments — a hard rule that development workloads cannot write into production-grade accounts. In this second post, we extend the building blocks from Par...

Govern models with MLflow and Amazon SageMaker AI Model Registry sync: Part 1

Data scientists track dozens of candidate runs in MLflow, while governance officers need one authoritative registry to validate, approve, and audit the models that reach production. Managed MLflow on Amazon SageMaker AI now closes that gap: every model registered in MLflow syncs into the SageMake...

Deploying MLflow models to Amazon SageMaker AI Hosting

Your best model sits in the MLflow Model Registry with a version number and a signature — and then someone asks: “Can we get that behind an endpoint?” With managed MLflow on Amazon SageMaker AI, the distance between “registered in MLflow” and “serving traffic” is much shorter than it used to be. ...