Paolo Di Francesco

Paolo Di Francesco

Researcher, SDE, Senior Solutions Architect

Organize your machine learning journey with Amazon SageMaker Experiments and Amazon SageMaker Pipelines

less than 1 minute read

The process of building a machine learning (ML) model is iterative until you find the candidate model that is performing well and is ready to be deployed. As data scientists iterate through that process, they need a reliable method to easily track experiments to understand how each model version was built and how it performed.

Full text here, and GitHub repository here GitHub stars

In this post, we take a closer look at the motivation behind having an automated process to track experiments with Amazon SageMaker Experiments and the native capabilities built into Amazon SageMaker Pipelines. We show how the native integration between Pipelines and Experiments allows data scientists to automatically organize, track, and visualize experiments during model development activities.

You May Also Enjoy

Simplify ModelOps with Amazon SageMaker AI Projects using Amazon S3-based templates

Managing ModelOps workflows can be complex and time-consuming. Amazon SageMaker AI Projects now offers an easier path with Amazon S3-based templates. With this new capability, you can store AWS CloudFormation templates directly in Amazon S3 and manage their entire lifecycle using familiar S3 features such as versioning, lifecycle policies, and cross-region replication.

Full text here, and GitHub repository here GitHub stars

Track LLM model evaluation using Amazon SageMaker managed MLflow and FMEval

In this post, we show how to use FMEval and Amazon SageMaker to programmatically evaluate LLMs. FMEval is an open source LLM evaluation library, designed to provide data scientists and machine learning (ML) engineers with a code-first experience to evaluate LLMs for various aspects, including accuracy, toxicity, fairness, robustness, and efficiency.

Full text here, and GitHub repository here GitHub stars

Secure MLflow in AWS Fine-grained access control with AWS native services

MLflow and Amazon SageMaker are two of many tools on the market to help data scientists to implement end-to-end Machine Learning workloads. SageMaker offers the possibility to run these workloads fully end-to-end on its own ecosystem as it has been designed to solve some of the common challenges that are peculiar to ML lifecycle workloads. Nonetheless, one of the great traits of the SageMaker ecosystem is also its flexibility and openess to integrate with other tools. Today, we ultimately want to show how you can securely integrate SageMaker with MLflow using native AWS services to enable access control on the open-source version of MLflow.

Full text here, and GitHub repository here GitHub stars