Paolo Di Francesco

Paolo Di Francesco

MLflow contributor · MLOps & agentic AI at AWS

Create a Generative AI Gateway to allow secure and compliant consumption of foundation models

less than 1 minute read

Every team in the enterprise wants to try foundation models. Nobody wants ten teams each wiring up their own model access, cost tracking, and compliance controls — and the security office wants that even less. In this post, we propose a Generative AI Gateway: one governed entry point through which the whole organization consumes foundation models, so teams can move fast without each of them re-solving privacy, cost, and compliance from scratch.

Full text here

What happened since. When we wrote this in 2023, “LLM gateway” wasn’t yet a category — it has since become a standard enterprise pattern, with a whole ecosystem of open-source and commercial gateways implementing the same core ideas: one governed entry point, centralized cost attribution, and compliance controls decoupled from the consuming teams. The architecture reasoning here still holds; the build-vs-buy answer has shifted.

You May Also Enjoy

Deploying MLflow models to Amazon SageMaker AI Hosting

Your best model sits in the MLflow Model Registry with a version number and a signature — and then someone asks: “Can we get that behind an endpoint?” With managed MLflow on Amazon SageMaker AI, the distance between “registered in MLflow” and “serving traffic” is much shorter than it used to be. ...

End-to-end lineage with DVC and Amazon SageMaker AI MLflow apps

Production ML teams often struggle to trace the full lineage of a model back to the exact data and code that trained it. In this post, we close that gap by combining DVC for data versioning, Amazon SageMaker AI for scalable processing and training, and Amazon SageMaker AI MLflow Apps for experime...