Most ML teams don\u2019t struggle with building models, they struggle with shipping them. A notebook that scores 95% accuracy is still useless to anyone outside the data science team until it\u2019s wrapped in an API, packaged with the right dependencies, and running somewhere reliable. <\/p>\n\n\n\n
That\u2019s the gap BentoML is built to close. Instead of hand-rolling a Flask app, writing a Dockerfile from scratch, and hoping the production environment matches your laptop, BentoML turns a Python class into a versioned, containerized, deployable service with a handful of decorators.<\/p>\n\n\n\n
This tutorial walks through that full path: saving a model into BentoML\u2019s model store, defining a Service with validated APIs, adding async and batched endpoints, wiring multiple Services together, configuring GPU resources, and finally building and deploying the whole thing.<\/p>\n\n\n\n
TL;DR<\/strong><\/p>\n\n\n\n BentoML is an open-source Python framework for packaging, serving, and deploying machine learning models as production-ready APIs. It helps developers save versioned model artifacts, define inference Services, expose prediction logic through REST endpoints, and containerize the full deployment environment.<\/p>\n\n\n\n Verify the install:<\/p>\n\n\n\n BentoML has a model store that versions and tracks artifacts separately from your service code. Save a model into it using the framework-specific integration here, bentoml.sklearn:<\/p>\n\n\n\n Run it:<\/p>\n\n\n\n You can confirm it landed in the store:<\/p>\n\n\n\n BentoML Services are plain Python<\/a> classes decorated with @bentoml.service. Each method exposed with @bentoml.api becomes an HTTP endpoint. Create service.py:<\/p>\n\n\n\n A few details that matter here, not cosmetic ones:<\/p>\n\n\n\n Build production-ready Python skills with HCL GUVI\u2019s <\/em>Python Course<\/em><\/a>, certified by IITM Pravartak. Learn Python from the basics to advanced features through 17 hours of recorded content, 4 modules with certifications, and best practices used by real employers. Start learning for free and pay for the certificate later.<\/em><\/p>\n\n\n\n bentoml serve service:IrisClassifier<\/p>\n\n\n\n By default this binds to http:\/\/localhost:3000. BentoML auto-generates an interactive Swagger UI at http:\/\/localhost:3000 where you can hit \/predict directly, and the route name is derived from the method name unless you override it.<\/p>\n\n\n\n Test it from another terminal:<\/p>\n\n\n\n For IO-bound work, calling another service, reading from disk, hitting a vector DB, async methods let BentoML interleave requests instead of blocking a worker thread per request:<\/p>\n\n\n\n Batching is a runtime-level optimization, not something your function implements manually: BentoML accumulates concurrent calls up to a configurable max batch size and latency window, then dispatches one combined call to your function.<\/p>\n\n\n\n Real systems are rarely one model. BentoML lets you wire Services together with bentoml.depends(), so one Service can call another\u2019s API as if it were a local method:<\/p>\n\n\n\n Each Service in the dependency graph can be scaled and resourced independently when deployed, which matters when one stage is CPU-light preprocessing and another needs a GPU inference.<\/p>\n\n\n\n For a model that needs a GPU, the resources block changes the scheduling behavior, not just a label:<\/p>\n\n\n\n bentoml.models.HuggingFaceModel pulls directly from the Hugging Face Hub and caches it through BentoML\u2019s model store, so it\u2019s versioned the same way as a locally trained model.<\/p>\n\n\n\n\n
<\/li>\n\n\n\n
<\/li>\n\n\n\n
<\/li>\n\n\n\n
<\/li>\n\n\n\n
<\/li>\n\n\n\n
<\/li>\n<\/ul>\n\n\n\nWhat is BentoML?<\/strong><\/h2>\n\n\n\n
\n The word \u201cBento\u201d<\/strong> in BentoML is a direct nod to the Japanese bento box<\/strong>, since a Bento packages your model, code, and environment into one neat, portable container, much like the meal itself.\n<\/div>\n\n\n\nBentoML Tutorial: From Model to Production API<\/strong><\/h2>\n\n\n\n
1. Installation<\/strong><\/h3>\n\n\n\n
pip install bentoml scikit-learn<\/code><\/pre>\n\n\n\nbentoml --version<\/code><\/pre>\n\n\n\n2. Train and save a model<\/strong><\/h3>\n\n\n\n
# train.py\n\nimport bentoml\n\nfrom sklearn import svm, datasets\n\niris = datasets.load_iris()\n\nX, y = iris.data, iris.target\n\nclf = svm.SVC(gamma=\"scale\")\n\nclf.fit(X, y)\n\n# Saves the model into BentoML's local model store with a version tag\n\nsaved_model = bentoml.sklearn.save_model(\"iris_clf\", clf)\n\nprint(f\"Model saved: {saved_model}\")<\/code><\/pre>\n\n\n\npython train.py<\/code><\/pre>\n\n\n\nbentoml models list<\/code><\/pre>\n\n\n\n3. Define a Service<\/strong><\/h3>\n\n\n\n
# service.py\n\nfrom __future__ import annotations\n\nimport numpy as np\n\nimport bentoml\n\nfrom pydantic import BaseModel, Field\n\nclass IrisFeatures(BaseModel):\n\n sepal_length: float = Field(..., ge=0)\n\n sepal_width: float = Field(..., ge=0)\n\n petal_length: float = Field(..., ge=0)\n\n petal_width: float = Field(..., ge=0)\n\n@bentoml.service(\n\n name=\"iris_classifier\",\n\n resources={\"cpu\": \"1\"},\n\n traffic={\"timeout\": 10},\n\n)\n\nclass IrisClassifier:\n\n # Reference the saved model by tag; BentoML resolves the path at runtime\n\n bento_model = bentoml.models.BentoModel(\"iris_clf:latest\")\n\n def __init__(self) -> None:\n\n import joblib\n\n self.model = joblib.load(self.bento_model.path_of(\"saved_model.pkl\"))\n\n @bentoml.api\n\n def predict(self, input_data: IrisFeatures) -> dict:\n\n input_series = np.array([[\n\n input_data.sepal_length,\n\n input_data.sepal_width,\n\n input_data.petal_length,\n\n input_data.petal_width,\n\n ]])\n\n prediction = self.model.predict(input_series)\n\n return {\"class\": int(prediction[0])}<\/code><\/pre>\n\n\n\n\n
4. Run it locally<\/strong><\/h3>\n\n\n\n
curl -X POST http:\/\/localhost:3000\/predict \\\n\n-H \"Content-Type: application\/json\" \\\n\n-d '{\"input_data\": {\"sepal_length\": 5.1, \"sepal_width\": 3.5, \"petal_length\": 1.4, \"petal_width<\/code><\/pre>\n\n\n\n5. Async <\/strong>APIs<\/strong><\/a> and batching<\/strong><\/h3>\n\n\n\n
@bentoml.api\n\nasync def predict_async(self, input_data: IrisFeatures) -> dict:\n\n result = await self._run_inference(input_data)\n\n return result\n\nFor CPU\/GPU-bound inference where you want the runtime to group concurrent requests into a single forward pass, mark the API as batchable:\n\n@bentoml.api(batchable=True)\n\ndef predict_batch(self, input_series: list[list[float]]) -> list[int]:\n\n arr = np.array(input_series)\n\n return self.model.predict(arr).tolist()<\/code><\/pre>\n\n\n\n6. Composing multiple Services<\/strong><\/h3>\n\n\n\n
@bentoml.service\n\nclass Preprocessing:\n\n @bentoml.api\n\n def clean(self, raw: dict) -> IrisFeatures:\n\n return IrisFeatures(**raw)\n\n@bentoml.service\n\nclass Pipeline:\n\n preprocessing = bentoml.depends(Preprocessing)\n\n classifier = bentoml.depends(IrisClassifier)\n\n @bentoml.api\n\n def predict(self, raw: dict) -> dict:\n\n features = self.preprocessing.clean(raw)\n\n return self.classifier.predict(features)<\/code><\/pre>\n\n\n\n7. GPU resources<\/strong><\/h3>\n\n\n\n
@bentoml.service(\n\n resources={\"gpu\": 1, \"gpu_type\": \"nvidia-l4\"},\n\n traffic={\"timeout\": 60},\n\n)\n\nclass GPUService:\n\n model = bentoml.models.HuggingFaceModel(\"meta-llama\/Meta-Llama-3.1-8B-Instruct\")\n\n @bentoml.api\n\n async def generate(self, prompt: str, max_tokens: int = 256) -> str:\n\n ...<\/code><\/pre>\n\n\n\n8. Building a Bento<\/strong><\/h3>\n\n\n\n