Ray Serve is a scalable model serving library built on Ray. It helps developers turn Python functions, machine learning models, and AI pipelines into production-ready APIs. Ray Serve is framework-agnostic, so it can serve models built with PyTorch, TensorFlow, scikit-learn, Hugging Face, or custom Python code.<\/p>\n\n\n\n
A Ray Serve tutorial is useful when you want to deploy an ML model as an HTTP API, scale replicas, handle multiple requests, and move from local testing to production deployment.<\/p>\n\n\n\n
Ray Serve is a scalable model serving framework built on top of Ray, designed to deploy machine learning models, Python functions, and AI applications as production-ready APIs. It lets developers define deployments using Python classes or functions, expose them through HTTP endpoints, and scale them across CPU or GPU resources using Ray\u2019s distributed execution engine.<\/p>\n\n\n\n
Defines a Python class<\/a> or function as a Ray Serve deployment. It is the main decorator used to wrap model inference logic, API logic, or pipeline steps. A deployment can run with one or more replicas for scaling. <\/p>\n\n\n\n Creates a Ray Serve application from a deployment. It connects the deployment with its arguments and prepares it to be run using serve.run() or deployed through a config file. <\/p>\n\n\n\n Runs a Ray Serve application locally or inside a Ray cluster. It accepts an application returned by .bind() and returns a handle to the ingress deployment. <\/p>\n\n\n\n Connects a Ray Serve deployment with a FastAPI app<\/a>. This is useful when you want clean HTTP routes such as \/predict, \/health, or \/classify. <\/p>\n\n\n\n Controls how many copies of a deployment run at the same time. More replicas allow Ray Serve to process more requests in parallel.<\/p>\n\n\n\n Defines compute resources for each deployment replica. It can assign CPUs, GPUs, memory, or custom resources to model-serving workloads. <\/p>\n\n\n\n Limits how many requests a replica can handle at once. This helps control latency and prevent a model replica from being overloaded. <\/p>\n\n\n\n Deploys a Ray Serve application using a YAML config file. This is commonly used for production because it separates deployment settings from Python code.<\/p>\n\n\n\n Create a virtual environment<\/a> first.<\/p>\n\n\n\n For Windows:<\/p>\n\n\n\n Install Ray Serve and required packages.<\/p>\n\n\n\n The ray[serve] extra installs Ray Core, dashboard, cluster launcher, and Serve components.<\/p>\n\n\n\n Create a file named app.py.<\/p>\n\n\n\n The @serve.deployment decorator turns the class into a Ray Serve deployment. Requests are handled by deployment replicas, and these replicas can be scaled independently.<\/p>\n\n\n\n Create another file named run.py.<\/p>\n\n\n\n Run the file.<\/p>\n\n\n\n The serve.run() method runs a Serve application created through .bind(). It also exposes the app through the route prefix when HTTP routing is enabled.<\/p>\n\n\n\n Open a new terminal and send a test request.<\/p>\n\n\n\n Expected response:<\/p>\n\n\n\n You can also test the health endpoint.<\/p>\n\n\n\n Expected response:<\/p>\n\n\n\n Build production-ready machine learning skills with HCL GUVI\u2019s <\/em>Machine Learning Course<\/em><\/a>. Learn ML model development, deployment workflows, MLOps basics, and real-world AI applications through structured, hands-on training designed for aspiring machine learning engineers.<\/em><\/p>\n\n\n\n Update the deployment line in app.py.<\/p>\n\n\n\n This runs two replicas of the deployment. More replicas help the service handle more concurrent requests, especially when model inference takes time.<\/p>\n\n\n\n For production, scaling should usually be managed through a Serve config file instead of changing code manually.<\/p>\n\n\n\n Create a file named serve_config.yaml.<\/p>\n\n\n\n Start a local Ray cluster.<\/p>\n\n\n\n Deploy the application.<\/p>\n\n\n\n Check deployment status.<\/p>\n\n\n\n Ray Serve recommends config files for production because they let teams configure applications, routes, replicas, HTTP options, and deployment parameters in one place.<\/p>\n\n\n\n Ray Serve makes model deployment easier by combining Python<\/a>-native development with scalable serving infrastructure. A developer can start with a simple deployment, expose it through FastAPI<\/a>, test it locally, and later move the same application into a production setup using Serve config files.<\/p>\n\n\n\n\n
\n
\n
\n
\n
\n
\n
Ray Serve Tutorial: Step-by-Step Guide<\/h2>\n\n\n\n
Step 1: Install Ray Serve<\/h3>\n\n\n\n
python -m venv rayserve-env\nsource rayserve-env\/bin\/activate\n<\/code><\/pre>\n\n\n\nrayserve-env\\Scripts\\activate\n<\/code><\/pre>\n\n\n\npip install -U \"ray[serve]\" fastapi requests\n<\/code><\/pre>\n\n\n\nStep 2: Create a Ray Serve Application<\/h3>\n\n\n\n
from fastapi import FastAPI\nfrom pydantic import BaseModel\nfrom ray import serve\n\napi = FastAPI()\n\n\nclass ReviewRequest(BaseModel):\n text: str\n\n@serve.deployment(num_replicas=1)\n@serve.ingress(api)\nclass SentimentModel:\n def __init__(self):\n self.positive_words = {\"good\", \"great\", \"excellent\", \"love\", \"fast\"}\n\n @api.get(\"\/\")\n def health_check(self):\n return {\"status\": \"running\", \"service\": \"ray-serve\"}\n\n @api.post(\"\/predict\")\n def predict(self, request: ReviewRequest):\n text = request.text.lower()\n score = sum(word in text for word in self.positive_words)\n\n label = \"positive\" if score > 0 else \"neutral\"\n\n return {\n \"input\": request.text,\n \"prediction\": label,\n \"score\": score\n }\n\napp = SentimentModel.bind()\n<\/code><\/pre>\n\n\n\n
\n Ray<\/strong> now has over 39,000 GitHub stars<\/strong> and has been downloaded more than 237 million times<\/strong>. It recently joined the PyTorch Foundation<\/strong> alongside projects like vLLM<\/strong> as part of the open-source AI compute stack.\n<\/div>\n\n\n\nStep 3: Run the Application Locally<\/h3>\n\n\n\n
from ray import serve\nfrom app import app\n\nserve.run(app, route_prefix=\"\/sentiment\")\n<\/code><\/pre>\n\n\n\npython run.py\n<\/code><\/pre>\n\n\n\nStep 4: Test the API<\/h3>\n\n\n\n
curl -X POST \"http:\/\/127.0.0.1:8000\/sentiment\/predict\" \\\n-H \"Content-Type: application\/json\" \\\n-d '{\"text\": \"Ray Serve is fast and great\"}'\n<\/code><\/pre>\n\n\n\n{\n \"input\": \"Ray Serve is fast and great\",\n \"prediction\": \"positive\",\n \"score\": 2\n}\n<\/code><\/pre>\n\n\n\ncurl http:\/\/127.0.0.1:8000\/sentiment\/\n<\/code><\/pre>\n\n\n\n{\n \"status\": \"running\",\n \"service\": \"ray-serve\"\n}\n<\/code><\/pre>\n\n\n\nStep 5: Scale the Deployment<\/h3>\n\n\n\n
@serve.deployment(num_replicas=2)\n<\/code><\/pre>\n\n\n\nStep 6: Create a Production Serve Config<\/h3>\n\n\n\n
applications:\n - name: sentiment_app\n route_prefix: \/sentiment\n import_path: app:app\n deployments:\n - name: SentimentModel\n num_replicas: 2\n<\/code><\/pre>\n\n\n\nray start --head\n<\/code><\/pre>\n\n\n\nserve deploy serve_config.yaml\n<\/code><\/pre>\n\n\n\nserve status\n<\/code><\/pre>\n\n\n\nConclusion<\/strong><\/h2>\n\n\n\n