TensorFlow Serving for Production

TFLite serves models on the edge; TensorFlow Serving serves them from the server side. It is a flexible, high-performance system purpose-built for putting trained models into production, handling model loading, versioning, and request serving without you writing a custom inference server from scratch.

Setting Up TensorFlow Serving

TensorFlow Serving is most commonly run as a Docker container. You point it at a directory containing one or more SavedModel versions, and it automatically loads the latest one and exposes both REST and gRPC endpoints.

docker pull tensorflow/serving

docker run -p 8501:8501 -p 8500:8500 \

--mount type=bind,source=/path/to/saved_models/my_model,target=/models/my_model \

-e MODEL_NAME=my_model \

-t tensorflow/serving

The directory structure matters: Serving expects each version of the model in its own numerically named subfolder, such as my_model/1/, my_model/2/, and so on, and it will serve the highest-numbered version by default.

Deploying Models via REST API

Once running, Serving exposes a REST endpoint on port 8501 (by default) that accepts JSON-formatted prediction requests.

import requests

import json

data = json.dumps({

'signature_name': 'serving_default',

'instances': x_test[:3].tolist()

})

headers = {'content-type': 'application/json'}

response = requests.post(

'http://localhost:8501/v1/models/my_model:predict',

data=data, headers=headers)

predictions = json.loads(response.text)['predictions']

print(predictions)

This format is simple enough to call from almost any language or platform, which is why REST is the default choice for most teams getting started with Serving.

Deploying Models via gRPC

For latency-sensitive applications, gRPC (port 8500 by default) is significantly faster than REST because it uses a compact binary protocol (Protocol Buffers) instead of JSON, and keeps a persistent connection open rather than negotiating a new HTTP request each time.

import grpc

import tensorflow as tf

from tensorflow_serving.apis import predict_pb2, prediction_service_pb2_grpc

channel = grpc.insecure_channel('localhost:8500')

stub = prediction_service_pb2_grpc.PredictionServiceStub(channel)

request = predict_pb2.PredictRequest()

request.model_spec.name = 'my_model'

request.model_spec.signature_name = 'serving_default'

request.inputs['input_1'].CopyFrom(

tf.make_tensor_proto(x_test[:3], dtype=tf.float32))

result = stub.Predict(request, timeout=5.0)

print(result.outputs['dense_1'])

gRPC's binary encoding and connection reuse typically cut latency noticeably compared to REST, which matters at high request volumes or in real-time applications such as fraud detection or recommendation serving.

Model Versioning

Because Serving watches a model's base directory for numbered subfolders, deploying a new model version is as simple as dropping a new SavedModel into model_name/2/. Serving detects it, loads it in the background, and switches incoming traffic to the new version, all without downtime.

# model_config_list config to pin specific versions

model_config_list {

config {

name: 'my_model'

base_path: '/models/my_model'

model_platform: 'tensorflow'

model_version_policy {

specific {

versions: 1

versions: 2

}

}

}

}

A/B Testing Models

Because Serving can load multiple versions at once, A/B testing simply means routing a percentage of incoming traffic to each version and comparing results. Serving itself does not split traffic; that logic lives in front of it, typically in a load balancer, an API gateway, or a thin routing layer in your application code.

import random

def route_request(request_data):

version = 1 if random.random() < 0.9 else 2   # 90/10 split

url = f'http://localhost:8501/v1/models/my_model/versions/{version}:predict'

return requests.post(url, json=request_data)

Did You Know?

A/B testing in production ML is not just about accuracy. Teams routinely monitor latency, error rates, and downstream business metrics (like click-through or conversion rate) alongside model accuracy, since a more accurate model that is slower or that shifts user behaviour unexpectedly may not actually be the better choice to fully roll out.