Building an LLM app is exciting in the beginning. A prompt works, the chatbot responds, and the demo looks impressive. The real challenge starts after that.<\/p>\n\n\n\n
Responses may change across runs. A good answer may fail with a slightly different input. A RAG app may retrieve the wrong document. An AI agent may call the wrong tool or take too many steps. These problems are hard to catch with normal logs. That is where LLMOps becomes important. It gives teams a structured way to test, monitor, debug, and improve LLM applications. LangSmith makes this process easier by helping developers trace model behavior, evaluate outputs, compare prompt versions, and monitor production performance.<\/p>\n\n\n\n
This LLMOps with LangSmith tutorial explains the key steps needed to move an LLM app from a working prototype to a more reliable production system.<\/p>\n\n\n\n
LLMOps means managing the full lifecycle of LLM applications, from building and testing to deployment, monitoring, and improvement. LLM apps are different from traditional software because their outputs can change across runs. They may hallucinate, miss context, or respond incorrectly. LLMOps brings structure through tracing, evaluation, monitoring, feedback, and continuous improvement.<\/p>\n\n\n\n
LangSmith is an LLMOps platform built by LangChain<\/a>. It helps developers trace, test, evaluate, and monitor LLM applications. With LangSmith, teams can see what happened inside an LLM workflow. They can inspect prompts, model responses, tool calls, retrieved documents, latency, token usage, and errors. This makes it easier to debug chatbots, RAG pipelines, AI agents, and other LLM-powered systems.<\/p>\n\n\n\n Start by creating a LangSmith account and generating an API key from the LangSmith dashboard. The API key connects your local LLM application with LangSmith so every run, trace, prompt, and model response can be tracked.<\/p>\n\n\n\n Install the required packages in your Python environment.<\/p>\n\n\n\n Then set the required environment variables.<\/p>\n\n\n\n LangSmith documentation recommends setting LANGSMITH_TRACING, LANGSMITH_API_KEY, and your model provider API key before running evaluation or tracing workflows.<\/p>\n\n\n\n Tracing is the first practical step in any LLMOps workflow. It records how your application behaves during each run, including prompts, model responses, tool calls, latency, errors, and intermediate steps.<\/p>\n\n\n\n This is important because LLM applications are hard to debug through logs alone. LangSmith traces help developers understand what happened inside a chain, chatbot, RAG pipeline, or AI agent.<\/a><\/p>\n\n\n\n A simple traced function can look like this:<\/p>\n\n\n\n LangSmith observability gives visibility into individual traces and production-level performance metrics, helping teams debug agent behavior, latency, cost, and response quality.<\/p>\n\n\n\n After tracing your LLM<\/a> application, create a dataset of test cases. A dataset usually contains inputs, expected outputs, and examples that define what a good response should look like.<\/p>\n\n\n\n For example, a customer support chatbot dataset may include common user questions, correct answers, refusal rules, and edge cases. A RAG application <\/a>dataset may include user queries, source documents, and reference answers.<\/p>\n\n\n\n LangSmith supports dataset management, versioning, filtering, splitting, sharing, and exporting. Every dataset update creates a new version, which helps teams track how evaluation data changes over time.<\/p>\n\n\n\n Offline evaluation helps test an LLM application before release. It compares model outputs against expected behavior, quality rules, or reference answers.<\/p>\n\n\n\n You can use offline evaluations for:<\/p>\n\n\n\n LangSmith defines offline evaluations as pre-deployment tests that run against curated datasets. These evaluations help compare versions, prevent quality drops, and test application behavior before production.<\/p>\n\n\n\n Evaluators measure whether the LLM output meets your quality standards. LangSmith<\/a> supports prebuilt evaluators and custom evaluators, depending on the use case.<\/p>\n\n\n\n Common evaluator types include correctness, relevance, helpfulness, hallucination checks, and custom business-rule checks.<\/p>\n\n\n\n For example, a correctness evaluator can check whether the answer matches the reference output. A custom evaluator can check whether the response follows brand tone, avoids restricted claims, or cites the right source.<\/p>\n\n\n\n LangSmith\u2019s evaluation quickstart includes creating a prompt, creating a dataset<\/a>, adding an evaluator, and running an evaluation experiment.<\/p>\n\n\n\n Prompt iteration is a major part of LLMOps. LangSmith helps compare different prompts, model settings, and application versions using evaluation results.<\/p>\n\n\n\n This makes it easier to answer questions like:<\/p>\n\n\n\n Use LangSmith experiments to compare runs side by side. This gives teams a clearer way to improve prompts instead of relying only on \u00a0manual testing.<\/a><\/p>\n\n\n\n Master the foundations of LLM applications with HCL GUVI\u2019s <\/em>LLMs and Their Applications Course<\/em><\/a>. Learn prompt engineering, LLM workflows, evaluation techniques, and real-world AI use cases through hands-on, self-paced learning. Earn a globally recognised certification, get lifetime access to course content, and strengthen your skills with gamified practice platforms.<\/em><\/p>\n\n\n\n Once the LLM application goes live, shift from offline evaluation to production monitoring. Production monitoring helps track real user behavior, model quality, errors, latency, token usage, and cost.<\/p>\n\n\n\n LangSmith monitoring can track metrics such as token usage, latency, error rates, cost breakdowns, feedback scores, and online evaluation results. It also supports alerts through tools like webhooks and PagerDuty.<\/p>\n\n\n\n
\n 85%<\/strong> of ML models never make it to production, which is exactly the gap LLMOps practices<\/strong> like tracing and evaluation are designed to close.\n<\/div>\n\n\n\nLLMOps with LangSmith Tutorial: Key Steps<\/strong><\/h2>\n\n\n\n
Step 1: Set Up LangSmith<\/h3>\n\n\n\n
pip install -U langsmith openai\n<\/code><\/pre>\n\n\n\nexport LANGSMITH_TRACING=true\nexport LANGSMITH_API_KEY=\"your-langsmith-api-key\"\nexport OPENAI_API_KEY=\"your-openai-api-key\"\n<\/code><\/pre>\n\n\n\nStep 2: Add Tracing to Your <\/strong>LLM Application<\/strong><\/a><\/h3>\n\n\n\n
from langsmith import traceable\nfrom openai import OpenAI\n\nclient = OpenAI()\n\n@traceable\ndef generate_answer(question):\n response = client.chat.completions.create(\n model=\"gpt-4o-mini\",\n messages=[\n {\"role\": \"system\", \"content\": \"Answer clearly and accurately.\"},\n {\"role\": \"user\", \"content\": question}\n ]\n )\n return response.choices[0].message.content\n\nprint(generate_answer(\"What is LLMOps?\"))\n<\/code><\/pre>\n\n\n\nStep 3: Create a Dataset for Evaluation<\/h3>\n\n\n\n
Step 4: Run Offline Evaluations<\/h3>\n\n\n\n
\n
Step 5: Add Evaluators<\/h3>\n\n\n\n
\n The global LLMOps software market<\/strong> is projected to grow from $7.14 billion in 2026<\/strong> to $15.59 billion by 2030<\/strong>, showing how fast enterprises are investing in tools that manage LLM reliability<\/strong>.\n<\/div>\n\n\n\nStep 6: Compare Prompts and Model Versions<\/h3>\n\n\n\n
\n
Step 7: Monitor the Application in Production<\/h3>\n\n\n\n