{"id":105949,"date":"2026-04-06T17:41:46","date_gmt":"2026-04-06T12:11:46","guid":{"rendered":"https:\/\/www.guvi.in\/blog\/?p=105949"},"modified":"2026-11-16T13:59:57","modified_gmt":"2026-11-16T08:29:57","slug":"setup-and-fine-tune-qwen-3-with-ollama","status":"publish","type":"post","link":"https:\/\/www.guvi.in\/blog\/setup-and-fine-tune-qwen-3-with-ollama\/","title":{"rendered":"Setup and Fine-Tune Qwen 3 with Ollama: Complete Guide (2026)"},"content":{"rendered":"\n<p>Running a large language model (LLM) used to mean paying for an API and sending your data to someone else&#8217;s servers. Today, open models like Qwen 3 run on an ordinary laptop or desktop, and Ollama makes starting one as simple as a single command.<\/p>\n\n\n\n<p>This guide takes you from installation to a model trained on your own examples. You will install Ollama, run Qwen 3, control its thinking mode, call it from Python, then fine-tune it with LoRA and serve the result locally.<\/p>\n\n\n\n<p><\/p>\n\n\n\n<p><strong>Quick Answer:<\/strong><\/p>\n\n\n\n<p>To run Qwen 3 locally, install Ollama from ollama.com and run <code>ollama run qwen3:8b<\/code>, which downloads the model and opens a chat. Ollama only runs models, so to fine-tune Qwen 3, you train a LoRA adapter with Unsloth, export the result as a GGUF file, and load it with <code>ollama create<\/code> and a Modelfile.<\/p>\n\n\n\n<p><\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>What is Qwen 3?<\/strong><\/h2>\n\n\n\n<figure class=\"wp-block-image size-full\"><img decoding=\"async\" width=\"1200\" height=\"630\" src=\"https:\/\/www.guvi.in\/blog\/wp-content\/uploads\/2026\/04\/01-4.png\" alt=\"Qwen 3\" class=\"wp-image-106706\" srcset=\"https:\/\/www.guvi.in\/blog\/wp-content\/uploads\/2026\/04\/01-4.png 1200w, https:\/\/www.guvi.in\/blog\/wp-content\/uploads\/2026\/04\/01-4-300x158.png 300w, https:\/\/www.guvi.in\/blog\/wp-content\/uploads\/2026\/04\/01-4-768x403.png 768w, https:\/\/www.guvi.in\/blog\/wp-content\/uploads\/2026\/04\/01-4-150x79.png 150w\" sizes=\"(max-width: 1200px) 100vw, 1200px\" title=\"\"><\/figure>\n\n\n\n<p>Qwen 3 is a series of <a href=\"https:\/\/www.guvi.in\/blog\/guide-to-large-language-models\/\" target=\"_blank\" rel=\"noreferrer noopener\">LLMs<\/a> built by Alibaba. You can download the models for free and use them commercially under the Apache 2.0 license. They come in sizes from 0.6B to 235B parameters, so there is a model for a laptop as well as for a powerful server.<\/p>\n\n\n\n<p>Qwen 3 models can answer in two ways. In thinking mode, the model works through the problem step by step before answering, which helps with maths, logic, and coding. In non-thinking mode, it answers directly, which is faster and suits simple questions.<\/p>\n\n\n\n<p><\/p>\n\n\n\n<p><strong><em>Running Qwen 3 locally is only the beginning. If LLMs have sparked something in you, HCL GUVI\u2019s Intel &amp; IITM Pravartak Certified AI\/ML Course takes you from Python basics to building real AI systems, no prior coding experience needed. Explore the course.<\/em><\/strong><\/p>\n\n\n<div class=\"wp-block-guvi-cta-button\" style=\"justify-content:center;\"><a href=\"https:\/\/www.guvi.in\/mlp\/artificial-intelligence-and-machine-learning?utm_source=blog&#038;utm_medium=cta_button&#038;utm_campaign=setup-and-fine-tune-qwen-3-with-ollama\" class=\"guvi-btn-enquire\" target=\"_blank\" rel=\"noopener noreferrer\">Enroll in HCL GUVI&#039;s AI\/ML Course<svg class=\"guvi-btn-chevron\" style=\"transform:rotate(0deg);\" width=\"16\" height=\"16\" viewBox=\"0 0 24 24\" fill=\"none\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" aria-hidden=\"true\"><path fill-rule=\"evenodd\" clip-rule=\"evenodd\" d=\"M9.39935 18.1007C9.0674 17.7687 9.0674 17.2305 9.39935 16.8986L13.6922 12.6057C13.7508 12.5471 13.7508 12.4521 13.6922 12.3935L9.39935 8.10065C9.0674 7.7687 9.0674 7.23051 9.39935 6.89857C9.7313 6.56662 10.2695 6.56662 10.6014 6.89857L14.8943 11.1915C15.6168 11.9139 15.6168 13.0853 14.8943 13.8078L10.6014 18.1007C10.2695 18.4326 9.7313 18.4326 9.39935 18.1007Z\" fill=\"currentColor\"\/><\/svg><\/a><\/div>\n\n\n<p><\/p>\n\n\n\n<h3 class=\"wp-block-heading\"><strong>Qwen 3 Model Variants and Hardware Requirements<\/strong><\/h3>\n\n\n\n<p>Qwen 3 has two model types. Dense models (0.6B to 32B) use all their parameters for every token. <\/p>\n\n\n\n<p>Mixture-of-Experts (MoE) models (30B and 235B) activate only a part of their parameters per token, making them faster, but all their weights must still fit in memory.<\/p>\n\n\n\n<figure class=\"wp-block-table\"><table><thead><tr><th><strong>Ollama Tag<\/strong><\/th><th><strong>Type<\/strong><\/th><th><strong>Download Size<\/strong><\/th><\/tr><\/thead><tbody><tr><td><strong>qwen3:0.6b<\/strong><\/td><td>Dense<\/td><td>523MB<\/td><\/tr><tr><td><strong>qwen3:1.7b<\/strong><\/td><td>Dense<\/td><td>1.4GB<\/td><\/tr><tr><td><strong>qwen3:4b<\/strong><\/td><td>Dense<\/td><td>2.5GB<\/td><\/tr><tr><td><strong>qwen3:8b (default)<\/strong><\/td><td>Dense<\/td><td>5.2GB<\/td><\/tr><tr><td><strong>qwen3:14b<\/strong><\/td><td>Dense<\/td><td>9.3GB<\/td><\/tr><tr><td><strong>qwen3:30b<\/strong><\/td><td>MoE (3B active)<\/td><td>19GB<\/td><\/tr><tr><td><strong>qwen3:32b<\/strong><\/td><td>Dense<\/td><td>20GB<\/td><\/tr><tr><td><strong>qwen3:235b<\/strong><\/td><td>MoE (22B active)<\/td><td>142GB<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n<p>A model&#8217;s memory needs are its download size plus the KV cache, which grows with context length. For qwen3:8b, the cache is about 4.8GB at a 32K context. If a model does not fit in your GPU memory, reduce the context length before switching to a smaller model.<\/p>\n\n\n\n<p><strong>Minimum VRAM for fine-tuning (Unsloth, QLoRA 4-bit)<\/strong><\/p>\n\n\n\n<figure class=\"wp-block-table\"><table><thead><tr><th><strong>Model<\/strong><\/th><th><strong>Minimum VRAM<\/strong><\/th><\/tr><\/thead><tbody><tr><td><strong>Qwen3-8B<\/strong><\/td><td>6GB<\/td><\/tr><tr><td><strong>Qwen3-14B<\/strong><\/td><td>8.5GB<\/td><\/tr><tr><td><strong>Qwen3-30B-A3B<\/strong><\/td><td>17.5GB<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n<p>These are minimums. If you run out of memory, lower the batch size to 1 or 2.<\/p>\n\n\n\n<div style=\"background-color: #099f4e; border: 3px solid #110053; border-radius: 12px; padding: 18px 22px; color: #FFFFFF; font-size: 18px; font-family: Montserrat, Helvetica, sans-serif; line-height: 1.6; box-shadow: 0 4px 12px rgba(0, 0, 0, 0.15); max-width: 750px; margin: 22px auto;\">\n  <h4 style=\"margin-top: 0; font-size: 22px; font-weight: 700; color: #ffffff;\">\ud83d\udca1 Did You Know?<\/h4>\n  <ul style=\"padding-left: 20px; margin: 10px 0;\">\n    <li><strong style=\"color: #110053;\">Qwen3-32B<\/strong> delivers Qwen2.5-72B-level performance while fitting on a single RTX 4090 GPU with 24GB VRAM.<\/li>\n    <li><strong style=\"color: #110053;\">Qwen3<\/strong> supports 119 languages and dialects, including Hindi, Bengali, Tamil, and Telugu.<\/li>\n    <li>All <strong style=\"color: #110053;\">Qwen3 models<\/strong> use the Apache 2.0 licence, allowing free commercial use, modification, and distribution.<\/li>\n  <\/ul>\n<\/div>\n\n\n\n<p><\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>What is Ollama?<\/strong><\/h2>\n\n\n\n<figure class=\"wp-block-image size-full\"><img decoding=\"async\" width=\"1200\" height=\"630\" src=\"https:\/\/www.guvi.in\/blog\/wp-content\/uploads\/2026\/04\/02-3.png\" alt=\"Ollama\" class=\"wp-image-106708\" srcset=\"https:\/\/www.guvi.in\/blog\/wp-content\/uploads\/2026\/04\/02-3.png 1200w, https:\/\/www.guvi.in\/blog\/wp-content\/uploads\/2026\/04\/02-3-300x158.png 300w, https:\/\/www.guvi.in\/blog\/wp-content\/uploads\/2026\/04\/02-3-768x403.png 768w, https:\/\/www.guvi.in\/blog\/wp-content\/uploads\/2026\/04\/02-3-150x79.png 150w\" sizes=\"(max-width: 1200px) 100vw, 1200px\" title=\"\"><\/figure>\n\n\n\n<p>Ollama is a free, open-source tool that lets you run large language models on your own computer. It works on macOS, <a href=\"https:\/\/www.guvi.in\/blog\/what-is-the-difference-between-windows-and-linux\/\" target=\"_blank\" rel=\"noreferrer noopener\">Linux and Windows<\/a>. You download a model with one command and chat with it directly from the terminal.<\/p>\n\n\n\n<p>Ollama uses your GPU automatically if it finds a supported one, and falls back to the CPU if not. It also runs a local API at <code>http:\/\/localhost:11434<\/code>, so you can use the same models from Python, JavaScript or curl, and your data never leaves your machine.<\/p>\n\n\n\n<p><\/p>\n\n\n\n<h3 class=\"wp-block-heading\"><strong>Features of Ollama<\/strong><\/h3>\n\n\n\n<ul>\n<li><strong>Private by default:<\/strong> Models run on your machine, so your data never leaves it.<\/li>\n\n\n\n<li><strong>Automatic GPU use:<\/strong> Detects supported NVIDIA, AMD and Apple Silicon GPUs, and falls back to the CPU.<\/li>\n\n\n\n<li><strong>Cross-platform:<\/strong> Works on macOS, Linux and Windows with the same commands.<\/li>\n\n\n\n<li><strong>Local API:<\/strong> Serves a REST API at <code>http:\/\/localhost:11434<\/code> for use from any language.<\/li>\n\n\n\n<li><strong>Custom models:<\/strong> Builds tailored models from a base model with a Modelfile.<\/li>\n<\/ul>\n\n\n\n<p><\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>How to Install Ollama<\/strong><\/h2>\n\n\n\n<p>Ollama installs with a single command on macOS, Linux and Windows. Run the command for your system below.<\/p>\n\n\n\n<h4 class=\"wp-block-heading\"><strong>1. macOS<\/strong><\/h4>\n\n\n\n<p>Ollama needs macOS 14 (Sonoma) or newer. Apple Silicon Macs use the GPU, and Intel Macs run on the CPU only. Run this in Terminal:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>curl -fsSL https:\/\/ollama.com\/install.sh | sh<\/code><\/pre>\n\n\n\n<p>You can also download the <code>.dmg<\/code> file from <a href=\"https:\/\/ollama.com\/download\" target=\"_blank\" rel=\"noreferrer noopener nofollow\">ollama.com\/download<\/a> and drag Ollama into the Applications folder.<\/p>\n\n\n\n<h4 class=\"wp-block-heading\"><strong>2. Linux<\/strong><\/h4>\n\n\n\n<p>Run the same install script. It installs Ollama and, on systems with systemd, sets it up as a service that starts on boot.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>curl -fsSL https:\/\/ollama.com\/install.sh | sh<\/code><\/pre>\n\n\n\n<p>If the script asks for <code>zstd<\/code>, install it first:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code># Debian \/ Ubuntu\nsudo apt-get install zstd\n\n# RHEL \/ CentOS \/ Fedora\nsudo dnf install zstd<\/code><\/pre>\n\n\n\n<h4 class=\"wp-block-heading\"><strong>3. Windows<\/strong><\/h4>\n\n\n\n<p>Ollama needs Windows 10 22H2 or newer. Open PowerShell and run:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>irm https:\/\/ollama.com\/install.ps1 | iex<\/code><\/pre>\n\n\n\n<p>You can also download <code>OllamaSetup.exe<\/code> from ollama.com\/download and run it. Ollama keeps running in the background after the installation.<\/p>\n\n\n\n<h4 class=\"wp-block-heading\"><strong>4. Verifying the Installation<\/strong><\/h4>\n\n\n\n<p>Open a new terminal window and check the installed version:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>ollama --version<\/code><\/pre>\n\n\n\n<p>To confirm the Ollama server is running, call its local API. It returns the server version:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>curl http:\/\/localhost:11434\/api\/version<\/code><\/pre>\n\n\n\n<p>If the <code>ollama<\/code> command isn&#8217;t found on macOS or Windows, open the Ollama app once to start the server. On Linux, run <code>ollama serve<\/code> in a separate terminal.<\/p>\n\n\n\n<p><strong>Explore: <\/strong><a href=\"https:\/\/www.guvi.in\/blog\/how-to-audit-llm-models-for-backdoors\/\" target=\"_blank\" rel=\"noreferrer noopener\">How to Audit LLM Models for Backdoors: Tools &amp; Frameworks<\/a><\/p>\n\n\n\n<p><\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>Downloading and Running Qwen 3 Models<\/strong><\/h2>\n\n\n\n<figure class=\"wp-block-image size-full\"><img decoding=\"async\" width=\"1200\" height=\"630\" src=\"https:\/\/www.guvi.in\/blog\/wp-content\/uploads\/2026\/04\/03-2.png\" alt=\"Steps to download and run Qwen 3 model\" class=\"wp-image-106709\" srcset=\"https:\/\/www.guvi.in\/blog\/wp-content\/uploads\/2026\/04\/03-2.png 1200w, https:\/\/www.guvi.in\/blog\/wp-content\/uploads\/2026\/04\/03-2-300x158.png 300w, https:\/\/www.guvi.in\/blog\/wp-content\/uploads\/2026\/04\/03-2-768x403.png 768w, https:\/\/www.guvi.in\/blog\/wp-content\/uploads\/2026\/04\/03-2-150x79.png 150w\" sizes=\"(max-width: 1200px) 100vw, 1200px\" title=\"\"><\/figure>\n\n\n\n<p>Ollama names every model as <code>name:tag<\/code>, where the tag is the size, for example <code>qwen3:8b<\/code>. The 8B model is the default tag and a good starting point for most machines. Pick a different size from the table in the first section.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\"><strong>Step 1: Pull a Qwen 3 Model<\/strong><\/h3>\n\n\n\n<p>Download the model to your computer. You only need to do this once.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>ollama pull qwen3:8b<\/code><\/pre>\n\n\n\n<p>For a smaller model on limited hardware, use a smaller tag:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>ollama pull qwen3:4b<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\"><strong>Step 2: List Your Downloaded Models<\/strong><\/h3>\n\n\n\n<p>Check which models are installed. The output shows each model\u2019s name, ID, size and last modified time.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>ollama list<\/code><\/pre>\n\n\n\n<p>To delete a model and free up disk space:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>ollama rm qwen3:8b<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\"><strong>Step 3: Run Qwen 3 in Chat Mode<\/strong><\/h3>\n\n\n\n<p>Start an interactive chat in the terminal. If the model is not downloaded yet, Ollama pulls it first.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>ollama run qwen3:8b<\/code><\/pre>\n\n\n\n<p>Type your message at the <code>&gt;&gt;&gt;<\/code> prompt and press Enter. Qwen 3 prints its reasoning between <code>Thinking...<\/code> and <code>...done thinking.<\/code> before the final answer. To exit the chat, type:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>\/bye<\/code><\/pre>\n\n\n\n<p><\/p>\n\n\n\n<p><\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>Understanding Qwen 3 Thinking Mode<\/strong><\/h2>\n\n\n\n<p>In thinking mode, Qwen 3 works through a problem step by step before it gives the final answer. The reasoning appears in the output ahead of the answer. This improves accuracy in maths, logic and coding, but responses take longer.<\/p>\n\n\n\n<p>In non-thinking mode, the model skips the reasoning and answers directly. This is faster and suits simple questions, summaries and translation. Ollama turns thinking on by default. The updated 2507 Instruct and Thinking models run in only one mode, so they do not switch.<\/p>\n\n\n\n<h4 class=\"wp-block-heading\"><strong>1. Switching Thinking Mode in the CLI<\/strong><\/h4>\n\n\n\n<p>Set the mode with a flag when you start the chat:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code># Thinking on\nollama run qwen3:8b --think\n\n# Thinking off\nollama run qwen3:8b --think=false\n\n# Model still thinks, but only the final answer is shown\nollama run qwen3:8b --hidethinking<\/code><\/pre>\n\n\n\n<p>Inside a running chat, switch modes with these commands:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>\/set think\n\/set nothink<\/code><\/pre>\n\n\n\n<h4 class=\"wp-block-heading\"><strong>2. Controlling Thinking Mode via the API<\/strong><\/h4>\n\n\n\n<p>Add the <code>think<\/code> field to a chat request. Ollama returns the reasoning in <code>message.thinking<\/code> and the final answer in <code>message.content<\/code>, so your app can show or hide them separately.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>curl http:\/\/localhost:11434\/api\/chat -d '{\n  \"model\": \"qwen3:8b\",\n  \"messages\": &#91;{\"role\": \"user\", \"content\": \"How many r are in strawberry?\"}],\n  \"think\": true,\n  \"stream\": false\n}'<\/code><\/pre>\n\n\n\n<p>The same call with the Python client (<code>pip install ollama<\/code>):<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>from ollama import chat\n\nresponse = chat(\n    model=\"qwen3:8b\",\n    messages=&#91;{\"role\": \"user\", \"content\": \"How many r are in strawberry?\"}],\n    think=True,\n)\n\nprint(\"Thinking:\", response.message.thinking)\nprint(\"Answer:\", response.message.content)<\/code><\/pre>\n\n\n\n<p>Set <code>think<\/code> to <code>false<\/code> (<code>False<\/code> in Python) for fast replies without the reasoning.<\/p>\n\n\n\n<p><\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>Using Qwen 3 with Python<\/strong><\/h2>\n\n\n\n<p>Ollama runs a local REST API, and you can call it from Python in two ways: the official Ollama library, or the standard requests module. Before running the examples, make sure Ollama is running and the model is downloaded <code>ollama pull qwen3:8b<\/code>.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\"><strong>1. Basic Chat with the Python Client<\/strong><\/h3>\n\n\n\n<p>Install the library. It needs Python 3.8 or newer.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>pip install ollama<\/code><\/pre>\n\n\n\n<p>Send a message and print the reply:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>from ollama import chat\n\nreply = chat(\n    model=\"qwen3:8b\",\n    messages=&#91;{\"role\": \"user\", \"content\": \"Explain what a REST API is in two sentences.\"}],\n)\n\nprint(reply.message.content)<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\"><strong>2. Using the requests Module<\/strong><\/h3>\n\n\n\n<p>If you do not want an extra library, call the API directly. Set <code>stream<\/code> to <code>False<\/code> to get the full reply as one JSON object.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>pip install requests<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>import requests\n\npayload = {\n    \"model\": \"qwen3:8b\",\n    \"messages\": &#91;{\"role\": \"user\", \"content\": \"List three uses of a hash map.\"}],\n    \"stream\": False,\n}\n\nresponse = requests.post(\"http:\/\/localhost:11434\/api\/chat\", json=payload, timeout=120)\nresponse.raise_for_status()\n\nprint(response.json()&#91;\"message\"]&#91;\"content\"])<\/code><\/pre>\n\n\n\n<p>Add <code>\"think\": False<\/code> to the payload to skip the reasoning and get a faster reply.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\"><strong>3. Streaming Responses<\/strong><\/h3>\n\n\n\n<p>Streaming prints the reply as it is generated, which gives the typing effect seen in chat apps. Set <code>stream=True<\/code> and loop over the chunks:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>from ollama import chat\n\nstream = chat(\n    model=\"qwen3:8b\",\n    messages=&#91;{\"role\": \"user\", \"content\": \"Write a short checklist for debugging a slow SQL query.\"}],\n    stream=True,\n    think=False,\n)\n\nfor chunk in stream:\n    print(chunk.message.content, end=\"\", flush=True)<\/code><\/pre>\n\n\n\n<p>With thinking on, the reasoning arrives first in <code>chunk.message.thinking<\/code> while <code>content<\/code> stays empty, and the answer follows in <code>content<\/code>. The example turns thinking off so every chunk contains answer text.<\/p>\n\n\n\n<p><\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>Fine-Tuning Qwen 3 with a Modelfile<\/strong><\/h2>\n\n\n\n<p>Fine-tuning Qwen 3 means training it on your own examples so it follows your format, tone or subject without a long prompt each time. Ollama cannot train models, so you fine-tune with LoRA in a library such as Unsloth, then export the model as a GGUF file and run it in Ollama.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\"><strong>1. How the Workflow Works<\/strong><\/h3>\n\n\n\n<p>Ollama runs models but cannot train them. Fine-tuning is done in a separate library, and the result is then loaded into Ollama:<\/p>\n\n\n\n<ul>\n<li>Prepare a dataset of example conversations.<\/li>\n\n\n\n<li>Train a LoRA adapter on Qwen 3 with Unsloth. Hugging Face PEFT with TRL follows the same steps.<\/li>\n\n\n\n<li>Compare the tuned model with the base model on held-out examples.<\/li>\n\n\n\n<li>Export the merged model as a GGUF file.<\/li>\n\n\n\n<li>Create a Modelfile and load the model into Ollama.<\/li>\n<\/ul>\n\n\n\n<p>Training needs an NVIDIA GPU. A free Colab T4 is enough for Qwen3-14B, and the VRAM table in the first section shows the minimums. If you only need a fixed role, tone or settings, skip training and use a Modelfile alone (see Practical Modelfile Examples).<\/p>\n\n\n\n<p><strong>Also Read:<\/strong> <em><a href=\"https:\/\/www.guvi.in\/blog\/fine-tuning-llms-with-unsloth-and-ollama\/\" target=\"_blank\" rel=\"noreferrer noopener\">Fine-Tuning LLMs with Unsloth and Ollama<\/a><\/em><\/p>\n\n\n\n<p><\/p>\n\n\n\n<h3 class=\"wp-block-heading\"><strong>2. Dataset Format and LoRA Settings<\/strong><\/h3>\n\n\n\n<p>Use a JSONL file with one conversation per line. Keep a separate test.jsonl with examples the model never trains on for evaluation.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>{\"messages\": &#91;{\"role\": \"user\", \"content\": \"How do I reset my password?\"}, {\"role\": \"assistant\", \"content\": \"Open Settings, choose Security, then select Reset password. A reset link is sent to your registered email.\"}]}<\/code><\/pre>\n\n\n\n<p>To keep thinking mode working after training, Qwen recommends mixing about 75% reasoning examples with 25% non-reasoning ones.<\/p>\n\n\n\n<figure class=\"wp-block-table\"><table><thead><tr><th><strong>Setting<\/strong><\/th><th><strong>Value<\/strong><\/th><th><strong>What it does<\/strong><\/th><\/tr><\/thead><tbody><tr><td><strong>load_in_4bit<\/strong><\/td><td>True<\/td><td>Loads the base model in 4-bit (QLoRA) to save VRAM<\/td><\/tr><tr><td><strong>max_seq_length<\/strong><\/td><td>2048<\/td><td>Longest sequence used in training<\/td><\/tr><tr><td><strong>r<\/strong><\/td><td>16<\/td><td>Adapter rank. Higher values (up to 64) learn more but need more VRAM<\/td><\/tr><tr><td><strong>lora_alpha<\/strong><\/td><td>16<\/td><td>Adapter scaling. Keep it equal to <code>r<\/code><\/td><\/tr><tr><td><strong>lora_dropout<\/strong><\/td><td>0<\/td><td>No dropout on the adapter<\/td><\/tr><tr><td><strong>target_modules<\/strong><\/td><td>all attention and MLP projections<\/td><td>Layers that receive adapters<\/td><\/tr><tr><td><strong>learning_rate<\/strong><\/td><td>2e-4<\/td><td>Step size for training<\/td><\/tr><tr><td><strong>batch size x accumulation<\/strong><\/td><td>1 x 4<\/td><td>Effective batch of 4. Lower the batch size if you run out of memory<\/td><\/tr><tr><td><strong>max_steps<\/strong><\/td><td>100<\/td><td>Short first run. Increase it, or use epochs, for real training<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n<h3 class=\"wp-block-heading\"><strong>3. Training the Model<\/strong><\/h3>\n\n\n\n<pre class=\"wp-block-code\"><code>pip install unsloth<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>from unsloth import FastLanguageModel\nfrom datasets import load_dataset\nfrom trl import SFTTrainer, SFTConfig\n\nmodel, tokenizer = FastLanguageModel.from_pretrained(\n    model_name=\"unsloth\/Qwen3-8B\",\n    max_seq_length=2048,\n    load_in_4bit=True,\n)\n\nmodel = FastLanguageModel.get_peft_model(\n    model,\n    r=16,\n    lora_alpha=16,\n    lora_dropout=0,\n    target_modules=&#91;\"q_proj\", \"k_proj\", \"v_proj\", \"o_proj\",\n                    \"gate_proj\", \"up_proj\", \"down_proj\"],\n    use_gradient_checkpointing=\"unsloth\",\n    random_state=3407,\n)\n\ndataset = load_dataset(\"json\", data_files={\"train\": \"train.jsonl\", \"test\": \"test.jsonl\"})\n\ndef to_text(example):\n    return {\"text\": tokenizer.apply_chat_template(example&#91;\"messages\"], tokenize=False)}\n\ndataset = dataset.map(to_text)\n\ntrainer = SFTTrainer(\n    model=model,\n    tokenizer=tokenizer,\n    train_dataset=dataset&#91;\"train\"],\n    args=SFTConfig(\n        dataset_text_field=\"text\",\n        per_device_train_batch_size=1,\n        gradient_accumulation_steps=4,\n        warmup_steps=10,\n        max_steps=100,\n        learning_rate=2e-4,\n        logging_steps=10,\n        optim=\"adamw_8bit\",\n        output_dir=\"outputs\",\n        seed=3407,\n    ),\n)\n\ntrainer.train()<\/code><\/pre>\n\n\n\n<p>The <code>apply_chat_template<\/code> step formats each conversation with Qwen 3\u2019s own template. Training and later use in Ollama must use the same template.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\"><strong>4. Before and After Evaluation<\/strong><\/h3>\n\n\n\n<p>Run the same unseen prompts through the base model and the tuned model, then compare the answers. Run the script below once the tuned model is loaded in Ollama; the next two steps cover that. The tuned model is called <code>qwen3-support<\/code> here.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>import json\nimport re\nfrom ollama import chat\n\ndef ask(model, prompt):\n    reply = chat(model=model, messages=&#91;{\"role\": \"user\", \"content\": prompt}])\n    return re.sub(r\"&lt;think&gt;.*?&lt;\/think&gt;\", \"\", reply.message.content, flags=re.DOTALL).strip()\n\nwith open(\"test.jsonl\") as f:\n    rows = &#91;json.loads(line) for line in f]\n\nfor row in rows&#91;:10]:\n    prompt = row&#91;\"messages\"]&#91;0]&#91;\"content\"]\n    print(\"PROMPT:  \", prompt)\n    print(\"EXPECTED:\", row&#91;\"messages\"]&#91;1]&#91;\"content\"])\n    print(\"BASE:    \", ask(\"qwen3:8b\", prompt))\n    print(\"TUNED:   \", ask(\"qwen3-support\", prompt))\n    print(\"-\" * 40)<\/code><\/pre>\n\n\n\n<p>For tasks with a fixed answer, such as classification, count how many answers match the expected one for each model. The tuned model should score clearly higher. If it doesn&#8217;t, fix the dataset before retraining.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\"><strong>5. Exporting the Model<\/strong><\/h3>\n\n\n\n<p>This call merges the adapter into the base model and writes a quantized GGUF file in one step:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>model.save_pretrained_gguf(\"qwen3-support-gguf\", tokenizer, quantization_method=\"q4_k_m\")<\/code><\/pre>\n\n\n\n<p><code>q4_k_m<\/code> is the recommended balance of size and quality. Use <code>q8_0<\/code> for higher quality or <code>f16<\/code> for full precision. Unsloth also writes a ready-made Modelfile into the export folder.<\/p>\n\n\n\n<p>If you trained with plain PEFT instead, merge the adapter with <code>merge_and_unload()<\/code>, then convert the merged model to GGUF with <code>convert_hf_to_gguf.py<\/code> from llama.cpp.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\"><strong>6. Creating the Modelfile and Running It in Ollama<\/strong><\/h3>\n\n\n\n<p>Open the generated Modelfile in the export folder. It points to the GGUF file and contains the Qwen 3 <code>TEMPLATE<\/code> used in training. Edit only the lines you need and leave the <code>TEMPLATE<\/code> block as it is:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>SYSTEM \"\"\"You are a support assistant for Acme Cloud. Answer briefly and accurately.\"\"\"\nPARAMETER temperature 0.2\nPARAMETER num_ctx 4096<\/code><\/pre>\n\n\n\n<p>Build and run the model:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>cd qwen3-support-gguf\nollama create qwen3-support -f Modelfile\nollama run qwen3-support<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\"><strong>7. Practical Modelfile Examples<\/strong><\/h3>\n\n\n\n<p>A Modelfile on its own does not train anything. It fixes a system prompt and settings on top of a base model, which is enough for many tasks.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>FROM qwen3:14b\nSYSTEM \"\"\"Review the code you receive. List bugs first, then suggest improvements, and explain each point briefly.\"\"\"\nPARAMETER temperature 0.2<\/code><\/pre>\n\n\n\n<figure class=\"wp-block-table\"><table><thead><tr><th><strong>Use Case<\/strong><\/th><th><strong>Base Model<\/strong><\/th><th><strong>SYSTEM Prompt<\/strong><\/th><th><strong>Parameters<\/strong><\/th><\/tr><\/thead><tbody><tr><td><strong>Code reviewer<\/strong><\/td><td>qwen3:14b<\/td><td>Review code, list bugs, suggest fixes<\/td><td>temperature 0.2<\/td><\/tr><tr><td><strong>Support assistant<\/strong><\/td><td>qwen3:8b<\/td><td>Product name, known issues, escalate when unsure<\/td><td>num_ctx 8192<\/td><\/tr><tr><td><strong>Classifier<\/strong><\/td><td>qwen3:4b<\/td><td>Reply with one label only<\/td><td>temperature 0<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n<p><strong>Also Read:<\/strong> <em><a href=\"https:\/\/www.guvi.in\/blog\/how-to-fine-tune-large-language-models\/\" target=\"_blank\" rel=\"noreferrer noopener\">How to Fine-Tune Large Language Models (LLMs)?<\/a><\/em><\/p>\n\n\n\n<p><\/p>\n\n\n\n<p><strong><em>Every good model starts with a few failed training runs. HCL GUVI\u2019s Intel &amp; IITM Pravartak Certified AI\/ML Course teaches LoRA, PEFT, dataset preparation and evaluation with live classes and 1:1 doubt sessions, so expert help is within reach. See what you will learn.<\/em><\/strong><\/p>\n\n\n<div class=\"wp-block-guvi-cta-button\" style=\"justify-content:center;\"><a href=\"https:\/\/www.guvi.in\/mlp\/artificial-intelligence-and-machine-learning?utm_source=blog&#038;utm_medium=cta_button&#038;utm_campaign=setup-and-fine-tune-qwen-3-with-ollama\" class=\"guvi-btn-enquire\" target=\"_blank\" rel=\"noopener noreferrer\">Enroll in HCL GUVI&#039;s AI\/ML Course<svg class=\"guvi-btn-chevron\" style=\"transform:rotate(0deg);\" width=\"16\" height=\"16\" viewBox=\"0 0 24 24\" fill=\"none\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" aria-hidden=\"true\"><path fill-rule=\"evenodd\" clip-rule=\"evenodd\" d=\"M9.39935 18.1007C9.0674 17.7687 9.0674 17.2305 9.39935 16.8986L13.6922 12.6057C13.7508 12.5471 13.7508 12.4521 13.6922 12.3935L9.39935 8.10065C9.0674 7.7687 9.0674 7.23051 9.39935 6.89857C9.7313 6.56662 10.2695 6.56662 10.6014 6.89857L14.8943 11.1915C15.6168 11.9139 15.6168 13.0853 14.8943 13.8078L10.6014 18.1007C10.2695 18.4326 9.7313 18.4326 9.39935 18.1007Z\" fill=\"currentColor\"\/><\/svg><\/a><\/div>\n\n\n<p><\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>Troubleshooting Qwen 3 and Ollama<\/strong><\/h2>\n\n\n\n<p>Start with <code>ollama ps<\/code>, which shows whether a model runs on the GPU or CPU and what context size it uses.<\/p>\n\n\n\n<ul>\n<li><strong><code>connection refused<\/code> on port 11434:<\/strong> The Ollama server is not running. Start it with <code>ollama serve<\/code>, or <code>sudo systemctl start ollama<\/code> on Linux.<\/li>\n\n\n\n<li><strong>Slow responses:<\/strong> If <code>ollama ps<\/code> shows CPU or a CPU\/GPU split, the model does not fit in GPU memory. Use a smaller tag or a shorter context.<\/li>\n\n\n\n<li><strong>Out of memory:<\/strong> Memory use is the model weights plus the context cache. Pull a smaller tag such as <code>qwen3:4b<\/code> or reduce <code>num_ctx<\/code>.<\/li>\n\n\n\n<li><strong>Qwen 3 repeats or loops:<\/strong> The context window is usually too small, or the temperature is 0. Raise it with <code>\/set parameter num_ctx 8192<\/code> and use the sampling values in the tips below.<\/li>\n\n\n\n<li><strong>Fine-tuned model gives gibberish in Ollama:<\/strong> The chat template differs from the one used in training. Use the Modelfile Unsloth generated and leave its <code>TEMPLATE<\/code> block unchanged.<\/li>\n<\/ul>\n\n\n\n<p><\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>Tips for Getting the Best Results with Qwen 3 and Ollama<\/strong><\/h2>\n\n\n\n<p>These settings keep Qwen 3 fast and stable.<\/p>\n\n\n\n<ul>\n<li><strong>Match the model to your GPU memory:<\/strong> A model that fits fully on the GPU runs much faster than one split between the GPU and CPU. Look for <code>100% GPU<\/code> in <code>ollama ps<\/code>.<\/li>\n\n\n\n<li><strong>Use Qwen\u2019s sampling values:<\/strong> Thinking mode uses temperature 0.6, top_p 0.95, top_k 20 and min_p 0. Non-thinking mode uses temperature 0.7, top_p 0.8, top_k 20 and min_p 0. Avoid temperature 0.<\/li>\n\n\n\n<li><strong>Set the context window on purpose:<\/strong> Ollama\u2019s default is small, so raise <code>num_ctx<\/code> for long inputs. A larger context uses more memory.<\/li>\n\n\n\n<li><strong>Turn thinking off for simple tasks:<\/strong> Use <code>think=False<\/code> for extraction, classification and summaries, and keep it on for maths, logic and code.<\/li>\n\n\n\n<li><strong>Keep the model loaded:<\/strong> Ollama unloads a model after 5 minutes by default. Set <code>OLLAMA_KEEP_ALIVE<\/code> (for example <code>30m<\/code>) to avoid the reload delay.<\/li>\n<\/ul>\n\n\n\n<p><\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>FAQs<\/strong><\/h2>\n\n\n<div id=\"rank-math-faq\" class=\"rank-math-block\">\n<div class=\"rank-math-list \">\n<div id=\"faq-question-1794815126753\" class=\"rank-math-list-item\">\n<h3 class=\"rank-math-question \">1. How do I run Qwen 3 locally with Ollama?<\/h3>\n<div class=\"rank-math-answer \">\n\n<p>Install Ollama, then run <code>ollama run qwen3:8b<\/code>, which downloads the model on first use and opens a chat.<\/p>\n\n<\/div>\n<\/div>\n<div id=\"faq-question-1794815127911\" class=\"rank-math-list-item\">\n<h3 class=\"rank-math-question \">2. How much GPU memory do I need to run Qwen 3?<\/h3>\n<div class=\"rank-math-answer \">\n\n<p>Memory use is the download size plus the context cache, so <code>qwen3:8b<\/code> (5.2GB) fits on an 8GB GPU at a modest context length.<\/p>\n\n<\/div>\n<\/div>\n<div id=\"faq-question-1794815128617\" class=\"rank-math-list-item\">\n<h3 class=\"rank-math-question \">3. Can I fine-tune Qwen 3 with Ollama?<\/h3>\n<div class=\"rank-math-answer \">\n\n<p>No, Ollama only runs models, so fine-tune with Unsloth or Hugging Face PEFT, export a GGUF file and load it with a Modelfile.<\/p>\n\n<\/div>\n<\/div>\n<div id=\"faq-question-1794815129468\" class=\"rank-math-list-item\">\n<h3 class=\"rank-math-question \">4. How do I turn off thinking mode in Qwen 3?<\/h3>\n<div class=\"rank-math-answer \">\n\n<p>Start the chat with <code>ollama run qwen3:8b --think=false<\/code>, type <code>\/set nothink<\/code> during a chat, or pass <code>think=False<\/code> in the API.<\/p>\n\n<\/div>\n<\/div>\n<div id=\"faq-question-1794815130553\" class=\"rank-math-list-item\">\n<h3 class=\"rank-math-question \">5. Is Qwen 3 free for commercial use?<\/h3>\n<div class=\"rank-math-answer \">\n\n<p>Yes, the open-weight Qwen 3 models are released under the Apache 2.0 license, which permits commercial use.<\/p>\n\n<\/div>\n<\/div>\n<\/div>\n<\/div>","protected":false},"excerpt":{"rendered":"<p>Running a large language model (LLM) used to mean paying for an API and sending your data to someone else&#8217;s servers. Today, open models like Qwen 3 run on an ordinary laptop or desktop, and Ollama makes starting one as simple as a single command. This guide takes you from installation to a model trained [&hellip;]<\/p>\n","protected":false},"author":64,"featured_media":106705,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[933],"tags":[],"views":"16003","authorinfo":{"name":"Abhishek Pati","url":"https:\/\/www.guvi.in\/blog\/author\/abhishek-pati\/"},"thumbnailURL":"https:\/\/www.guvi.in\/blog\/wp-content\/uploads\/2026\/04\/Feature-image-2-300x116.png","_links":{"self":[{"href":"https:\/\/www.guvi.in\/blog\/wp-json\/wp\/v2\/posts\/105949"}],"collection":[{"href":"https:\/\/www.guvi.in\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.guvi.in\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.guvi.in\/blog\/wp-json\/wp\/v2\/users\/64"}],"replies":[{"embeddable":true,"href":"https:\/\/www.guvi.in\/blog\/wp-json\/wp\/v2\/comments?post=105949"}],"version-history":[{"count":15,"href":"https:\/\/www.guvi.in\/blog\/wp-json\/wp\/v2\/posts\/105949\/revisions"}],"predecessor-version":[{"id":148214,"href":"https:\/\/www.guvi.in\/blog\/wp-json\/wp\/v2\/posts\/105949\/revisions\/148214"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.guvi.in\/blog\/wp-json\/wp\/v2\/media\/106705"}],"wp:attachment":[{"href":"https:\/\/www.guvi.in\/blog\/wp-json\/wp\/v2\/media?parent=105949"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.guvi.in\/blog\/wp-json\/wp\/v2\/categories?post=105949"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.guvi.in\/blog\/wp-json\/wp\/v2\/tags?post=105949"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}