{"id":105949,"date":"2026-04-06T17:41:46","date_gmt":"2026-04-06T12:11:46","guid":{"rendered":"https:\/\/www.guvi.in\/blog\/?p=105949"},"modified":"2026-04-06T17:41:48","modified_gmt":"2026-04-06T12:11:48","slug":"setup-and-fine-tune-qwen-3-with-ollama","status":"publish","type":"post","link":"https:\/\/www.guvi.in\/blog\/setup-and-fine-tune-qwen-3-with-ollama\/","title":{"rendered":"Setup and Fine-Tune Qwen 3 with Ollama: Complete Guide (2026)"},"content":{"rendered":"\n

What if you could run one of the most capable open-source AI models in the world entirely on your own laptop, with no cloud fees, no API keys, and no internet connection after the initial download? That is exactly what you get when you set up Qwen 3 with Ollama, making it one of the most powerful local LLM setups available in 2026. The combination of Alibaba’s powerful Qwen 3 model family and Ollama’s dead-simple local LLM inference platform has made running state-of-the-art AI locally more accessible than ever.<\/p>\n\n\n\n

This blog walks you through every step: what Qwen 3 is, what Ollama does, how to install and run different Qwen 3 models, how to control thinking mode, how to connect Qwen 3 to Python, and how to fine-tune Qwen 3 with a custom Modelfile. By the end, you will know how to fine-tune a model for a specific task using nothing but a text file. Everything in this guide is beginner-friendly and verified against current documentation.<\/p>\n\n\n\n

Quick Answer <\/strong><\/p>\n\n\n\n

To set up Qwen 3 with Ollama, install Ollama from Ollama.com, run Ollama pull qwen3:8b<\/strong> to download the model, and run Ollama run qwen3:8b<\/strong> to start chatting. It runs entirely on your local machine with full privacy and zero cloud cost. You can customize the model’s behaviour by creating a Modelfile and using Ollama create<\/strong> to build a fine-tuned version.<\/p>\n\n\n\n

What is Qwen 3?<\/strong><\/h2>\n\n\n\n

Qwen 3 is the third generation of large language models developed by Alibaba. It was released on April 28-29, 2025, and represents a significant leap forward from Qwen 2.5. The model family includes eight main variants and six specialised models for retrieval and ranking tasks.<\/p>\n\n\n\n

The model was trained on 36 trillion tokens across 119 languages, nearly double the training data<\/a> used for Qwen 2.5. This makes it one of the most multilingual open-source model families available. All models in the family are released under the Apache 2.0 licence, which means you can download, use, and build on them for both personal and commercial projects without any restrictions.<\/p>\n\n\n\n

Qwen 3 Model Variants and Hardware Requirements<\/strong><\/h3>\n\n\n\n

Qwen 3 comes in two architectural types: dense models and Mixture-of-Experts (MoE) models. Dense models are straightforward and easier to understand. MoE models are more complex but activate only a fraction of their parameters during each inference step, which means a model with 30 billion total parameters might only use 3 billion parameters per response, keeping it fast and memory-efficient.<\/p>\n\n\n\n

Here is a plain-language breakdown of the available Qwen 3 models and what hardware you need to run them comfortably with Ollama.<\/p>\n\n\n\n

Model<\/strong><\/td>Parameters<\/strong><\/td>Disk Size<\/strong><\/td>Minimum VRAM<\/strong><\/td>Best For<\/strong><\/td><\/tr>
qwen3:0.6b<\/td>0.6B dense<\/td>~400MB<\/td>4GB<\/td>Testing, very fast responses<\/td><\/tr>
qwen3:1.7b<\/td>1.7B dense<\/td>~1.1GB<\/td>4GB<\/td>Basic tasks on limited hardware<\/td><\/tr>
qwen3:4b<\/td>4B dense<\/td>~2.6GB<\/td>6GB<\/td>Good starter model<\/td><\/tr>
qwen3:8b<\/td>8B dense<\/td>~5.2GB<\/td>8GB<\/td>Best default choice<\/td><\/tr>
qwen3:14b<\/td>14B dense<\/td>~9GB<\/td>10-12GB<\/td>Strong reasoning quality<\/td><\/tr>
qwen3:32b<\/td>32B dense<\/td>~20GB<\/td>20-24GB<\/td>Best quality single GPU<\/td><\/tr>
qwen3:30b-a3b<\/td>30B MoE (3B active)<\/td>~18GB<\/td>19-24GB<\/td>Efficient for 24GB GPU<\/td><\/tr>
qwen3:235b-a22b<\/td>235B MoE (22B active)<\/td>~140GB<\/td>140GB+<\/td>Maximum quality, multi-GPU<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n

The recommended starting point for most developers new to local LLM inference is qwen3:8b<\/strong>. It runs on any machine with 8GB of VRAM or 16GB of system RAM, delivers strong performance across reasoning, coding, and writing tasks, and downloads in a few minutes on a decent internet connection.<\/p>\n\n\n\n

Riddle:<\/em><\/strong> The qwen3:30b-a3b model has 30 billion total parameters but only activates 3 billion during each inference step. A developer with an 18GB GPU is choosing between this MoE model and the qwen3:14b dense model. Which one should they choose, and why?<\/em><\/p>\n\n\n\n

Answer:<\/em><\/strong> The qwen3:30b-a3b MoE model is generally the better choice. Because it only activates 3 billion parameters per step, it fits comfortably in 18-24GB of VRAM despite its large total parameter count. It offers more overall model capacity and stronger performance than the 14B dense model, while using similar or less memory during actual inference. The “30B” in the name refers to stored knowledge, not runtime cost.<\/em><\/p>\n\n\n\n

What is Ollama?<\/strong><\/h2>\n\n\n\n

Ollama is a free, open-source tool that makes running a local LLM<\/a> on your own machine as simple as running a command in your terminal. It handles model downloading, version management, GPU acceleration, and serving a local LLM API, all automatically.<\/p>\n\n\n\n

When you run Ollama pull qwen3:8b<\/strong>, Ollama downloads the model in GGUF format, which is the standard format for local CPU and GPU inference. GGUF bundles the model weights, tokeniser, and metadata into a single file, so there are no Python environments to configure and no separate config files to manage. When you run Ollama run qwen3:8b<\/strong>, your local LLM starts immediately in chat mode.<\/p>\n\n\n\n

Ollama also runs a local REST API server on port 11434. This means you can integrate any local LLM into Python, JavaScript, or curl-based applications using standard HTTP requests, without sending your data to any cloud service.<\/p>\n\n\n\n