Running LLM locally

Setting up open soure llm locally and builing simple application on top of it
llm
gemini
AI
python
Author

Aakash Basnet

Published

January 12, 2025

LLM and AI (Generated by Imagen3)

Installation

To get started with running a Large Language Model (LLM) locally, you’ll need to install a few tools and libraries. Below are the steps with explanations for each:

1. Install the Ollama Python module

Ollama is a tool that allows you to run open-source LLMs on your local machine. Install it using pip:

pip install ollama

2. Pull and run your chosen model

Ollama supports various models. For this tutorial, we’ll use the Gemma2 2B model, which is both small and powerful. Pull the model with:

ollama pull gemma3:1b

This command downloads the model weights to your machine so you can use them locally.

3. Install LangChain-Ollama integration

LangChain is a framework for developing applications powered by language models. The langchain-ollama module allows you to use Ollama models with LangChain:

pip install langchain-ollama

With these steps, your environment is ready to run and experiment with local LLMs.

Import Model

Now, let’s load the Gemma2 2B model using Ollama and LangChain. We’ll also discuss the concept of “temperature” in LLMs:

  • Model Loading: We use the OllamaLLM class from langchain_ollama to load the model. This allows us to interact with the LLM locally.
  • Temperature Parameter:
    • Setting temperature=0 makes the model’s output more deterministic and consistent. This is ideal for tasks where you want reliable, repeatable results (e.g., translation, summarization).
    • Higher temperatures (closer to 1) make the model more creative and introduce randomness, which is useful for creative writing or brainstorming.

In this experiment, we set the temperature to 0 for predictable outputs.

from langchain_ollama.llms import OllamaLLM
from langchain_core.prompts import ChatPromptTemplate
import json
from pprint import pprint

llm = OllamaLLM(model="gemma3:1b",
                temperature = 0)

Advantages of Running LLM Locally

Running a Large Language Model (LLM) on your own machine offers several benefits compared to using cloud-based APIs:

  • Privacy: Your data never leaves your computer, ensuring sensitive information remains secure.
  • No API Costs: You avoid recurring charges or usage limits imposed by cloud providers.
  • Speed: Local inference can be faster, especially for small models and repeated queries.
  • Customization: You can experiment with different models, parameters, and workflows without restrictions.
  • Offline Access: You can use the model even without an internet connection.
  • No Vendor Lock-in: You are not tied to a specific provider or platform.

These advantages make local LLMs ideal for prototyping, research, and privacy-sensitive applications.

Building a Simple LLM Agent

Let’s build a simple LLM-powered agent that translates text from one language to several others. We’ll use a prompt template to instruct the model and process the results programmatically.

Goal: - Translate a given sentence from English to Spanish, German, and Korean. - Receive the output in a structured JSON format for easy parsing.

prompt = ChatPromptTemplate.from_messages(
    [
        (
            "system",
            """
            You are a helpful assistant that translates {input_language} to {output_language}. 
            Provides output in json format as language as key and outpus as value
            """,
        ),
        ("human", "{input}"),
    ]
)

chain = prompt | llm

ai_message = chain.invoke(
    {
        "input_language": "English",
        "output_language": "Spanish, German, Korean",
        "input": "I love grilled chicken",
    }
)

pprint(ai_message)
('```json\n'
 '{\n'
 '  "language": "Spanish",\n'
 '  "output": "Me encanta el pollo a la parrilla."\n'
 '}\n'
 '```\n')

Parsing the Model Response

The response from the LLM is returned as a string, even if it looks like JSON. To work with the translations programmatically, we need to convert this string into a Python dictionary.

Below, we clean up the response and parse it as JSON so we can easily access each translated sentence.

# Replace and assign back to original content
ai_message = ai_message.replace("```json", "")
ai_message = ai_message.replace("```", "")

# Don't forget to convert to JSON as it is a string right now:
json_result = json.loads(ai_message)
pprint(json_result)
{'language': 'Spanish', 'output': 'Me encanta el pollo a la parrilla.'}
Back to top