How to Run Local LLMs Using Tailscale, Ollama, and OpenWebUI

by

Running large language models (LLMs) locally on your own computer can offer numerous advantages, such as enhanced privacy, reduced latency, and the ability to experiment with cutting-edge AI without relying on cloud services. In this blog post, we’ll guide you through setting up a local LLM environment using Tailscale for network connectivity, Ollama for model management, and OpenWebUI for a user-friendly interface.

Prerequisites

Before we dive into the setup process, ensure you have the following:

  • A computer with sufficient hardware (8-16 GB RAM recommended).
  • Basic knowledge of command-line operations.
  • Tailscale installed on your devices.
  • Ollama installed and configured.
  • OpenWebUI installed and set up.

Step 1: Set Up Tailscale

Tailscale makes creating software-defined networks easy: securely connecting users, services, and devices. Here’s how to get started:

  1. Install Tailscale : Visit Tailscale’s website and follow the installation instructions for your operating system.
  2. Create an Account : Sign up for a free account or log in if you already have one.
  3. Connect Devices : Install Tailscale on all devices you want to connect, including your local computer and any remote devices you wish to access the LLM from.

Step 2: Set Up Ollama

Ollama is an open-source backend service that manages and serves LLMs. It provides a simple API for loading and interacting with models. Here’s how to set it up:

1. Install Ollama : Follow the installation instructions on Ollama’s GitHub repository .

2. Load Models : Use Ollama to load your desired LLMs. For example, you can use the following command to load a model:

ollama model run <model_name>

3. Start the Service : Run Ollama as a service to ensure it’s always available.

ollama serve

Step 3: Set Up OpenWebUI

OpenWebUI is a web-based frontend that provides a user-friendly interface for interacting with LLMs managed by Ollama. Here’s how to get it set up:

  1. Install OpenWebUI : Follow the installation instructions on OpenWebUI’s documentation

Step 4: Configure Tailscale to serve OpenWebUI

Run the following command to serve OpenWebUI

sudo tailscale funnel 3000

Accessing Your Local LLM

Once everything is set up, you can access your local LLM from any device connected to your Tailscale network. Here’s how:

  1. Open Browser : On any device connected to Tailscale, open a web browser and navigate to the URL of your OpenWebUI instance.
  2. Start Chatting : Use the interface provided by OpenWebUI to interact with your LLMs.

Additional Tips

  • Keep It Running : Ensure your local computer is always on and connected to Tailscale if you want to access the LLM from remote devices.
  • Network Interference : Be aware that network issues can affect the availability of your LLM. Consider using a reliable internet connection and keeping your Tailscale service active.

Conclusion

Running local LLMs using Tailscale, Ollama, and OpenWebUI is a straightforward process that offers significant benefits. By following these steps, you can set up a robust local AI environment that enhances privacy, reduces latency, and provides a seamless user experience. Whether you’re experimenting with new models or building custom AI systems, this setup is a great starting point.

Happy coding!