Google Gemini's “Ragnarok scenery image”

Host Ollama using Ngrok

Spoiler: Run Ollama and use ngrok to expose your Mac mini to the internet

Who should read this?

• If you are looking for on Private Team or Organization AI Assistant, This guide is for you.
• If you are looking for a Private LLM REST API that is hosted on your team or organization Mac mini, This guide is also for you.
• You fit the criteria above and also want a quick and easy setup.

Prerequisite

• You have a Mac mini, or Macbook(Other platforms should work too)
• You understand how REST API works, or use REST API before.

Tunnel Tools

• Ngrok is a secure reverse proxy service(aka expose your laptop/cloud to the internet, or in other words, make it accessible from the internet) that comes with great features such as free domain. “You can think of ngrok as the front door to your applications” — ngrok doc
• Alternative: https://zrok.io/ is another candidate recommended by the reader. On the front page, it said “An open source sharing solution built on OpenZiti, the zero trust networking platform. Available as SaaS or self-hosted.”(not included here)

Large Language Model Tools

• Ollama is a software package that allows you to run the LLM model on your local. With a few clicks, you got the REST API of LLM of your choosing running on your local machine.
• Alternative: Hugging Face Transformer Library with your Favorite Language(not included here)

Overview

Here is a list of what we are going to do to achieve this setup.

• Install and Run Ollama
• Install and Run Ngrok
• Test the Setup from an outside network(Mobile or Outside network Laptop)

Install and Run Ollama

• Go to https://ollama.com/
• Download and Run a model, maybe LLama2

Install and Run Ngrok

• Go to https://ngrok.com/docs/getting-started/
• Only step 3 that you need to change

ngrok http http://localhost:11434

Because Ollama is running at http://localhost:11434

Test the Setup

From MacOS/Linux

Run curl in your terminal

curl http://localhost:11434/

From Mobile Phone(Android)

Download API Tester App

and use the same curl as MacOS/Linux

More Configuration and Integrations

• You can increase security layer for ngrok by adding basic auth.
• You can Set Up a Custom Domain with in ngrok.
• You can check out Community Integrations Section for more usage of Ollama