Running LLMs Locally: A Quick Guide to LM Studio

Why should you run a large language model (LLM) like ChatGPT locally, on your own computer? Because you can, that’s why, and LM Studio makes it easier than ever to get started.

Countless models are waiting to be experimented with in this accessible and versatile desktop app.

Local LLM Limitations

An unlisted but important step in the process is setting expectations. What not to expect, among other things, is getting something like GPT-4 running on your Chromebook.

As the name suggests, large language models perform a range of language-based tasks based on training on extensive datasets. They cannot magically do much of anything if it falls outside their scope and training.

A local LLM is consequently limited by its training/dataset capabilities and by extension its computational resource requirements. This is not an issue for cloud-based online LLMs, but their local counterparts will quickly and easily run out of processing power and memory.

That said, open-source models are continuously improving and can be very useful. Many are specialized in certain areas, where they may provide personal assistance e.g. with coding or web development.

LM Studio System Requirements

LM Studio’s hardware requirements are quite specific, and these are just for basic compatibility. Higher is better, especially when it comes to RAM and GPU (graphics card).

Compatible Devices

• Mac: ARM-based Apple Silicon Mac (M1/M2/M3) with macOS 13.6 Ventura or newer.
• Windows/Linux: Must have a processor that supports AVX2. This includes most Intel and AMD CPUs from the past decade.

Memory and Graphics

• A minimum of 16GB of RAM is recommended.
• For PCs, a graphics card with at least 6GB of VRAM (graphics memory) is recommended. NVIDIA and AMD GPUs are supported.

Plenty of storage is also needed. LM Studio’s download size is not an issue, but the size of additional models starts at several gigabytes each.

Installing and Configuring LM Studio

With such disclaimers out of the way, here are the steps to get started:

1. Go to lmstudio.ai to download the latest version of LM Studio for your platform.
2. Once downloaded, open the installer and follow the on-screen instructions to install LM Studio on your device.
3. Launch LM Studio via the newly created shortcut/app icon.

Download and Install LLM Models

With LM Studio open, use the search bar to find models by name. I will be looking to try Mistral (or Mixtral), which are some of the most popular models right now. The GUI filters out all but the compatible models with the GGUF file extension.

LM Studio’s search functionality includes a “compatibility guess” that tries to determine if the model will fit in your GPU’s VRAM. In the case of Apple Silicon Macs with integrated graphics, this will be a chunk of your system RAM (as opposed to a dedicated graphics card that has onboard, separate memory chips).

LM Studio will warn of models that are too large for your system. Note that the relationship between file size and the quality of the output is not linear. Smaller models are less accurate, but the difference isn’t necessarily substantial. It’s no secret that even the most sophisticated models, up to and including GPT-4, sometimes come up with objectively wrong answers.

Once your model is downloaded, go to the chat tab and proceed to load it. You are now free to start chatting with the model to find out how it handles different directives. If the replies are very slow, complete gibberish, or fail to appear at all, there’s a good chance that the model is too heavy for your system.

Other than downloading and testing different models, there are plenty of optional settings to adjust. One of the more basic settings is ensuring that GPU acceleration is enabled. This will greatly improve performance as long as your system has enough resources. You can also inject a ‘System/Pre-Prompt’ to prime the model for your requests.

As for the advanced settings, I’m not nearly knowledgeable enough to comment on what to expect from altering ‘Inference Parameters’, but I fully intend to try them out. The good news is that they are thoroughly explained in the GUI.

Concluding Thoughts

LM Studio eliminates most of the challenges with setting up your local chatbot. While there are plenty of settings to dive into down the line, the hardest part of getting started is to find a model that suits your needs and works with your hardware. If you don’t know exactly what to look for, sorting the models by popularity and compatibility might be a good way to start.