Using LM Studio With BIGGI

BIGGI supports running models locally using LM Studio. LM Studio provides a user-friendly interface for downloading, configuring, and running local language models. It also includes a built-in local inference server that emulates the OpenAI API, making it easy to integrate with BIGGI.

Website: https://lmstudio.ai/

Setting Up LM Studio

  1. Download and Install LM Studio: Download LM Studio from the LM Studio website.

  2. Download a Model: Use the LM Studio interface to search for and download a model. Some recommended models include:

    • CodeLlama models (e.g., codellama:7b-code, codellama:13b-code, codellama:34b-code)
    • Mistral models (e.g., mistralai/Mistral-7B-Instruct-v0.1)
    • DeepSeek Coder models (e.g., deepseek-coder:6.7b-base)
    • Any other model that is supported by BIGGI, or for which you can set the context window.

    Look for models in the GGUF format. LM Studio provides a search interface to find and download models.

  3. Start the Local Server:

    • Open LM Studio.
    • Click the "Local Server" tab (the icon looks like <->).
    • Select the model you downloaded.
    • Click "Start Server".

Configuration in BIGGI

Open Settings (gear icon) and go to the Providers tab to add LM Studio. No API key is needed since LM Studio runs locally. You can configure the base URL if LM Studio is running on a different host or port.

The extension stores this in your biggi.json config file. You can also edit the file directly.

Using Custom or Unlisted Models

If the model you loaded in LM Studio doesn't appear in the BIGGI model picker, you can register it as a custom model in your config file:

{
  "model": "lmstudio/my-custom-model",
  "provider": {
    "lmstudio": {
      "models": {
        "my-custom-model": {
          "name": "My Custom Model",
        },
      },
    },
  },
}

The model key (my-custom-model) must match the model identifier that LM Studio serves. If the display name you want differs from the API identifier, use the id field to set the API-facing name separately:

{
  "provider": {
    "lmstudio": {
      "models": {
        "my-llama": {
          "id": "meta-llama-3.1-8b-instruct",
          "name": "Llama 3.1 8B (Local)",
        },
      },
    },
  },
}

See Custom Models for the full list of configuration fields and more examples.

Tips and Notes

  • Resource Requirements: Running large language models locally can be resource-intensive. Make sure your computer meets the minimum requirements for the model you choose.
  • Model Selection: LM Studio provides a wide range of models. Experiment to find the one that best suits your needs.
  • Local Server: The LM Studio local server must be running for BIGGI to connect to it.
  • LM Studio Documentation: Refer to the LM Studio documentation for more information.
  • Troubleshooting: If you see a "Please check the LM Studio developer logs to debug what went wrong" error, you may need to adjust the context length settings in LM Studio.