2.2.4 Backend: TabbyAPI

TabbyAPI

Handle: tabbyapi URL: http://localhost:33931

An OAI compatible exllamav2 API that's both lightweight and fast

Models

• Supports same set of models as exllamav2

HuggingFaceDownloader

# Quickly lookup models
harbor hf find exl2

# Using HuggingFaceDownloader
harbor hf dl -m Annuvin/gemma-2-2b-it-abliterated-4.0bpw-exl2 -s ./hf
harbor hf dl -m bartowski/Phi-3.1-mini-4k-instruct-exl2 -s ./hf -b 8_0

# If pulling from the closed or gated repo
harbor hf token <your-token>

# You can then use model specifier
harbor tabbyapi model Annuvin/gemma-2-2b-it-abliterated-4.0bpw-exl2
harbor tabbyapi model bartowski/Phi-3.1-mini-4k-instruct-exl2

Native HF CLI

# Download with a model specifier
harbor hf download ChenMnZ/Mistral-Large-Instruct-2407-EfficientQAT-w2g64-GPTQ
# With a specific revision
harbor hf download turboderp/Llama-3.1-8B-Instruct-exl2 --revision 6.0bpw

# Grab actual name for the folder
harbor find ChenMnZ

# Set the model to run
harbor config set tabbyapi.model.specifier /hub/models--ChenMnZ--Mistral-Large-Instruct-2407-EfficientQAT-w2g64-GPTQ/snapshots/f46105941fa36d2663f77f11840c2f49a69d6681/

Starting

TabbyAPI exposes an OpenAI-compatible API and can be used with related services directly.

# [Optional] Pull the tabbyapi images
harbor pull tabbyapi

# Start the service
harbor up tabbyapi

# [Optional] Set additional arguments
harbor tabbyapi args --log-prompt true

# See TabbyAPI docs
harbor tabbyapi docs

Configuration

• Server options