Skip to content

g1ibby/llm-deploy

Repository files navigation

LLM deploy - tool to manage your LLMs on vast.ai servers

Introduction

"llm-deploy" is a Python tool for deploying and managing large language models (LLMs) on vast.ai using ollama. It uses Typer for command-line interactions.

Requirements

  • Python 3.11 or later
  • Poetry for dependency management

Installation

  1. Clone the repository or download the source code.
  2. Navigate to the project directory.
  3. Run poetry install to install dependencies.

Configuration

Create a llms.yaml file with your model configurations, like this:

models:
  llama:
    model: "phi:2.7b-chat-v2-q5_K_M"
    priority: low

Copy file env.sh.dist to env.sh and set your keys there.

Run source env.sh

Usage

Config-Mode Commands:

  • Apply LLMs Configuration: poetry run llm-deploy apply Applies configurations from llms.yaml.

  • Destroy LLMs Configuration: poetry run llm-deploy destroy Reverts configurations and destroys created instances based on the current state.

Manual-Mode Commands:

  • List Current Instances: poetry run llm-deploy infra ls Lists all current instances.

  • Create New Instance (Manual): poetry run llm-deploy infra create --gpu-memory <memory_in_GB> --disk <disk_space_in_GB> Manually creates a new instance with specified GPU memory, disk space, and public IP option.

  • Remove an Instance: poetry run llm-deploy infra destroy <instance_id> Removes an instance by ID.

  • Show Instance Details: poetry run llm-deploy infra inspect <instance_id> Shows details of an instance.

  • Retrieve Logs for an Instance: poetry run llm-deploy logs <instance_id> --max-logs <number> Retrieves and displays logs for a specified instance.

  • Deploy a Model to an Instance: poetry run llm-deploy model deploy <model_name> <instance_id> Deploys a specified model to an instance.

  • Remove a Model from an Instance: poetry run llm-deploy model remove <model_name> <instance_id> Removes a deployed model from an instance.

  • List Models on Instances: poetry run llm-deploy model ls Lists models deployed across instances.

About

Tool to manage ollama model on vast.ai

Topics

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published