add auto vector support, convert project to python package, add tests (…

…#9) * resolve merge conflicts, huge refactor * update readme * add license apgl 3.0 * update readme * add TODO's * remove fastapi app * move folders into quickllm dir * add setup.py, init and change fname to auto_vector_store * minofr fix * change autovs method name to from_vector_store_type * add setup.cfg * add TODO's to readme * add auto trial module * major refactor * minor fix * refactor style with pre-commit * uddate docs * major refactor * update docstrings * add cost_calculation module * some fixes * add error handling and llm name on top of model cost dict * minor fix * improve query engine and cost calculation * update service context * minor fix * remove unnecessary arg * update readme * simplify auto classes * support all bedrock openai palm etc models via litellm * update autollm usage in readme * improve query engine * fix readme * bugfix and update docstrings on cost_calculation module * minor fix * update readme for cost calculater * add tests * add test to autoServiceContext * filename fix * add test auto_llm * major update * bugfix and add test to auto vector store * minor fix * big refactor and add test * update readme * resolve todos * removed accidental pushed file * fix tests * add missing file * ignore vscode settings * update readme * minor fix * minor updates * update readme * update readme * fix a comment * slight update to template * add pypi publish action * add previous version of app * redesign readme * udpate action workflows * update precommit * remove incorrect load_dotenv line * refactor llm_utils * some fixes * fix typo * add an important todo * fix ci * minor update * improve roadmap --------- Co-authored-by: fcakyon <fcakyon@gmail.com>
viddexa · Oct 16, 2023 · b206025 · b206025
1 parent b5fb999
commit b206025
Show file tree

Hide file tree

Showing 52 changed files with 2,172 additions and 650 deletions.
diff --git a/.github/workflows/ci.yaml b/.github/workflows/ci.yaml
@@ -0,0 +1,72 @@
+name: Continuous Integration
+on:
+  push:
+    branches: [main]
+    paths-ignore:
+      - '**.md'
+      - '**.ipynb'
+
+  pull_request:
+    branches: [main]
+    paths-ignore:
+      - '**.md'
+      - '**.ipynb'
+
+jobs:
+  build:
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        operating-system: [ubuntu-latest, windows-latest, macos-latest]
+        python-version: [3.8, 3.9, '3.10', '3.11']
+      fail-fast: false
+
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v3
+
+      - name: Set up Python
+        uses: actions/setup-python@v4
+        with:
+          python-version: ${{ matrix.python-version }}
+
+      - name: Restore Ubuntu cache
+        uses: actions/cache@v3
+        if: matrix.operating-system == 'ubuntu-latest'
+        with:
+          path: ~/.cache/pip
+          key: ${{ matrix.os }}-${{ matrix.python-version }}-${{ hashFiles('**/setup.py')}}
+          restore-keys: ${{ matrix.os }}-${{ matrix.python-version }}-
+
+      - name: Restore MacOS cache
+        uses: actions/cache@v3
+        if: matrix.operating-system == 'macos-latest'
+        with:
+          path: ~/Library/Caches/pip
+          key: ${{ matrix.os }}-${{ matrix.python-version }}-${{ hashFiles('**/setup.py')}}
+          restore-keys: ${{ matrix.os }}-${{ matrix.python-version }}-
+
+      - name: Restore Windows cache
+        uses: actions/cache@v3
+        if: matrix.operating-system == 'windows-latest'
+        with:
+          path: ~\AppData\Local\pip\Cache
+          key: ${{ matrix.os }}-${{ matrix.python-version }}-${{ hashFiles('**/setup.py')}}
+          restore-keys: ${{ matrix.os }}-${{ matrix.python-version }}-
+
+      - name: Update pip
+        run: python -m pip install --upgrade pip
+
+      - name: Install local package with dev dependencies
+        run: >
+          pip install -e .[dev]
+
+      - name: Check styling with pre-commit
+        run: |
+          pre-commit install
+          pre-commit run --all-files
+
+      - name: Run tests
+        env:
+          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
+        run: pytest
diff --git a/.github/workflows/pypi_publish.yaml b/.github/workflows/pypi_publish.yaml
@@ -0,0 +1,27 @@
+name: Publish Python Package
+
+on:
+  release:
+    types: [created]
+
+jobs:
+  deploy:
+    runs-on: ubuntu-20.04
+
+    steps:
+    - uses: actions/checkout@v3
+    - name: Set up Python
+      uses: actions/setup-python@v4
+      with:
+        python-version: '3.x'
+    - name: Install dependencies
+      run: |
+        python -m pip install --upgrade pip
+        pip install setuptools wheel twine
+    - name: Build and publish
+      env:
+        TWINE_USERNAME: __token__
+        TWINE_PASSWORD: ${{ secrets.PYPI_TOKEN }}
+      run: |
+        python setup.py sdist bdist_wheel
+        twine upload dist/*
diff --git a/.gitignore b/.gitignore
@@ -166,4 +166,6 @@ testing.py
 # Hash storage file
 file_hashes.txt
 # For vector index storage
-storage/
+storage/
+# vscode settings
+.vscode
diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
@@ -0,0 +1,70 @@
+# Define bot property if installed via https://github.com/marketplace/pre-commit-ci
+ci:
+  autofix_prs: true
+  autoupdate_commit_msg: '[pre-commit.ci] pre-commit suggestions'
+  autoupdate_schedule: monthly
+  # submodules: true
+
+repos:
+  - repo: https://github.com/pre-commit/pre-commit-hooks
+    rev: v4.5.0
+    hooks:
+      - id: end-of-file-fixer
+      - id: trailing-whitespace
+      - id: check-case-conflict
+      - id: no-commit-to-branch
+      # - id: check-yaml
+      - id: detect-private-key
+      - id: detect-aws-credentials
+        args:
+          - --allow-missing-credentials
+
+  - repo: https://github.com/asottile/pyupgrade
+    rev: v3.15.0
+    hooks:
+      - id: pyupgrade
+        name: Upgrade code
+
+  - repo: https://github.com/PyCQA/isort
+    rev: 5.12.0
+    hooks:
+      - id: isort
+        name: Sort imports
+
+  - repo: https://github.com/google/yapf
+    rev: v0.40.2
+    hooks:
+      - id: yapf
+        name: YAPF formatting
+        args:
+          - -i
+
+  - repo: https://github.com/executablebooks/mdformat
+    rev: 0.7.17
+    hooks:
+      - id: mdformat
+        name: MD formatting
+        additional_dependencies:
+          - mdformat-gfm
+          - mdformat-black
+        exclude: 'docs/.*\.md'
+
+  - repo: https://github.com/PyCQA/flake8
+    rev: 6.1.0
+    hooks:
+      - id: flake8
+        name: Flake8 linting
+
+  - repo: https://github.com/codespell-project/codespell
+    rev: v2.2.6
+    hooks:
+      - id: codespell
+        args:
+          - --ignore-words-list=crate,nd,strack,dota,ane,segway,fo,gool,winn
+          # skip inline comments
+          - --skip="*.py:.*#.*"
+
+  - repo: https://github.com/PyCQA/docformatter
+    rev: v1.7.5
+    hooks:
+      - id: docformatter
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
@@ -0,0 +1,44 @@
+### PR recommendations
+
+To allow your work to be integrated as seamlessly as possible, we advise you to:
+
+- **Create a new branch** for your PR. This will allow you to keep your changes separate from the `main` branch and
+  facilitate the review process.
+
+- **Keep your PR small**. If you want to contribute a new feature, consider splitting it into multiple PRs. This will
+  allow us to review your work more easily and provide you with feedback faster.
+
+- Verify your PR is **up-to-date** with `safevideo/autollm` `main` branch. If your PR is behind you can update
+  your code by clicking the 'Update branch' button or by running `git pull` and `git merge main` locally.
+
+### Docstrings
+
+Not all functions or classes require docstrings but when they do, we
+follow [google-style docstrings format](https://google.github.io/styleguide/pyguide.html#38-comments-and-docstrings).
+Here is an example:
+
+```python
+"""
+    What the function does. Performs NMS on given detection predictions.
+
+    Args:
+        arg1: The description of the 1st argument
+        arg2: The description of the 2nd argument
+
+    Returns:
+        What the function returns. Empty if nothing is returned.
+
+    Raises:
+        Exception Class: When and why this exception can be raised by the function.
+"""
+```
+
+### Code style
+
+We use pre-commit hooks to ensure that all code is formatted according to
+[flake8](https://flake8.pycqa.org/en/latest/) and
+[isort](https://pycqa.github.io/isort/). To install the pre-commit hooks, run:
+
+```bash
+pre-commit install
+```