Generic Multimodal Support (huggingface#1021)

* feat: multimodal anthropic support * docs: add claude haiku and multimodal support * feat: uploaded file detection and image conversion * fix deps with sharp * fix resvg deps? * fix: image conversion, retry with files * feat: generic image processing and size target * docs: multimodal review comments Co-authored-by: Mishig <mishig.davaadorj@coloradocollege.edu> * docs: multimodal review comments Co-authored-by: Mishig <mishig.davaadorj@coloradocollege.edu> * feat: review comment resolution * fix: type error on image params * feat: add multimodal for vertex ai anthropic * style: uploadFile timeout number Co-authored-by: Mishig <mishig.davaadorj@coloradocollege.edu> --------- Co-authored-by: Nathan Sarrazin <sarrazin.nathan@gmail.com> Co-authored-by: Mishig <mishig.davaadorj@coloradocollege.edu>
ice91 · May 15, 2024 · c6b428a · c6b428a
1 parent 33e8a3c
commit c6b428a
Show file tree

Hide file tree

Showing 25 changed files with 831 additions and 342 deletions.
diff --git a/README.md b/README.md
@@ -230,7 +230,7 @@ The following is the default `chatPromptTemplate`, although newlines and indenti
 
 #### Multi modal model
 
-We currently only support IDEFICS as a multimodal model, hosted on TGI. You can enable it by using the following config (if you have a PRO HF Api token):
+We currently support [IDEFICS](https://huggingface.co/blog/idefics) (hosted on TGI), OpenAI and Claude 3 as multimodal models. You can enable it by setting `multimodal: true` in your `MODELS` configuration. For IDEFICS, you must have a [PRO HF Api token](https://huggingface.co/settings/tokens). For OpenAI, see the [OpenAI section](#OpenAI). For Anthropic, see the [Anthropic section](#Anthropic).
 
 ```env
     {
@@ -465,14 +465,34 @@ MODELS=`[
 
 #### Anthropic
 
-We also support Anthropic models through the official SDK. You may provide your API key via the `ANTHROPIC_API_KEY` env variable, or alternatively, through the `endpoints.apiKey` as per the following example.
+We also support Anthropic models (including multimodal ones via `multmodal: true`) through the official SDK. You may provide your API key via the `ANTHROPIC_API_KEY` env variable, or alternatively, through the `endpoints.apiKey` as per the following example.
 
 ```
 MODELS=`[
+  {
+      "name": "claude-3-haiku-20240307",
+      "displayName": "Claude 3 Haiku",
+      "description": "Fastest and most compact model for near-instant responsiveness",
+      "multimodal": true,
+      "parameters": {
+        "max_new_tokens": 4096,
+      },
+      "endpoints": [
+        {
+          "type": "anthropic",
+          // optionals
+          "apiKey": "sk-ant-...",
+          "baseURL": "https://api.anthropic.com",
+          "defaultHeaders": {},
+          "defaultQuery": {}
+        }
+      ]
+  },
   {
       "name": "claude-3-sonnet-20240229",
       "displayName": "Claude 3 Sonnet",
       "description": "Ideal balance of intelligence and speed",
+      "multimodal": true,
       "parameters": {
         "max_new_tokens": 4096,
       },
@@ -491,6 +511,7 @@ MODELS=`[
       "name": "claude-3-opus-20240229",
       "displayName": "Claude 3 Opus",
       "description": "Most powerful model for highly complex tasks",
+      "multimodal": true,
       "parameters": {
          "max_new_tokens": 4096
       },
@@ -516,6 +537,7 @@ MODELS=`[
       "name": "claude-3-sonnet@20240229",
       "displayName": "Claude 3 Sonnet",
       "description": "Ideal balance of intelligence and speed",
+      "multimodal": true,
       "parameters": {
         "max_new_tokens": 4096,
       },
@@ -534,6 +556,7 @@ MODELS=`[
       "name": "claude-3-haiku@20240307",
       "displayName": "Claude 3 Haiku",
       "description": "Fastest, most compact model for near-instant responsiveness",
+      "multimodal": true,
       "parameters": {
          "max_new_tokens": 4096
       },