Multimodal Input with large files for Gemini

[g1henx]

Checked other resources

• I added a very descriptive title to this question.
• I searched the LangChain documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.

Commit to Help

• I commit to help with one of those options 👆

Example Code

export function messageToLangchain(message: MessageType, provider: string, files: FileType[]) {
    if(message.sender === 'ai'){
        return new AIMessage({
            content: 'text',
        });
    }
    let messageText = message.text;
    const content: any[] = [
        
      ];
      console.log('message files', message.files);
    if(message.files){
        for(const getFile of message.files){
            const file = files.find((f) => f.name === getFile.name);
            console.log('file', file);
            if(!file){
                continue;
            }
            if(provider === google){
                content.push({
                    type: "fileData",
                    mimeType: file!.mime_type,
                    fileUri: file!.data,
                });
            }
            if (provider === anthropic || provider === openai) {
                if(file.mime_type && file.mime_type.startsWith('image')){
                    content.push({
                        type: "image_url",
                        image_url: {
                            url: `data:${file.mime_type};base64,${file.data}`,
                        },
                    });
                }else{
                    messageText += ` 
                    
                    Trasncripciones de los archivos adjuntos:
                    -------------- ${file.name} --------------
                    ${file.data}
                    -----------------------------------------
                    `;
                }
            }

        }
    }
    content.push({
        type: "text",
        text: messageText,
    });
    if (message.sender === 'human') {
        
        return new HumanMessage({
            content: content,
        });
    }
    return new SystemMessage({
        content: content,
    });
}

Description

How am i supposed to input google genai files to the message?

System Info

langchain@0.2.11 | MIT | deps: 16 | versions: 283
Typescript bindings for langchain
https://github.com/langchain-ai/langchainjs/tree/main/langchain/

keywords: llm, ai, gpt3, chain, prompt, prompt engineering, chatgpt, machine learning, ml, openai, embeddings, vectorstores

dist
.tarball: https://registry.npmjs.org/langchain/-/langchain-0.2.11.tgz
.shasum: d97e5bbd57e8954f21356fd85603aa39e3efe03f
.integrity: sha512-6FQWKNAXuTmwuhHHMOmurLo8pydSRu5C/FwCYvYbR4ulCLqcsj+jre/kfXvA5BdHOZHNo6oQn0/5kxDNnhxMUA==
.unpackedSize: 5.4 MB

dependencies:
@langchain/core: >=0.2.11 <0.3.0 jsonpointer: ^5.0.1 uuid: ^10.0.0
@langchain/openai: >=0.1.0 <0.3.0 langchainhub: ~0.0.8 yaml: ^2.2.1
@langchain/textsplitters: ~0.0.0 langsmith: ~0.1.30 zod-to-json-schema: ^3.22.3
binary-extensions: ^2.2.0 ml-distance: ^4.0.0 zod: ^3.22.4
js-tiktoken: ^1.0.12 openapi-types: ^12.1.3
js-yaml: ^4.1.0 p-retry: 4

maintainers:

• nfcampos nuno@boringbits.io
• jacoblee93 jacoblee93@gmail.com
• andrewnguonly andrewnguonly@gmail.com
• davidduong david@duong.cz
• hwchase17 hw.chase.17@gmail.com
• basproul braceasproul@gmail.com

dist-tags:
latest: 0.2.11 next: 0.2.3-rc.0

published 2 days ago by basproul braceasproul@gmail.com
Node.js v18.17.1

[dosubot[bot]]

Hello, @g1henx! I'm here to assist you with any issues or questions you may have. I can help you solve bugs, answer questions, and even guide you on how to become a contributor. Let's tackle this together!

To handle Google GenAI files in your messageToLangchain function, you need to ensure that the files are correctly processed and included in the message content. Here's how you can modify your function to handle Google GenAI files:

1. Convert files to base64 strings: Ensure that the files are read and converted to base64 strings.
2. Include the files in the message content: Add the files to the message content with the appropriate MIME type.

Here's the updated code:

import { HumanMessage, AIMessage, SystemMessage } from "@langchain/core/messages";
import fs from "fs";
import path from "path";

export function messageToLangchain(message: MessageType, provider: string, files: FileType[]) {
    if (message.sender === 'ai') {
        return new AIMessage({
            content: 'text',
        });
    }

    let messageText = message.text;
    const content: any[] = [];

    if (message.files) {
        for (const getFile of message.files) {
            const file = files.find((f) => f.name === getFile.name);
            if (!file) {
                continue;
            }

            // Convert file to base64
            const fileData = fs.readFileSync(path.join(__dirname, file.data)).toString("base64");

            if (provider === 'google') {
                content.push({
                    type: "fileData",
                    mimeType: file.mime_type,
                    fileUri: `data:${file.mime_type};base64,${fileData}`,
                });
            } else if (provider === 'anthropic' || provider === 'openai') {
                if (file.mime_type && file.mime_type.startsWith('image')) {
                    content.push({
                        type: "image_url",
                        image_url: `data:${file.mime_type};base64,${fileData}`,
                    });
                } else {
                    messageText += ` 
                    
                    Transcriptions of the attached files:
                    -------------- ${file.name} --------------
                    ${fileData}
                    -----------------------------------------
                    `;
                }
            }
        }
    }

    content.push({
        type: "text",
        text: messageText,
    });

    if (message.sender === 'human') {
        return new HumanMessage({
            content: content,
        });
    }

    return new SystemMessage({
        content: content,
    });
}

Key Changes:

1. File Conversion: The files are read from the file system and converted to base64 strings.
2. Content Construction: The files are included in the message content with the appropriate MIME type.

This approach ensures that the files are correctly processed and included in the message content for Google GenAI. For more details on handling multimodal inputs, refer to the LangChain.js documentation ^[1].

To continue talking to Dosu, mention @dosu.

[g1henx]

That is not the solution, you can't send files of more than 20mb this way

[DevDeepakBhattarai]

@g1henx Did you find any solution for this ?