How can I retrieve the full OpenAI response (including top_logprobs) when calling Langchain's ChatOpenAI?

[EgorKraevTransferwise]

Checked other resources

• I added a very descriptive title to this question.
• I searched the LangChain documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.

Commit to Help

• I commit to help with one of those options 👆

Example Code

from openai import OpenAI

statement = "I feel good today"
message = """
Return the a digit describing the sentiment of the following message:
"{statement}"

Return the digit 1 if the sentiment is positive, 0 if the sentiment is neutral, and 2 if the sentiment is negative.
Return just the one character digit, nothing else
""".format(
    statement=statement
)

client = OpenAI()

m = client.models.list()

response = client.chat.completions.create(
    model="gpt-3.5-turbo-0125",
    messages=[
        {"role": "user", "content": message},
    ],
    logprobs=True,
    top_logprobs=5,
)
print(response.choices[0].message.content)
print(response.choices[0].logprobs.content[0].top_logprobs)

Description

The above snippet shows how I can retrieve the log probability information from OpenAI models using their raw client. How can I achieve the same within LCEL? I can use bind to set the logprobs and top_logprobs arguments to the ChatOpenAI model within the chain, but how do I retrieve the logprobs in the reply? Can I inject a callback somewhere to retrieve the raw answer of the OpenAI API?

System Info

System Information

OS: Windows
OS Version: 10.0.19044
Python Version: 3.11.5 | packaged by Anaconda, Inc. | (main, Sep 11 2023, 13:26:23) [MSC v.1916 64 bit (AMD64)]

Package Information

langchain_core: 0.1.23
langchain: 0.0.336
langsmith: 0.0.87

Packages not installed (Not Necessarily a Problem)

The following packages were not found:

langgraph
langserve

[dosubot[bot]]

🤖

Hey @EgorKraevTransferwise, great to see you back here! Diving into another interesting challenge, I see. Let's see what we can do about that.

Based on the context provided, it seems you're looking to retrieve the full OpenAI response, including top_logprobs, when using the ChatOpenAI model within the LangChain framework.

In LangChain, you can use the logprobs=True parameter when calling the generate or agenerate method. This will include the logprobs in the generation_info of the response. Here's an example:

from langchain_community.llms.openai import ChatOpenAI
from langchain_community.llms.openai import HumanMessage

llm = ChatOpenAI()
result = llm.generate([[HumanMessage(content="I'm PickleRick")]], logprobs=True)
assert result.generations[0][0].generation_info
assert "content" in result.generations[0][0].generation_info["logprobs"]

This code creates an instance of the ChatOpenAI model, generates a response with the logprobs=True parameter, and then checks that the generation_info of the response includes the logprobs.

The LangChain framework also provides a function to retrieve the full OpenAI response, including top_logprobs, when using the ChatOpenAI model. This is done through the _stream_response_to_generation_chunk function, which converts a stream response to a generation chunk. The function retrieves the logprobs from the response and includes it in the generation_info of the GenerationChunk object.

Here is the relevant code:

def _stream_response_to_generation_chunk(
    stream_response: Dict[str, Any],
) -> GenerationChunk:
    """Convert a stream response to a generation chunk."""
    if not stream_response["choices"]:
        return GenerationChunk(text="")
    return GenerationChunk(
        text=stream_response["choices"][0]["text"],
        generation_info=dict(
            finish_reason=stream_response["choices"][0].get("finish_reason", None),
            logprobs=stream_response["choices"][0].get("logprobs", None),
        ),
    )

In this function, stream_response["choices"][0].get("logprobs", None) is used to retrieve the logprobs from the response. If logprobs is not present in the response, None is returned.

Please note that the logprobs and top_logprobs are only available if you set logprobs=True when calling the generate or agenerate method. If you don't set logprobs=True, the logprobs and top_logprobs will not be included in the response.

I hope this helps! If you have any other questions, feel free to ask.

Sources

• libs/partners/openai/tests/integration_tests/chat_models/test_base.py
• libs/community/langchain_community/llms/openai.py

About Dosu

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[EgorKraevTransferwise]

@dosu-bot
Thanks a lot! Now can you please provide an example how to integerate this into a simple LCEL chain, first a PromtTemplate then an llm, and returning some object that contains both the reply and the top_logprobs info?

[EgorKraevTransferwise]

Thanks! the llm.generate call does the trick. To integrate it into LCEL, I guess I can wrap it into a RunnableLambda or something ;)

[dosubot[bot]]

🤖

To integrate the retrieval of the full OpenAI response, including top_logprobs, into a simple LCEL chain using the LangChain framework, you would need to modify the LLMChain class to return the full response from the OpenAI API.

Currently, the LLMChain class only returns the generated text. To include top_logprobs, you would need to modify the run method of the LLMChain class to return the full response.

Here is an example of how you could modify the run method:

class LLMChain(Chain):
    ...
    def run(self, **inputs: Any) -> Dict[str, Any]:
        ...
        response = self.llm.generate(prompt, **llm_kwargs)
        return {
            "generated_text": response["choices"][0]["text"]["content"],
            "top_logprobs": response["choices"][0]["logprobs"]["top_logprobs"],
        }

Then, in your ConversationalRetrievalChain, you would need to modify the _call method to include top_logprobs in the output:

class ConversationalRetrievalChain(Chain):
    ...
    def _call(self, inputs: Dict[str, Any], run_manager: Optional[CallbackManagerForChainRun] = None) -> Dict[str, Any]:
        ...
        new_question = self.question_generator.run(
            question=question, chat_history=chat_history_str, callbacks=callbacks
        )
        ...
        output[self.output_key] = answer
        output["top_logprobs"] = new_question["top_logprobs"]
        ...
        return output

Please note that this is a simplified example and you may need to adjust it to fit your specific use case. Also, keep in mind that returning top_logprobs will increase the amount of data returned by the API, which may impact performance and costs.

You can find more information about the LLMChain and ConversationalRetrievalChain classes in the LangChain repository.

Sources

• libs/langchain/langchain/chains/conversational_retrieval/base.py
• libs/core/langchain_core/language_models/llms.py

About Dosu

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[EgorKraevTransferwise]

@dosu-bot
Thanks, but I don't really want to modify the library classes if there's another way. Perhaps I could wrap the llm.generate call into a RunnableLambda instead?

[dosubot[bot]]

🤖

Based on your question, it seems you're looking to retrieve the full OpenAI response, including top_logprobs, using the ChatOpenAI model in the LangChain framework without modifying the library classes. You've also suggested the possibility of integrating this into a simple LCEL chain or wrapping the llm.generate call into a RunnableLambda.

You're on the right track with the idea of using a RunnableLambda. The LangChain framework does indeed support the use of custom RunnableLambda classes. RunnableLambda is designed to convert a Python callable into a Runnable, making it usable within either a synchronous or asynchronous context. This class can be composed as any other Runnable and provides seamless integration with LangChain tracing.

Here's an example of how you can create a custom RunnableLambda that wraps the llm.generate call and extracts the required information from the LLMResult object:

from langchain_core.language_models.llms import LLMResult
from langchain_core.runnables import RunnableLambda
from langchain_community.llms.openai import ChatOpenAI

# Initialize the ChatOpenAI model
llm = ChatOpenAI(model_name="gpt-3.5-turbo-instruct")

# Create a RunnableLambda that wraps the llm.generate call
class CustomRunnableLambda(RunnableLambda):
    async def run(self, input_: Any, **kwargs: Any) -> Any:
        # Call the generate method of the language model
        result: LLMResult = await llm.generate(input_, **kwargs)
        
        # Extract the full OpenAI response
        full_response = result.generation_info
        
        return full_response

# Initialize the custom RunnableLambda
runnable = CustomRunnableLambda()

# Use the RunnableLambda to generate a response
response = await runnable.run("Hello, how are you?")

In this example, the CustomRunnableLambda class overrides the run method to call the generate method of the ChatOpenAI model and extract the full OpenAI response from the LLMResult object. The run method of the RunnableLambda class is then used to generate a response, which includes the top_logprobs.

Please note that you need to replace "Hello, how are you?" with your actual input.

You can find more information about the RunnableLambda class in the LangChain core library.

I hope this helps! If you have any further questions, feel free to ask.

Sources

• libs/community/langchain_community/llms/openai.py
• libs/core/langchain_core/runnables/base.py

About Dosu

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[EgorKraevTransferwise]

@dosu-bot
Thanks a lot! Can you please also give a sync version?

[dosubot[bot]]

🤖

Sure, I can help you convert your asynchronous implementation to a synchronous one. Here's how you can do it:

First, you need to replace the async keyword from the run method definition in your CustomRunnableLambda class and remove the await keyword from the llm.generate call. The llm.generate method is asynchronous, but its synchronous equivalent is the invoke method.

Here's how your updated CustomRunnableLambda class would look:

from langchain_core.language_models.llms import LLMResult
from langchain_core.runnables import RunnableLambda
from langchain_community.llms.openai import ChatOpenAI

# Initialize the ChatOpenAI model
llm = ChatOpenAI(model_name="gpt-3.5-turbo-instruct")

# Create a RunnableLambda that wraps the llm.invoke call
class CustomRunnableLambda(RunnableLambda):
    def run(self, input_: Any, **kwargs: Any) -> Any:
        # Call the invoke method of the language model
        result: LLMResult = llm.invoke(input_, **kwargs)
        
        # Extract the full OpenAI response
        full_response = result.generation_info
        
        return full_response

# Initialize the custom RunnableLambda
runnable = CustomRunnableLambda()

# Use the RunnableLambda to generate a response
response = runnable.run("Hello, how are you?")

In this updated code, the run method of your CustomRunnableLambda class is now synchronous and uses the invoke method of the ChatOpenAI model to generate a response. The invoke method is the synchronous equivalent of the llm.generate method.

Please note that the invoke method returns the generated text directly, so you might need to adjust your code accordingly if you're expecting a different type of result.

I hope this helps! If you have any other questions, feel free to ask.

Sources

• libs/community/langchain_community/llms/openai.py
• libs/core/langchain_core/language_models/llms.py

About Dosu

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[gassis]

🤖

Sure, I can help you convert your asynchronous implementation to a synchronous one. Here's how you can do it:

First, you need to replace the async keyword from the run method definition in your CustomRunnableLambda class and remove the await keyword from the llm.generate call. The llm.generate method is asynchronous, but its synchronous equivalent is the invoke method.

Here's how your updated CustomRunnableLambda class would look:

from langchain_core.language_models.llms import LLMResult
from langchain_core.runnables import RunnableLambda
from langchain_community.llms.openai import ChatOpenAI

# Initialize the ChatOpenAI model
llm = ChatOpenAI(model_name="gpt-3.5-turbo-instruct")

# Create a RunnableLambda that wraps the llm.invoke call
class CustomRunnableLambda(RunnableLambda):
    def run(self, input_: Any, **kwargs: Any) -> Any:
        # Call the invoke method of the language model
        result: LLMResult = llm.invoke(input_, **kwargs)
        
        # Extract the full OpenAI response
        full_response = result.generation_info
        
        return full_response

# Initialize the custom RunnableLambda
runnable = CustomRunnableLambda()

# Use the RunnableLambda to generate a response
response = runnable.run("Hello, how are you?")

In this updated code, the run method of your CustomRunnableLambda class is now synchronous and uses the invoke method of the ChatOpenAI model to generate a response. The invoke method is the synchronous equivalent of the llm.generate method.

Please note that the invoke method returns the generated text directly, so you might need to adjust your code accordingly if you're expecting a different type of result.

I hope this helps! If you have any other questions, feel free to ask.

Sources

• libs/community/langchain_community/llms/openai.py
• libs/core/langchain_core/language_models/llms.py

About Dosu

What if I need to use the CustomRunnableLambda function with a load_qa_with_sources_chain chain?

[sofiabmorgado]

Or any other chains, for example just a qa_chain for a RAG?