Create Chain using LCEL using with_structured_output and RetryOutputParser as fallback

[pleelapr]

Checked other resources

• I added a very descriptive title to this question.
• I searched the LangChain documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.

Commit to Help

• I commit to help with one of those options 👆

Example Code

from langchain.pydantic_v1 import BaseModel, Field


class Triplet(BaseModel):
    subject: str
    relation: str
    object: str


class TripletExtraction(BaseModel):
    extraction_list: List[Triplet] = Field(description="List of triplets extracted from the conversation")

def _initialize_chain(self):
        parser = PydanticOutputParser(pydantic_object=TripletExtraction)
        retry_parser = RetryOutputParser.from_llm(
            parser=parser, llm=self._model.with_structured_output(TripletExtraction), max_retries=3
        )
        # Initialize the chain with the prompt and the model
        chain = self._prompt | self._model.with_structured_output(TripletExtraction, include_raw=True) | retry_parser

        # Add config to the chain and return the chain
        return chain.with_config(RunnableConfig(tags=[self._agent_prefix], metadata={"agent": self._agent_prefix}))

Description

I am trying to create a chain using LCEL for OpenAI using with_structured_output on pydantic BaseModel.
However, there are some cases where the LLM generated the wrong structure. Hence, I would like to use the RetryOutputParser. However, the graph seems to stuck, and I don't know which part of the code I need to change.

System Info

langchain==0.2.5
langchain-chroma==0.1.1
langchain-community==0.2.5
langchain-core==0.2.9
langchain-experimental==0.0.61
langchain-google-vertexai==1.0.5
langchain-openai==0.1.8
langchain-postgres==0.0.9
langchain-text-splitters==0.2.1
langchainhub==0.1.14

platform (mac)
python version 3.11.5

[dosubot[bot]]

Hey @pleelapr! I'm here to help you with your LangChain issue. Let's get this sorted out together!

To ensure your chain works correctly with the RetryOutputParser when using LCEL for OpenAI with with_structured_output on a Pydantic BaseModel, you need to modify your code to properly integrate the RetryOutputParser. Here's how you can do it:

1. Import the necessary modules:
   Ensure you have the correct imports for the RetryOutputParser and other required components.

2. Modify the _initialize_chain method:
   Update the method to use the RetryOutputParser correctly.

Here is the updated code:

from langchain.pydantic_v1 import BaseModel, Field
from langchain_core.output_parsers.retry import RetryOutputParser
from langchain_core.output_parsers.pydantic import PydanticOutputParser
from langchain_core.chains import RunnableConfig

class Triplet(BaseModel):
    subject: str
    relation: str
    object: str

class TripletExtraction(BaseModel):
    extraction_list: List[Triplet] = Field(description="List of triplets extracted from the conversation")

def _initialize_chain(self):
    parser = PydanticOutputParser(pydantic_object=TripletExtraction)
    retry_parser = RetryOutputParser.from_llm(
        parser=parser, llm=self._model.with_structured_output(TripletExtraction), max_retries=3
    )
    # Initialize the chain with the prompt and the model
    chain = self._prompt | self._model.with_structured_output(TripletExtraction, include_raw=True) | retry_parser

    # Add config to the chain and return the chain
    return chain.with_config(RunnableConfig(tags=[self._agent_prefix], metadata={"agent": self._agent_prefix}))

Explanation:

1. Parser Initialization:

   • PydanticOutputParser is initialized with the TripletExtraction Pydantic model.
   • RetryOutputParser is created using the from_llm method, which takes the PydanticOutputParser, the language model with structured output, and the maximum number of retries.

2. Chain Initialization:

   • The chain is created by combining the prompt, the language model with structured output, and the RetryOutputParser.
   • Configuration is added to the chain using with_config.

This setup ensures that the RetryOutputParser is used as a fallback to handle cases where the LLM generates the wrong structure, retrying up to three times as specified.

For more details on handling parsing errors and retry mechanisms, you can refer to the LangChain documentation on RetryOutputParser ^[1][2][3][4].

To continue talking to Dosu, mention @dosu.

[pleelapr]

I would like to try using with_fallbacks() instead of LCEL styling? @dosu

[dosubot[bot]]

To modify your code to use with_fallbacks() instead of the current LCEL styling, follow these steps:

1. Install the necessary packages:

   %pip install --upgrade --quiet langchain langchain-openai

2. Import the required modules:

   from langchain_community.chat_models import ChatAnthropic
   from langchain_openai import ChatOpenAI
   from langchain_core.output_parsers.pydantic import PydanticOutputParser
   from langchain_core.output_parsers.retry import RetryOutputParser
   from langchain_core.chains import RunnableConfig
   from langchain_core.prompts import ChatPromptTemplate

3. Define your Pydantic models:

   from langchain.pydantic_v1 import BaseModel, Field
   from typing import List
   
   class Triplet(BaseModel):
       subject: str
       relation: str
       object: str
   
   class TripletExtraction(BaseModel):
       extraction_list: List[Triplet] = Field(description="List of triplets extracted from the conversation")

4. Set up the LLMs and configure fallbacks:

   openai_llm = ChatOpenAI(max_retries=0)
   anthropic_llm = ChatAnthropic()
   llm = openai_llm.with_fallbacks([anthropic_llm])

5. Initialize the chain with the prompt, model, and parser:

   def _initialize_chain(self):
       parser = PydanticOutputParser(pydantic_object=TripletExtraction)
       retry_parser = RetryOutputParser.from_llm(
           parser=parser, llm=llm.with_structured_output(TripletExtraction), max_retries=3
       )
       
       prompt = ChatPromptTemplate.from_messages(
           [
               ("system", "Answer the user query. Wrap the output in `json` tags\n{format_instructions}"),
               ("human", "{query}"),
           ]
       ).partial(format_instructions=parser.get_format_instructions())
   
       chain = prompt | llm.with_structured_output(TripletExtraction, include_raw=True) | retry_parser
   
       return chain.with_config(RunnableConfig(tags=[self._agent_prefix], metadata={"agent": self._agent_prefix}))

This setup ensures that the RetryOutputParser is used as a fallback to handle cases where the LLM generates the wrong structure, retrying up to three times as specified. The with_fallbacks() method allows you to switch to an alternative LLM (Anthropic) if the primary LLM (OpenAI) fails.

For more details on using with_fallbacks(), refer to the LangChain documentation ^[1].

To continue talking to Dosu, mention @dosu.