Detect incorrectly formatted LLM output in the MRKL chat and remind LLM to correct itself?

[mlejva]

Hey there, I've been working and exploring the Python library for the past few weeks. I'm using gpt-3.5 model and I created an agent that follows the MRKL framework.

agent = ChatAgent.from_llm_and_tools(
    llm=chat,
    tools=tools,
    prefix=PREFIX,
    suffix=SUFFIX,
    format_instructions=FORMAT_INSTRUCTIONS,
    input_variables=input_variables,
    verbose=True,
)
agent_executor = AgentExecutor.from_agent_and_tools(agent=agent, tools=tools)

All works well when I run my agent

agent_executor.run(...)

Sometimes though, the LLM responds with badly formatted output which causes the "Could not parse LLM output" exception.

What I'm trying to achieve is to catch this exception and respond to the LLM with something along the lines like

I just ran your response via json.loads and received this error:
{error}
Please try again

The problem is, I'm struggling to find a way to alter the messages that are being sent to the LLM when this exception is caught.

Here's the code that I'd like to have. Do you have any tips?

try:
    # Run
    agent_executore.run(...)
except ValueError as e:
    if "Could not parse LLM output" in str(e):
        # How to send the following message and keep going with the run?
        # "I just ran your response via json.loads and received this error:
        # {error}
        # Please try again"
        print("FAILED TO PARSE LLM")

Do I need to implement a completely custom agent that overrides some methods to achieve this?