How does GraphCypherQAChain.from_llm construct answers based on a cypher query's results?

[AsmaaMHadir]

I am building a RAG chain over my Neo4j graph database using the GraphCypherQAChain as defined from the docs. The code runs as a chainlit app in which the user asks questions about the data in my graph and the chatbot queries the graph and uses the returned result of the query to answer the user's question. My issue is that for some questions, the chatbot constructs the correct queries and returns the correct response, and for many others the chatbot doesn't return an answer even though the cypher query is correct and the results of the query contain information to answer the user's question, in which case the chatbot just says the knowledge base is empty. My questions are:

• What could be the reason for the LLM not being able to construct an answer, although the results of the query it constructs based on the user's questions are not empty and contain information that are immediate answers to the user's question? is there something about the code that I have not implemented correctly? if so, what should I fix?
• Under the hood, how does the GraphCypherQAChain pass context to the Q&A prompt?

Here's a breakdown of my code:

1. I construct examples of cypher queries for my cypher llm to use when constructing a query based on a user's question:

`query_examples = data

graph = Neo4jGraph(
    url=NEO4J_URL, username=NEO4J_USERNAME, password=NEO4J_PASSWORD
    )


EXAMPLES_PROMPT_TEMPLATE = """   
                Input: {db_question},
                Output: {db_query}
            """
                         

example_prompt = PromptTemplate(input_variables=["db_question", "db_query"], template=EXAMPLES_PROMPT_TEMPLATE)

example_selector = MaxMarginalRelevanceExampleSelector.from_examples(
    # This is the list of examples available to select from.
    query_examples,
    # This is the embedding class used to produce embeddings which are used to measure semantic similarity.
    OpenAIEmbeddings(),
    # This is the VectorStore class that is used to store the embeddings and do a similarity search over.
    Chroma,
    # This is the number of examples to produce.
    k=2
)`

2. I construct my cypher query construction prompt:
   ` prompt_cypher = FewShotPromptTemplate(
   example_selector=example_selector,
   example_prompt=example_prompt,
   suffix="The question is:\n{question}",
   prefix="""Task: Generate a Cypher query to query a graph database based on the user's question.
   Instructions:
   Use the provided schema for node types, relationship types, and properties in the graph database. Only incorporate these defined elements.
   Avoid utilizing any other node types, relationship types, or properties not present in the provided schema. Here's the schema definition:

{schema}

if the question matches one of the sample questions in the knowledge base then just return the query used to answer it.
if the user asks to retrieve a piece of information about a document or section given their name, then use a WHERE statement
and a cypher regular expression matching without case sensitivity like in the queries in your knowledge base when filtering by the name.

whenever you are looking for a section node, always use the statement (t:transaction)-[:CONTAINS*]->(s) in the cypher query
with the * sign next to the relationship label CONTAINS and where s is the section node you are looking for.

Ensure the generated query captures relevant information from the graph database without reducing the retrieved data due to variations in user wording.
Note: do not include any explanations or apologies in your responses. Return just the cypher query.
Do not respond to inquiries seeking information other than the construction of a Cypher statement.
""",
input_variables=["schema", "question"]
)
`

3. Define my Q&A prompt:
   `
   QA_GENERATION_TEMPLATE = """
   Task: answer the question you are given based on the context provided.
   Instructions:
   You are an assistant that helps to form nice and human understandable answers.
   Use the context information provided to generate a well organized and comprehensve answer to the user's question.
   When the provided information contains multiple elements, structure your answer as a bulleted or numbered list to enhance clarity and readability.
   You must use the information to construct your answer.
   The provided information is authoritative; do not doubt it or try to use your internal knowledge to correct it.
   Make the answer sound like a response to the question without mentioning that you based the result on the given information.
   If there is no information provided, say that the knowledge base returned empty results.

        Here's the information:
        {context}
   
        Question: {question}
        Answer:
            """
   

   prompt_qa = PromptTemplate(input_variables=["context", "question"], template=QA_GENERATION_TEMPLATE)`

4. Then finally, I call the GraphCypherQAChain:
   chain = GraphCypherQAChain.from_llm( cypher_llm=ChatOpenAI(temperature=0, model="gpt-4"), qa_llm=ChatOpenAI(temperature=0, model="gpt-4"), graph=graph, verbose=True, cypher_prompt=prompt_cypher, qa_prompt=prompt_qa, return_intermediate_steps=True, validate_cypher=True )

these steps alongside the Graph Q&A chain are defined as part of a chainlit app as below:
`
class DocumentInput(BaseModel):
question: str = Field()

@cl.on_chat_start
async def on_chat_start():

file_path = "examples.json"
OPENAI_API_KEY = os.getenv("OPENAI_API_KEY")
NEO4J_URL = os.getenv("NEO4J_URL")
NEO4J_USERNAME = os.getenv("NEO4J_USERNAME")
NEO4J_PASSWORD = os.getenv("NEO4J_PASSWORD")

# Open the file and parse the JSON
with open(file_path, 'r', encoding='utf-8') as file:
    data = json.load(file)

query_examples = data


graph = Neo4jGraph(
    url=NEO4J_URL, username=NEO4J_USERNAME, password=NEO4J_PASSWORD
    )


EXAMPLES_PROMPT_TEMPLATE = """   
                Input: {db_question},
                Output: {db_query}
            """
                         

example_prompt = PromptTemplate(input_variables=["db_question", "db_query"], template=EXAMPLES_PROMPT_TEMPLATE)

example_selector = MaxMarginalRelevanceExampleSelector.from_examples(
    # This is the list of examples available to select from.
    query_examples,
    # This is the embedding class used to produce embeddings which are used to measure semantic similarity.
    OpenAIEmbeddings(),
    # This is the VectorStore class that is used to store the embeddings and do a similarity search over.
    Chroma,
    # This is the number of examples to produce.
    k=2
)

prompt_cypher = FewShotPromptTemplate(
    example_selector=example_selector,
    example_prompt=example_prompt,
    suffix="The question is:\n{question}",
    prefix="""Task: Generate a Cypher query to query a graph database based on the user's question.

Instructions:
Use the provided schema for node types, relationship types, and properties in the graph database. Only incorporate these defined elements.
Avoid utilizing any other node types, relationship types, or properties not present in the provided schema. Here's the schema definition:

{schema}

if the question matches one of the sample questions in the knowledge base then just return the query used to answer it.
if the user asks to retrieve a piece of information about a document or section given their name, then use a WHERE statement
and a cypher regular expression matching without case sensitivity like in the queries in your knowledge base when filtering by the name.

whenever you are looking for a section node, always use the statement (t:transaction)-[:CONTAINS*]->(s) in the cypher query
with the * sign next to the relationship label CONTAINS and where s is the section node you are looking for.

Ensure the generated query captures relevant information from the graph database without reducing the retrieved data due to variations in user wording.
Note: do not include any explanations or apologies in your responses. Return just the cypher query.
Do not respond to inquiries seeking information other than the construction of a Cypher statement.
""",
input_variables=["schema", "question"]
)

QA_GENERATION_TEMPLATE = """
       Task: answer the question you are given based on the context provided.
       Instructions:
        You are an assistant that helps to form nice and human understandable answers. 
        Use the context information provided to generate a well organized and comprehensve answer to the user's question. 
        When the provided information contains multiple elements, structure your answer as a bulleted or numbered list to enhance clarity and readability.
        You must use the information to construct your answer. 
        The provided information is authoritative; do not doubt it or try to use your internal knowledge to correct it. 
        Make the answer sound like a response to the question without mentioning that you based the result on the given information. 
        If there is no information provided, say that the knowledge base returned empty results.

        Here's the information:
        {context}

        Question: {question}
        Answer:
            """
prompt_qa = PromptTemplate(input_variables=["context", "question"], template=QA_GENERATION_TEMPLATE)

chain = GraphCypherQAChain.from_llm(
    cypher_llm=ChatOpenAI(temperature=0, model="gpt-4"),
    qa_llm=ChatOpenAI(temperature=0, model="gpt-4"),
    graph=graph,
    verbose=True,
    cypher_prompt=prompt_cypher,
    qa_prompt=prompt_qa,
    return_intermediate_steps=True,
    validate_cypher=True
)

cl.user_session.set("chain", chain)

@cl.on_message
async def main(message: cl.Message):
#agent = cl.user_session.get("agent") # type: Agent
chain = cl.user_session.get("chain") # type: Agent

cb = cl.AsyncLangchainCallbackHandler()

#res = await agent.acall(message.content, callbacks=[cb])
res = await chain.acall(message.content, callbacks=[cb])

#print(res)
#answer = res["answer"]
#answer = res["output"]
answer = res['result']

#print(len(answer))
if len(answer) == 2:
    answer = answer['result']

text_elements = []  # type: List[cl.Text]

await cl.Message(content=answer, elements=text_elements).send()`

I would appreciate any leads on how to handle this. Thanks!

[dosubot[bot]]

🤖

Hello @AsmaaMHadir! Welcome to the LangChain repository. I'm Dosu, a friendly bot here to assist you while we wait for a human maintainer. I'm designed to help with troubleshooting, answer your queries, and guide you on your contribution journey. Let's work together to resolve this issue you're encountering!

Based on the code and the description you provided, it seems like you have correctly implemented the GraphCypherQAChain.from_llm method. However, there are a few things that might be causing the issue you're experiencing.

Firstly, the GraphCypherQAChain.from_llm method uses the qa_chain to generate an answer based on the question and the retrieved results. If the qa_chain fails to generate an answer, the method will not construct an answer. This could be due to the way the qa_chain is set up or due to the specific question and results. You might want to check the setup of your qa_chain and try different questions and results to see if this is the issue.

Secondly, the GraphCypherQAChain.from_llm method might not construct an answer even when the query results contain relevant information if the cypher_query_corrector is enabled and it identifies the generated Cypher query as invalid. In this case, the method will not retrieve any results from the graph database, and hence, it will not construct an answer. You might want to check if your cypher_query_corrector is working correctly.

Lastly, if return_direct is set to True, the method will return the results directly without using the qa_chain to generate an answer. You might want to check if this is the case in your setup.

As for your second question, the GraphCypherQAChain passes context to the Q&A prompt in the _call method. It retrieves the results from the graph database using the generated Cypher query and passes these results as the context to the qa_chain. The qa_chain then uses this context along with the question to generate an answer.

Here's the relevant code snippet:

if generated_cypher:
    context = self.graph.query(generated_cypher)[: self.top_k]
else:
    context = []

if self.return_direct:
    final_result = context
else:
    result = self.qa_chain(
        {"question": question, "context": context},
        callbacks=callbacks,
    )
    final_result = result[self.qa_chain.output_key]

I hope this helps! If you have any more questions or if something is not clear, please let me know.

Sources

• libs/langchain/langchain/chains/graph_qa/cypher.py

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[AsmaaMHadir]

Thank you for the explanation. Could you please show me how I could define the qa_chain for my GraphCypherQAChain? Based on the what I have seen in the GraphCypherQAChain class, qa_chain is passed as an attribute of type LLMChain. In my current code I am not passing a qa_chain but instead I pass a qa_prompt and qa_llm for the final answer construction. I also do not have the validate_cypher parameter passed either. The intermediate results for a sample question answer look like this:
{ "result": "The knowledge base returned empty results.", "intermediate_steps": [ { "query": "MATCH (t:transaction)-[:CONTAINS]->(s:section)\nWHERE t.name=~'(?i).*node name1.*' AND s.name=~'(?i).*node name2.*'\nRETURN s.detail" }, { "context": [ { "s.detail": "correct answer" } ] } ] }
yet the chatbot does not make use of the correct answer and says it doesn't have the information to answer the question. How can I inspect the qa_chain of the chain I have defined in my code?

[prakhar200998]

Hi, could you figure out a solution to the problem you stated above? I am stuck at a similar spot.

[samsanjith77]

Hey, same here did you solve this ...

[Leon-afk1]

Hello, maybe it's too late, but I think I got an answer to your question.
From what I understood, your problem is that the result of your cypher query only include the 'answer'.
The answer from dosubot :
As for your second question, the GraphCypherQAChain passes context to the Q&A prompt in the _call method. It retrieves the results from the graph database using the generated Cypher query and passes these results as the context to the qa_chain. The qa_chain then uses this context along with the question to generate an answer.

It mean that qa_chain write the answer with your question, and the result of your cypher query as the context. But if the result of your cypher query doesn't include any link to your question, qa_chain won't understand the link between your question and the context.
If I am not clear, here an example using something you put in your first question:
If you want the transactions that contain 's'
This is the basic MATCH and there is no problem there :
MATCH (t:transaction)-[:CONTAINS*]->(s)
But I think the query generated by the Q&A prompt in the _call method is something like that :
RETURN t as transaction
Here qa_chain will get a list of transaction but will wonder if they are really connected to your question, and then won't answer.
The query generated should be something like this to have some results :
RETURN t as transaction, s as containing

So I think the solution is in your prompt, you should add something like this :

• Always assign a meaningful name to every node and relationship in the MATCH clause
• Never return components not explicitly named in the MATCH clause.
• In the RETURN clause, include all named components (nodes, relationships, or properties) to ensure consistency and understanding.
• Always return all the nodes used in the MATCH clause to provide complete information to the user.