Loading Large CSV file

[dagheggelund]

Hello, my name is Dag and I have a question regarding using the .net driver to load
CSV files.
I split my incoming data into smaller sets and execute the following query in a loop.
LOAD CSV WITH HEADERS FROM $file AS row
MATCH (from:{fromLink} {{id: row.FromID}})
MATCH (to:{toLink} {{id: row.ToID}})
MERGE (from)-[:{predicate}]->(to)";

The query is executed using the _session.RunAsync(query, parameters);
Sometimes it appears to be failing without throwing an exception.
If I insert a call to execute a Count query before and after the load query then
it appears to not fail.
A second question I see, is that when it fails and throws an exception, the file name it
is trying to load is not the current file. (I am using a guid as the file name).
Any suggestions would be appreciated.

[thelonelyvulpes]

Hi @dagheggelund, sorry about the delay, I didn't notice you opened a discussion instead of an issue.

You execute the query using Session.RunAsync:

LOAD CSV WITH HEADERS FROM $file AS row
MATCH (from:{fromLink} {{id: row.FromID}})
MATCH (to:{toLink} {{id: row.ToID}})
MERGE (from)-[:{predicate}]->(to)"

You say it sometimes fails, without throwing any exceptions.
I have a few questions:

1. How does it fail?
2. Are you awaiting the Task return value?
3. Are you calling ConsumeAsync() on the result cursor.

I'd expect it look a little something like:

async Task ImportFileAsync(string file, IDriver driver, string database)
{
    await using IAsyncSession session = driver.AsyncSession(x => 
        x.WithDatabase(database)
        .WithDefaultAccessMode(AccessMode.Write));

    const string query = @"
        LOAD CSV WITH HEADERS FROM $file AS row
        MATCH (from:{fromLink} {{id: row.FromID}})
        MATCH (to:{toLink} {{id: row.ToID}})
        MERGE (from)-[:{predicate}]->(to)";
    
    IResultCursor cursor = await session.RunAsync(query, new { file });
    await cursor.ConsumeAsync();
}

API Docs for IResultCursor.ConsumeAsync
It is important to consume all cursors for write queries.

Your second question states that when it fails and does throw an exception that the file name it
is trying to load is not the current file.

This very much makes me think you are having issues with your sessions, note that sessions must be used in a thread safe fashion and make no effort to protect against misuse. Opening & closing sessions is very cheap so it is best to avoid reusing them.
If you want to do parallel workloads, make sure you use a separate session.

If you have any further questions please let me know.

[dagheggelund]

Thanks for the reply and explanations.
You are correct in that it was a session problem. I incorrectly used the same session for several CSV upload calls. I have modified the code so that it is using a new session for each CSV upload call, that appears to work very well.
I am not calling the await cursor.ConsumeAsync();, I will update my code to make that call.
Thanks again for your help.
Best regards Dag