Error reading ingest file /opt/bloodhound/work/tmp/*: error validating meta tag: file is not valid json

[Fenr1r-g]

Description:

While trying to ingest JSONs collected via Sharphound 2.4.1 using the webUI, the ingestion fails with the backend reporting the following error: "Error reading ingest file /opt/bloodhound/work/tmp/{temp_filename}: error validating meta tag: file is not valid json"

sample metatag collected using Sharphound 2.4.3: "meta":{"methods":29695,"type":"domains","count":1,"version":5}}

After testing collection with SharpHound 2.3.0 and 2.3.4, both have been successfully ingested and are displayed appropriately.

I am using the latest version of BloodhoundCE using the docker-compose instructions with no special modifications made.

Component(s) Affected:

• [X ] UI
• API
• Neo4j
• PostgreSQL
• [ X] Data Collector (SharpHound, AzureHound)
• Other (tooling, documentation, etc.)

Steps to Reproduce:

1. Collect data using the latest version of SharpHound
2. Go to the Administration page of the bloodhound webUI
3. Upload a JSON for ingestion
4. See error in the docker terminal

Expected Behavior:

Successful ingestion.

Actual Behavior:

An error.

Environment Information:

BloodHound: Latest

Collector: Sharphound 2.4.3

OS: Ubuntu 22.03

Browser (if UI related): Firefox latest

Docker (if using Docker): latest

Additional Information:

Sharphound 2.3.0 and 2.3.4 both create JSONs that can be ingested and viewed. Additionally, when changing the JSON from UTF8BOM > UTF8, bloodhound claims to succesfully ingests the file, but doesn't actually update statistics or show anything in the UI. A quick check of the Neo4j database shows that the data is successfully uploaded in this way.

Potential Solution (Optional):

If you have any ideas about what might be causing the issue or how it could be fixed, you can share them here.

Related Issues:

If you've found related issues in the project's issue tracker, mention them here.

Contributor Checklist:

• I have searched the issue tracker to ensure this bug hasn't been reported before or is not already being addressed.
• I have provided clear steps to reproduce the issue.
• I have included relevant environment information details.
• I have attached necessary supporting documents.
• I have checked that any JSON files I am attempting to upload to BloodHound are valid.

[StephenHinck]

@Fenr1r-g - we have a team actively looking at SharpHound collection and file ingest. I'll make sure this is pulled into that effort for review!

[brandonshearin]

@Fenr1r-g I believe a PR just merged that should address this problem , which will be included in our next release. If you are seeking to test out this fix sooner than the next release date, you can do so by pulling the edge image from docker:

docker pull specterops/bloodhound:edge

Let me know if this issue persists!

[Anon-Exploiter]

@Fenr1r-g I believe a PR just merged that should address this problem , which will be included in our next release. If you are seeking to test out this fix sooner than the next release date, you can do so by pulling the edge image from docker:

docker pull specterops/bloodhound:edge

Let me know if this issue persists!

Ran into the same issue with edge version

[StephenHinck]

That PR was a first attempt but didn't fully solve the problem. We've got a second crack at it going here: #764. Still hopefully tracking to the next release (Targeting the week of August 19th)!