Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

samtools sort: fail to open "./samtools.36604.8950.tmp.1019.bam": Too many open files #343

Open
sanjin725 opened this issue Feb 19, 2024 · 4 comments
Labels

Comments

@sanjin725
Copy link

Dear administrator.

Hello, this error occurred when I was using juicer to process HIC data. How should I solve this problem?
The error is:
[bam_sort_core] merging from 1530 files and 170 in-memory blocks...
[E::hts_open_format] Failed to open file "./samtools.214728.6262.tmp.1019.bam" : Too many open files
samtools sort: fail to open "./samtools.214728.6262.tmp.1019.bam": Too many open files
***! Failure during chimera handling of /gpfs/home/liyixian/soft/3DDNA/juicer/splits/hic.fastq.gz
I also provided a complete log below:
juicer.log
Good luck, looking forward to your reply
Xiaoxin Pei

@sanjin725 sanjin725 added the bug label Feb 19, 2024
@yuan102379
Copy link

I have also meet with the simmilar issue? Have this comment been solved ?

@BRNiu
Copy link

BRNiu commented Apr 10, 2024

Same issue. Is there any solution?

@cohen-r
Copy link

cohen-r commented Sep 9, 2024

Hi! I am having the same issue. Has anyone figured out a solution?

@ycl6
Copy link

ycl6 commented Oct 30, 2024

Hi all, I have submitted a PR (#362) which hopefully will address this issue.

When running juicer (from CPU/juicer.sh), you can set the memory per thread used by samtools sort with a new --mem flag. The default memory used by samtools sort is 768M per thread. By setting a larger memory per thread, more data can be stored in memory before the content is written to a temporary file, therefore reducing the total number of opened temporary files required during the sort process.

Please take note of the total amount of memory that is available on your processing platform and the number of threads you are using. For example, if there are 64G memory available to you and you are running with 8 threads. Then the maximum memory per thread should be 8GB (i.e. -t 8 --mem 8G).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

5 participants