Add a progress bar in read write file

[pascalisfort]

Hello, I integrate plyfile in my python application to read very large point cloud file (ply).

I would like to follow the progress of the reading of my files. unless you have another proposal, here is the beginning of my reflection:
I separate the file into several pieces and then I read them one by one in a loop allowing me to follow the progress. the problem is that in this way some data is corrupted between my cuts and another problem is that I add additional processing (not insignificant).

Thank You
Pascal

[dranjan]

Hello! That's an interesting question, and I can see a few possibilities.

• I think the first and ideal option would be if you can memory-map the input file, in which case reading it would probably be fast enough that you wouldn't need the additional feedback of the progress bar. This has restrictions, though. I'll also point out that memory mapping is normally automatic, except in the case where list properties exist. If the only issue is the presence of list properties, and they all happen to have fixed lengths (like the vertex_indices property in a triangular mesh), then you can use the known_list_len argument as shown in the documentation to enable the memory mapping.

• Assuming you can't memory-map the file, there are still options. The first would be to insert some code in the loops that read in the data:

  • ASCII:

    python-plyfile/plyfile.py

    Line 611 in f4a6d2b

    for line in _islice(iter(stream.readline, ''), self.count):

  • Binary:

    python-plyfile/plyfile.py

    Line 663 in f4a6d2b

    for k in range(self.count):

  In both loops, the number of loop iterations is known in advance, so you could use something like tqdm to quickly spin up a progress bar on a per-element basis. I think this solution would require very little code, but you would have to fork and customize plyfile.py rather than installing it directly from PyPI, which could be a significant drawback.

• Alternatively, you could wrap your input file with a custom file-like object type that overloads its read (or readline for text files) method with this extra progress bar functionality, since plyfile does support file-like objects. This seems like a lot of work, but you wouldn't need to modify the library itself. Note that this would prevent all memory mapping on your file without even more extra work, even for elements that may have been memory-mapped automatically before.

• Maybe the simplest option is to pipe your input file through something like pv and load the file through standard input:

  # assuming the first argument to the script is the input file
  pv somefile.ply | python somescript.py /dev/stdin

  If there's only one input file and you're able to wrap the Python script with some shell logic like this, then this could be the simplest solution of all. (This can still work with multiple files and without any shell logic, again with a few different options that I'll skip over for now.) Like the previous option, though, this would prevent all memory mapping, since pipes can't be memory-mapped as far as I'm aware.

Do any of those sound plausible?