Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Error streaming: increase errors file payload limit
Summary: Context: errors are "streamed" by being written into the errors.bin file. That file consists of "payloads", which usually contains a batch of errors. Each payload is marshalled data with a header indicating the size in bytes of the payload. When reading the errors.bin file, the client first reads the payload size from the header. To avoid allocating too much memory, we assert that the payload size we read is below a certain limit. That limit was initially set to 20MB, even though the comment explaining the limit was ``` (* This assert is in case the file is garbled, and we read a crazy-big size, to avoid allocating say a 20gb bytes array and having the machine get stuck. *) ``` So it's likely the author meant to set the limit at 20GB instead of 20MB, but I'm not sure. 20MB was too small for typechecks with large amounts of errors, so I've kept conservatively increasing that limit to the current 200MB. But this still isn't enough according to telemetry, which shows payload of 80k+ errors and of 600MB+. In this diff, I'm setting the limit to 4GB, and hoping this is the last increase. Reviewed By: patriciamckenzie Differential Revision: D67334771 fbshipit-source-id: 3f6e14ab2f10cf9a27d4471732f0e8d373604263
- Loading branch information