Skip to content

Commit

Permalink
Error streaming: increase errors file payload limit
Browse files Browse the repository at this point in the history
Summary:
Context: errors are "streamed" by being written into the errors.bin file. That file consists of "payloads", which usually contains a batch of errors. Each payload is marshalled data with a header indicating the size in bytes of the payload.

When reading the errors.bin file, the client first reads the payload size from the header. To avoid allocating too much memory, we assert that the payload size we read is below a certain limit. That limit was initially set to 20MB, even though the comment explaining the limit was
```
(* This assert is in case the file is garbled, and we read a crazy-big size,
to avoid allocating say a 20gb bytes array and having the machine get stuck. *)
```
So it's likely the author meant to set the limit at 20GB instead of 20MB, but I'm not sure.

20MB was too small for typechecks with large amounts of errors, so I've kept conservatively increasing that limit to the current 200MB. But this still isn't enough according to telemetry, which shows payload of 80k+ errors and of 600MB+.

In this diff, I'm setting the limit to 4GB, and hoping this is the last increase.

Reviewed By: patriciamckenzie

Differential Revision: D67334771

fbshipit-source-id: 3f6e14ab2f10cf9a27d4471732f0e8d373604263
  • Loading branch information
Catherine Gasnier authored and facebook-github-bot committed Dec 17, 2024
1 parent 3bf74e4 commit 2021916
Showing 1 changed file with 2 additions and 2 deletions.
4 changes: 2 additions & 2 deletions hphp/hack/src/client_and_server/server_progress.ml
Original file line number Diff line number Diff line change
Expand Up @@ -271,7 +271,7 @@ module ErrorsFile = struct
log_message: string;
}

let max_payload_size = 200_000_000
let max_payload_size = 4_000_000_000 (* 4 GB *)

let assert_payload_size_ok size direction (message : message option) =
if size >= max_payload_size then (
Expand Down Expand Up @@ -332,7 +332,7 @@ module ErrorsFile = struct
| Some preamble ->
let size = Marshal_tools.parse_preamble preamble in
(* This assert is in case the file is garbled, and we read a crazy-big size,
to avoid allocating say a 20gb bytes array and having the machine get stuck. *)
to avoid allocating say a 10gb bytes array and having the machine get stuck. *)
assert_payload_size_ok size `Read None;
(match Sys_utils.read_non_intr fd size with
| None -> Error Malformed
Expand Down

0 comments on commit 2021916

Please sign in to comment.