Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ensure sender channel is not dropped until receiver reads exit request #82

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 11 additions & 0 deletions src/bom/bitcode.rs
Original file line number Diff line number Diff line change
Expand Up @@ -209,6 +209,17 @@
// Send a token to shut down the event collector thread
sender.send(None)?;
let summary = event_consumer.join().unwrap();

// Explicitly drop the sender resource. Note that this explicit
// reference to sender here ensures that sender remains valid up
// through this point. Previously, this drop was not present, which
// caused a race condition wherein the sender was never needed after
// the `sender.send(None)` above and thus it could be destroyed at that
// point; if it was destroyed before the collect_events could read the
// `None` then the latter would instead get a Recv Error, causing
// failures to be reported.
drop(sender);
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, I'm a bit skeptical that this was the issue - resources are freed at the end of their scope, which is somewhat after this drop. The join should have always joined before the channel was released before.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The sender.send ends in a ?, which is a choice point that will end the basic block in which the send was called. Rust's Non-Lexical Lifetimes (NLL, https://rust-lang.github.io/rfcs/2094-nll.html) would then be able to conclude that since sender was unused in the remainder of the CFG that it could be removed at that point, which is prior to the thread join and could therefore open this race window.

Without explicit examination of the compilation output, I can't be sure of this, but NLL appears to be deployed (https://blog.rust-lang.org/2022/08/05/nll-by-default.html), so it is a likely culprit. Do you have any ideas for other culprits?

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is the error you are observing occurring sporadically or deterministically?

I can't tell from the NLL RFC if it affects when destructors are called or if it is just about lifetime scope computation.

I'd just be really surprised if this was the culprit... in cases where sender.send(None) fails (i.e., ? causes the early return), that means the other thread already crashed (or some crazy OOM). That call might as well be .unwrap(). Maybe we should just do that?

In cases where it isn't crashing, the queue is definitely still live until the .join() returns (or the .unwrap() on that line fails).

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh and as for my hypothesis for the underlying cause: I think the real error might be in the ptrace code. I see Error: Tracee died while in ptrace-stop in #81. I am pretty confident that there is at least one gnarly race condition in the ptrace code (either in build-bom or the ptrace wrapper).


let exitmod =
if bitcode_options.any_fail && summary.has_failures() {
// If the mainline succeeded (ec == 0), return an error indicating
Expand Down Expand Up @@ -830,7 +841,7 @@
}
}
}
}

Check warning on line 844 in src/bom/bitcode.rs

View workflow job for this annotation

GitHub Actions / build

field `env` is never read

Check warning on line 844 in src/bom/bitcode.rs

View workflow job for this annotation

GitHub Actions / build

field `env` is never read


/// A command that was run by a process we are tracing
Expand All @@ -841,7 +852,7 @@
pub struct RunCommand {
bin : OsString,
args : Vec<OsString>,
env : Vec<u8>,

Check warning on line 855 in src/bom/bitcode.rs

View workflow job for this annotation

GitHub Actions / build

field `env` is never read

Check warning on line 855 in src/bom/bitcode.rs

View workflow job for this annotation

GitHub Actions / build

field `env` is never read
cwd : PathBuf
}

Expand Down
Loading