-
Notifications
You must be signed in to change notification settings - Fork 323
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Consume fetch stream output in tests #11885
Conversation
Oh sorry, ignore my comment - I realize now these are 2 different tickets, their numbers were so close that I missed that - perhaps not enough coffee today 😅 |
@@ -228,16 +228,16 @@ add_specs suite_builder = | |||
group_builder.specify "Cache policy should work for HTTP.fetch" pending=pending_has_url <| Test.with_retries <| | |||
with_default_cache <| | |||
expect_counts [0, 0] <| | |||
HTTP.fetch url0 cache_policy=Cache_Policy.No_Cache | |||
HTTP.fetch url1 cache_policy=Cache_Policy.No_Cache | |||
HTTP.fetch url0 cache_policy=Cache_Policy.No_Cache . decode_as_text |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah, the internal methods were made with assumption that their result will always be consumed. Perhaps that assumption is a bit too trusting on the consumer, so we may want to revisit it, but for now it's probably good enough workaround.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Other alternative is - shall we cease testing the private HTTP.fetch
method and instead rely on Data.fetch
?
HTTP.fetch
is the low-level one that relies on the consumer ensuring that the stream is processed, and avoids any auto-materialization for efficiency.
Data.fetch
is the user-facing one that already handles the materialization - so even if the result is not consumed, the stream will have already been closed. That was the rationale of stream materialization - wrap streams passed to the user ensuring that they can be used 0 or multiple times VS exactly-once semantics of HTTP.fetch
.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, I think this is a good idea -- I've added it to #11879, which will be done sooner than a full solution to the problem.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Workaround looks good.
I'm wondering if Data.fetch
could also work as an alternative. But I think it's good enough as-is
and this is just to stabilize the tests, we need to look into the root cause anyway.
This does not solve the problem behind #11878, but it does consume the stream and close it, so that the problem is not triggered in CI. The problem is not consistently reproducible since it relies on cache size pressure.
Closes #11879.
Checklist
Please ensure that the following checklist has been satisfied before submitting the PR:
Scala,
Java,
TypeScript,
and
Rust
style guides. In case you are using a language not listed above, follow the Rust style guide.
or the Snowflake database integration, a run of the Extra Tests has been scheduled.