Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Consume fetch stream output in tests #11885

Merged
merged 1 commit into from
Dec 17, 2024

Conversation

GregoryTravis
Copy link
Contributor

@GregoryTravis GregoryTravis commented Dec 16, 2024

This does not solve the problem behind #11878, but it does consume the stream and close it, so that the problem is not triggered in CI. The problem is not consistently reproducible since it relies on cache size pressure.

Closes #11879.

Checklist

Please ensure that the following checklist has been satisfied before submitting the PR:

  • The documentation has been updated, if necessary.
  • Screenshots/screencasts have been attached, if there are any visual changes. For interactive or animated visual changes, a screencast is preferred.
  • All code follows the
    Scala,
    Java,
    TypeScript,
    and
    Rust
    style guides. In case you are using a language not listed above, follow the Rust style guide.
  • Unit tests have been written where possible.
  • If meaningful changes were made to logic or tests affecting Enso Cloud integration in the libraries,
    or the Snowflake database integration, a run of the Extra Tests has been scheduled.
    • If applicable, it is suggested to paste a link to a successful run of the Extra Tests.

@GregoryTravis GregoryTravis marked this pull request as ready for review December 16, 2024 22:12
@GregoryTravis GregoryTravis added the CI: No changelog needed Do not require a changelog entry for this PR. label Dec 16, 2024
@radeusgd
Copy link
Member

radeusgd commented Dec 17, 2024

This does not solve the problem behind #11878

Should it then close that ticket? If we close it, shall we create a new ticket to track the remaining issue?

Oh sorry, ignore my comment - I realize now these are 2 different tickets, their numbers were so close that I missed that - perhaps not enough coffee today 😅

@@ -228,16 +228,16 @@ add_specs suite_builder =
group_builder.specify "Cache policy should work for HTTP.fetch" pending=pending_has_url <| Test.with_retries <|
with_default_cache <|
expect_counts [0, 0] <|
HTTP.fetch url0 cache_policy=Cache_Policy.No_Cache
HTTP.fetch url1 cache_policy=Cache_Policy.No_Cache
HTTP.fetch url0 cache_policy=Cache_Policy.No_Cache . decode_as_text
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, the internal methods were made with assumption that their result will always be consumed. Perhaps that assumption is a bit too trusting on the consumer, so we may want to revisit it, but for now it's probably good enough workaround.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Other alternative is - shall we cease testing the private HTTP.fetch method and instead rely on Data.fetch?

HTTP.fetch is the low-level one that relies on the consumer ensuring that the stream is processed, and avoids any auto-materialization for efficiency.

Data.fetch is the user-facing one that already handles the materialization - so even if the result is not consumed, the stream will have already been closed. That was the rationale of stream materialization - wrap streams passed to the user ensuring that they can be used 0 or multiple times VS exactly-once semantics of HTTP.fetch.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I think this is a good idea -- I've added it to #11879, which will be done sooner than a full solution to the problem.

Copy link
Member

@radeusgd radeusgd left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Workaround looks good.

I'm wondering if Data.fetch could also work as an alternative. But I think it's good enough as-is
and this is just to stabilize the tests, we need to look into the root cause anyway.

@GregoryTravis GregoryTravis added the CI: Ready to merge This PR is eligible for automatic merge label Dec 17, 2024
@mergify mergify bot merged commit d53c51f into develop Dec 17, 2024
38 of 39 checks passed
@mergify mergify bot deleted the wip/gmt/11879-consume-fetch-stream-in-tests branch December 17, 2024 20:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
CI: No changelog needed Do not require a changelog entry for this PR. CI: Ready to merge This PR is eligible for automatic merge
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Fix fetch tests to always consume the fetched data and close the stream
3 participants