Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

remove busy wait from data_streamer #140

Merged
merged 13 commits into from
Jun 20, 2024
Merged

Conversation

aniketmaurya
Copy link
Collaborator

@aniketmaurya aniketmaurya commented Jun 17, 2024

Before submitting
  • Was this discussed/agreed via a Github issue? (no need for typos and docs improvements)
  • Did you read the contributor guideline, Pull Request section?
  • Did you make sure to update the docs?
  • Did you write any new necessary tests?

What does this PR do?

data_streamer polls the multiprocessing Pipe Connection for new data and busy waiting causes async loop to hang as discovered in #133.

  • This PR, removes the busy polling and uses async queue to consume high speed data.

  • When speed of LitAPI generator (prediction, let's say a for loop of 1B) then consumption speed of data_streamer fails to keep up that pace and data is lost. This PR makes sure that all the data is consumed in a go whenever it's available.

PR review

Anyone in the community is free to review the PR once the tests have passed.
If we didn't discuss your PR in GitHub issues there's a high chance it will not be merged.

Did you have fun?

Make sure you had fun coding 🙃

@aniketmaurya aniketmaurya requested a review from lantiga as a code owner June 17, 2024 15:17
Copy link

codecov bot commented Jun 17, 2024

Codecov Report

Attention: Patch coverage is 89.28571% with 3 lines in your changes missing coverage. Please review.

Project coverage is 80%. Comparing base (53dd29f) to head (adae805).

Additional details and impacted files
@@         Coverage Diff         @@
##           main   #140   +/-   ##
===================================
  Coverage    80%    80%           
===================================
  Files        13     13           
  Lines       890    903   +13     
===================================
+ Hits        715    726   +11     
- Misses      175    177    +2     

Copy link
Collaborator

@lantiga lantiga left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, but please address the comments before merging

src/litserve/server.py Show resolved Hide resolved
src/litserve/server.py Show resolved Hide resolved
src/litserve/server.py Outdated Show resolved Hide resolved
@aniketmaurya aniketmaurya enabled auto-merge (squash) June 20, 2024 11:08
@aniketmaurya aniketmaurya merged commit 178e6c1 into main Jun 20, 2024
19 checks passed
@aniketmaurya aniketmaurya deleted the aniket/fix-streaming-loop branch June 20, 2024 11:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants