Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: Hotfit for Nitro loading on CPU with hyper-threading support #931

Merged
merged 6 commits into from
Dec 12, 2023
Merged

feat: Hotfit for Nitro loading on CPU with hyper-threading support #931

merged 6 commits into from
Dec 12, 2023

Conversation

hiro-v
Copy link
Contributor

@hiro-v hiro-v commented Dec 9, 2023 •
edited
Loading

Problem statement:

  • Nitro needs to know exactly the number of physical core and can combine it to run with numactl on linux

Solution:

  • Add systeminformation to get cpu.physicalCores as INT an pass to nitro

  • Mac M

  • Windows

  • Linux

References:

@hiro-v hiro-v added P0: critical Mission critical type: enhancement Improves a current feature type: epic A major feature or initiative labels Dec 9, 2023
@hiro-v hiro-v self-assigned this Dec 9, 2023
@github-actions github-actions bot added the type: feature request A new feature label Dec 9, 2023
@hiro-v hiro-v added this to the v0.5.0 milestone Dec 9, 2023
@linhtran174
Copy link
Contributor

Model cannot be loaded on my Windows PC, how do you check the logging when spawning Nitro process?

@hiro-v hiro-v changed the title feat: Nitro sensing hardware feat: Hotfit for Nitro loading on CPU with hyper-threading support Dec 11, 2023
@dan-homebrew dan-homebrew removed the type: epic A major feature or initiative label Dec 11, 2023
@hiro-v hiro-v force-pushed the fix/nitro_perf_cpu branch from c8f4af9 to 577921f Compare December 11, 2023 23:21
@hiro-v hiro-v removed the request for review from linhtran174 December 11, 2023 23:25
tikikun
tikikun approved these changes Dec 12, 2023
View reviewed changes
Copy link
Contributor

@tikikun tikikun left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtn

extensions/inference-nitro-extension/src/module.ts Outdated Show resolved Hide resolved
@hiro-v hiro-v merged commit fef97f6 into main Dec 12, 2023
@hiro-v hiro-v deleted the fix/nitro_perf_cpu branch December 12, 2023 01:15
@linhtran174
Copy link
Contributor

lgtm

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@tikikun tikikun tikikun approved these changes

@louis-jan louis-jan Awaiting requested review from louis-jan

Assignees

@hiro-v hiro-v

Labels
P0: critical Mission critical type: enhancement Improves a current feature type: feature request A new feature
Projects
Jan & Cortex
Archived in project
Milestone
Jan Windows MVP
Development

Successfully merging this pull request may close these issues.
4 participants
@hiro-v @linhtran174 @tikikun @dan-homebrew