-
Notifications
You must be signed in to change notification settings - Fork 361
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
RocM: filter VRAM fetch by HIP_VISIBLE_DEVICES / CUDA_VISIBLE_DEVICES #1106
base: concedo
Are you sure you want to change the base?
Conversation
…fixes ggerganov#1104) Signed-off-by: Jeroen Mostert <jeroen.mostert@cm.com>
I don't think this is the optimal way to do it. The problem is that it requires the user to correctly set HIP_VISIBLE_DEVICES manually. Instead, it should be determining the GPU type of the device from rocminfo |
This change is not about making things work different if the user has not set
This change is only about fixing 2. Any changes you want to make to 1, whether using I would also like to reiterate that we cannot use |
The GPU to run on is up to the user (the names are displayed), but the VRAM estimation should only come from dGPUs, thats why I would like to exclude iGPUs from the calculation. I would not like to make them completely unselectable, the user should make that choice. |
So if I understand correctly, when autolayering is used and we end up running on an iGPU (exclusively), the number of layers offloaded should always be 0? I could see that working. Even if you want that, though, that's still orthogonal to what this patch does, since you also need this for a selection of dGPUs (which will become relevant for my own system around oh, next week, I hope). The behavior I myself would like to see for autolayering is:
I still don't know of any reliable way to detect an iGPU, though. Various heuristics suggest themselves, but I couldn't tell you which if any are reliable. (Do iGPUs always have "AMD Radeon Graphics" for the marketing name? Is their uuid always "GPU-XX"? Is this never true for dGPUs? Maybe, I couldn't tell you.) |
For APU detection, there is some good news: |
Fixes #1104.
Or at least expresses the idea I had. Tested with a grand total of one machine, namely mine, which has one discrete GPU and an iGPU.