Support for Intel iGPU? #1725

DenuxPlays · 2024-02-19T16:19:31Z

DenuxPlays
Feb 19, 2024

Hey,
I've wanted to try out LocalAI and I've wondered if this project has support for Intel iGPUs?
And if so than maybe you can share some experience on how to get it working and anything else I might have to know.

Answered by mudler

Feb 19, 2024

Intel GPU is supported with sycl and, specifically sycl images.

While documentation is not there yet, you can already use the sycl images, for instance:

docker run -ti -p 8080:8080 -v $PWD/models:/build/models \
   --rm -e DEBUG=1 -e GGML_SYCL_DEVICE=0 \
   --device /dev/dri:/dev/dri -t \
   quay.io/go-skynet/local-ai:v2.8.2-sycl-f32-ffmpeg-core https://gist.githubusercontent.com/mudler/103de2576a8fd4b583f9bd53f4e4cefd/raw/9181d4add553326806b8fdbf4ff0cd65d2145bff/phi-2-sycl.yaml

If you are running it manually (by copying models config file into the model directory), mind to disable mmap. I also suggest to disable f16 which otherwise would yield into poor results.

View full answer

mudler · 2024-02-19T23:03:56Z

mudler
Feb 19, 2024
Maintainer

Intel GPU is supported with sycl and, specifically sycl images.

While documentation is not there yet, you can already use the sycl images, for instance:

docker run -ti -p 8080:8080 -v $PWD/models:/build/models \
   --rm -e DEBUG=1 -e GGML_SYCL_DEVICE=0 \
   --device /dev/dri:/dev/dri -t \
   quay.io/go-skynet/local-ai:v2.8.2-sycl-f32-ffmpeg-core https://gist.githubusercontent.com/mudler/103de2576a8fd4b583f9bd53f4e4cefd/raw/9181d4add553326806b8fdbf4ff0cd65d2145bff/phi-2-sycl.yaml

If you are running it manually (by copying models config file into the model directory), mind to disable mmap. I also suggest to disable f16 which otherwise would yield into poor results.

0 replies

Jamy-L · 2024-03-02T21:15:07Z

Jamy-L
Mar 2, 2024

With a core i5 12500, I am unable to run the documentation example docker run -e DEBUG=true --privileged -ti -v $PWD/models:/build/models -p 8080:8080 -v /dev/dri:/dev/dri --rm quay.io/go-skynet/local-ai:master-sycl-f32-ffmpeg-core phi-2 or the example given above. I have tried several models, including manual install, the failling point is the same as mentioned here #1660 (comment). SYCL reports 4 devices, I have tried the 4 of them. Surprisingly it find two iGPU with different compute capabilities.

8:28PM DBG GRPC(84fe2946ae6893ef8c676f248157ad62-127.0.0.1:41007): stderr found 4 SYCL devices:
8:28PM DBG GRPC(84fe2946ae6893ef8c676f248157ad62-127.0.0.1:41007): stderr   Device 0: Intel(R) UHD Graphics 770,        compute capability 1.3,
8:28PM DBG GRPC(84fe2946ae6893ef8c676f248157ad62-127.0.0.1:41007): stderr       max compute_units 32,   max work group size 512,        max sub group size 32,    global mem size 13245333504
8:28PM DBG GRPC(84fe2946ae6893ef8c676f248157ad62-127.0.0.1:41007): stderr   Device 1: Intel(R) FPGA Emulation Device,   compute capability 1.2,
8:28PM DBG GRPC(84fe2946ae6893ef8c676f248157ad62-127.0.0.1:41007): stderr       max compute_units 12,   max work group size 67108864,   max sub group size 64,    global mem size 16556670976
8:28PM DBG GRPC(84fe2946ae6893ef8c676f248157ad62-127.0.0.1:41007): stderr   Device 2: 12th Gen Intel(R) Core(TM) i5-12500,      compute capability 3.0,
8:28PM DBG GRPC(84fe2946ae6893ef8c676f248157ad62-127.0.0.1:41007): stderr       max compute_units 12,   max work group size 8192,       max sub group size 64,    global mem size 16556670976
8:28PM DBG GRPC(84fe2946ae6893ef8c676f248157ad62-127.0.0.1:41007): stderr   Device 3: Intel(R) UHD Graphics 770,        compute capability 3.0,
8:28PM DBG GRPC(84fe2946ae6893ef8c676f248157ad62-127.0.0.1:41007): stderr       max compute_units 32,   max work group size 512,        max sub group size 32,    global mem size 13245333504
8:28PM DBG GRPC(84fe2946ae6893ef8c676f248157ad62-127.0.0.1:41007): stderr Using device 0 (Intel(R) UHD Graphics 770) as main device

When doing a query, the gpu usage reaches 100% during a few seconds, wich can be seen using sudo intel_gpu_top. After that, i reach the error

8:49PM DBG GRPC(c0c3c83d0ec33ffe925657a56b06771b-127.0.0.1:38535): stderr The number of work-items in each dimension of a work-group cannot exceed {512, 512, 512} for this device -54 (PI_ERROR_INVALID_WORK_GROUP_SIZE)Exception caught at file:/build/backend/cpp/llama/llama.cpp/ggml-sycl.cpp, line:12708

I also tried to run manually installed models following, but the result is the same. Some model are not compatible (like CodeLlama) and won't offload any layer to the GPU. In this case, the inference will be successful (but runs 100% on CPU).

I am not sure if the problem is general or if it's only for me though

0 replies

lenaxia · 2024-03-25T19:31:50Z

lenaxia
Mar 25, 2024

I'm also getting the work items error. I had tried with and without REBUILD=TRUE to see if there was some compliation issue. This only happens with llava though, gpt4all works without issue.

The number of work-items in each dimension of a work-group cannot exceed {256, 256, 256} for this device -54 (PI_ERROR_INVALID_WORK_GROUP_SIZE)Exception caught at file:/build/backend/cpp/llama/llama.cpp/ggml-sycl.cpp, line:14577

ubuntu@terraform:~/workspace/temp(⎈|prod:home)$ curl $LOCALAI/v1/completions -H "Content-Type: application/json" -d '{
  "model": "llava",
  "prompt": "A long time ago in a galaxy far, far away",
  "temperature": 0.7
}'
{"error":{"code":500,"message":"rpc error: code = Unavailable desc = error reading from server: EOF","type":""}}

ubuntu@terraform:~/workspace/home-ops-prod/cluster/apps/home/localai/app(⎈|prod:home)$ curl $LOCALAI/v1/chat/completions -H "Content-Type: application/json" -d '{
   "model": "llava",
   "messages": [{"role": "user", "content": [{"type":"text", "text": "What is in the image?"}, {"type": "image_url", "image_url": {"url": "https://www.pixelstalk.net/wp-content/uploads/2016/05/1280x1024-Background.jpg" }}], "temperature": 0.9}]}'
{"error":{"code":500,"message":"rpc error: code = Unavailable desc = error reading from server: EOF","type":""}}

Unfortuantely the debug log does not provide a whole lot of useful information:

7:26PM DBG GRPC Service Ready
7:26PM DBG GRPC: Loading model with options: {state:{NoUnkeyedLiterals:{} DoNotCompare:[] DoNotCopy:[] atomicMessageInfo:<nil>} sizeCache:0 unknownFields:[] Model:ggml-model-q4_k.gguf ContextSize:4096 Seed:1520027211 NBatch:512 F16Memory:true MLock:false MMap:true VocabOnly:false LowVRAM:false Embeddings:false NUMA:false NGPULayers:90 MainGPU: TensorSplit: Threads:11 LibrarySearchPath: RopeFreqBase:0 RopeFreqScale:0 RMSNormEps:0 NGQA:0 ModelFile:/models/ggml-model-q4_k.gguf Device: UseTriton:false ModelBaseName: UseFastTokenizer:false PipelineType: SchedulerType: CUDA:false CFGScale:0 IMG2IMG:false CLIPModel: CLIPSubfolder: CLIPSkip:0 ControlNet: Tokenizer: LoraBase: LoraAdapter: LoraScale:0 NoMulMatQ:false DraftModel: AudioPath: Quantization: GPUMemoryUtilization:0 TrustRemoteCode:false EnforceEager:false SwapSpace:0 MaxModelLen:0 MMProj:mmproj-model-f16.gguf RopeScaling: YarnExtFactor:0 YarnAttnFactor:0 YarnBetaFast:0 YarnBetaSlow:0 Type:}
7:26PM DBG GRPC(ggml-model-q4_k.gguf-127.0.0.1:41719): stderr ggml_init_sycl: GGML_SYCL_DEBUG: 0
7:26PM DBG GRPC(ggml-model-q4_k.gguf-127.0.0.1:41719): stderr ggml_init_sycl: GGML_SYCL_F16: no
7:26PM DBG GRPC(ggml-model-q4_k.gguf-127.0.0.1:41719): stderr found 4 SYCL devices:
7:26PM DBG GRPC(ggml-model-q4_k.gguf-127.0.0.1:41719): stderr |  |                  |                                             |Compute   |Max compute|Max work|Max sub|               |
7:26PM DBG GRPC(ggml-model-q4_k.gguf-127.0.0.1:41719): stderr |ID|       Device Type|                                         Name|capability|units      |group   |group  |Global mem size|
7:26PM DBG GRPC(ggml-model-q4_k.gguf-127.0.0.1:41719): stderr |--|------------------|---------------------------------------------|----------|-----------|--------|-------|---------------|
7:26PM DBG GRPC(ggml-model-q4_k.gguf-127.0.0.1:41719): stdout {"timestamp":1711394788,"level":"INFO","function":"load_model","line":449,"message":"Multi Modal Mode Enabled"}
7:26PM DBG GRPC(ggml-model-q4_k.gguf-127.0.0.1:41719): stderr | 0|[level_zero:gpu:0]|                     Intel(R) HD Graphics 530|       1.3|         24|     256|     32|    53845114880|
7:26PM DBG GRPC(ggml-model-q4_k.gguf-127.0.0.1:41719): stderr | 1|    [opencl:gpu:0]|                     Intel(R) HD Graphics 530|       3.0|         24|     256|     32|    53845114880|
7:26PM DBG GRPC(ggml-model-q4_k.gguf-127.0.0.1:41719): stderr | 2|    [opencl:cpu:0]|     Intel(R) Core(TM) i7-6700T CPU @ 2.80GHz|       3.0|          8|    8192|     64|    67306397696|
7:26PM DBG GRPC(ggml-model-q4_k.gguf-127.0.0.1:41719): stderr | 3|    [opencl:acc:0]|               Intel(R) FPGA Emulation Device|       1.2|          8|67108864|     64|    67306397696|
7:26PM DBG GRPC(ggml-model-q4_k.gguf-127.0.0.1:41719): stderr key clip.vision.image_grid_pinpoints not found in file
7:26PM DBG GRPC(ggml-model-q4_k.gguf-127.0.0.1:41719): stderr key clip.vision.mm_patch_merge_type not found in file
7:26PM DBG GRPC(ggml-model-q4_k.gguf-127.0.0.1:41719): stderr key clip.vision.image_crop_resolution not found in file
7:26PM DBG GRPC(ggml-model-q4_k.gguf-127.0.0.1:41719): stderr llama_model_loader: loaded meta data with 19 key-value pairs and 291 tensors from /models/ggml-model-q4_k.gguf (version GGUF V2)
7:26PM DBG GRPC(ggml-model-q4_k.gguf-127.0.0.1:41719): stderr llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
7:26PM DBG GRPC(ggml-model-q4_k.gguf-127.0.0.1:41719): stderr llama_model_loader: - kv   0:                       general.architecture str              = llama
7:26PM DBG GRPC(ggml-model-q4_k.gguf-127.0.0.1:41719): stderr llama_model_loader: - kv   1:                               general.name str              = ..
7:26PM DBG GRPC(ggml-model-q4_k.gguf-127.0.0.1:41719): stderr llama_model_loader: - kv   2:                       llama.context_length u32              = 32768
7:26PM DBG GRPC(ggml-model-q4_k.gguf-127.0.0.1:41719): stderr llama_model_loader: - kv   3:                     llama.embedding_length u32              = 4096
7:26PM DBG GRPC(ggml-model-q4_k.gguf-127.0.0.1:41719): stderr llama_model_loader: - kv   4:                          llama.block_count u32              = 32
7:26PM DBG GRPC(ggml-model-q4_k.gguf-127.0.0.1:41719): stderr llama_model_loader: - kv   5:                  llama.feed_forward_length u32              = 14336
7:26PM DBG GRPC(ggml-model-q4_k.gguf-127.0.0.1:41719): stderr llama_model_loader: - kv   6:                 llama.rope.dimension_count u32              = 128
7:26PM DBG GRPC(ggml-model-q4_k.gguf-127.0.0.1:41719): stderr llama_model_loader: - kv   7:                 llama.attention.head_count u32              = 32
7:26PM DBG GRPC(ggml-model-q4_k.gguf-127.0.0.1:41719): stderr llama_model_loader: - kv   8:              llama.attention.head_count_kv u32              = 8
7:26PM DBG GRPC(ggml-model-q4_k.gguf-127.0.0.1:41719): stderr llama_model_loader: - kv   9:     llama.attention.layer_norm_rms_epsilon f32              = 0.000010
7:26PM DBG GRPC(ggml-model-q4_k.gguf-127.0.0.1:41719): stderr llama_model_loader: - kv  10:                       llama.rope.freq_base f32              = 10000.000000
7:26PM DBG GRPC(ggml-model-q4_k.gguf-127.0.0.1:41719): stderr llama_model_loader: - kv  11:                          general.file_type u32              = 15
7:26PM DBG GRPC(ggml-model-q4_k.gguf-127.0.0.1:41719): stderr llama_model_loader: - kv  12:                       tokenizer.ggml.model str              = llama
7:26PM DBG GRPC(ggml-model-q4_k.gguf-127.0.0.1:41719): stderr llama_model_loader: - kv  13:                      tokenizer.ggml.tokens arr[str,32000]   = ["<unk>", "<s>", "</s>", "<0x00>", "<...
7:26PM DBG GRPC(ggml-model-q4_k.gguf-127.0.0.1:41719): stderr llama_model_loader: - kv  14:                      tokenizer.ggml.scores arr[f32,32000]   = [0.000000, 0.000000, 0.000000, 0.0000...
7:26PM DBG GRPC(ggml-model-q4_k.gguf-127.0.0.1:41719): stderr llama_model_loader: - kv  15:                  tokenizer.ggml.token_type arr[i32,32000]   = [2, 3, 3, 6, 6, 6, 6, 6, 6, 6, 6, 6, ...
7:26PM DBG GRPC(ggml-model-q4_k.gguf-127.0.0.1:41719): stderr llama_model_loader: - kv  16:                tokenizer.ggml.bos_token_id u32              = 1
7:26PM DBG GRPC(ggml-model-q4_k.gguf-127.0.0.1:41719): stderr llama_model_loader: - kv  17:                tokenizer.ggml.eos_token_id u32              = 2
7:26PM DBG GRPC(ggml-model-q4_k.gguf-127.0.0.1:41719): stderr llama_model_loader: - kv  18:               general.quantization_version u32              = 2
7:26PM DBG GRPC(ggml-model-q4_k.gguf-127.0.0.1:41719): stderr llama_model_loader: - type  f32:   65 tensors
7:26PM DBG GRPC(ggml-model-q4_k.gguf-127.0.0.1:41719): stderr llama_model_loader: - type q4_K:  193 tensors
7:26PM DBG GRPC(ggml-model-q4_k.gguf-127.0.0.1:41719): stderr llama_model_loader: - type q6_K:   33 tensors
7:26PM DBG GRPC(ggml-model-q4_k.gguf-127.0.0.1:41719): stderr llm_load_vocab: special tokens definition check successful ( 259/32000 ).
7:26PM DBG GRPC(ggml-model-q4_k.gguf-127.0.0.1:41719): stderr llm_load_print_meta: format           = GGUF V2
7:26PM DBG GRPC(ggml-model-q4_k.gguf-127.0.0.1:41719): stderr llm_load_print_meta: arch             = llama
7:26PM DBG GRPC(ggml-model-q4_k.gguf-127.0.0.1:41719): stderr llm_load_print_meta: vocab type       = SPM
7:26PM DBG GRPC(ggml-model-q4_k.gguf-127.0.0.1:41719): stderr llm_load_print_meta: n_vocab          = 32000
7:26PM DBG GRPC(ggml-model-q4_k.gguf-127.0.0.1:41719): stderr llm_load_print_meta: n_merges         = 0
7:26PM DBG GRPC(ggml-model-q4_k.gguf-127.0.0.1:41719): stderr llm_load_print_meta: n_ctx_train      = 32768
7:26PM DBG GRPC(ggml-model-q4_k.gguf-127.0.0.1:41719): stderr llm_load_print_meta: n_embd           = 4096
7:26PM DBG GRPC(ggml-model-q4_k.gguf-127.0.0.1:41719): stderr llm_load_print_meta: n_head           = 32
7:26PM DBG GRPC(ggml-model-q4_k.gguf-127.0.0.1:41719): stderr llm_load_print_meta: n_head_kv        = 8
7:26PM DBG GRPC(ggml-model-q4_k.gguf-127.0.0.1:41719): stderr llm_load_print_meta: n_layer          = 32
7:26PM DBG GRPC(ggml-model-q4_k.gguf-127.0.0.1:41719): stderr llm_load_print_meta: n_rot            = 128
7:26PM DBG GRPC(ggml-model-q4_k.gguf-127.0.0.1:41719): stderr llm_load_print_meta: n_embd_head_k    = 128
7:26PM DBG GRPC(ggml-model-q4_k.gguf-127.0.0.1:41719): stderr llm_load_print_meta: n_embd_head_v    = 128
7:26PM DBG GRPC(ggml-model-q4_k.gguf-127.0.0.1:41719): stderr llm_load_print_meta: n_gqa            = 4
7:26PM DBG GRPC(ggml-model-q4_k.gguf-127.0.0.1:41719): stderr llm_load_print_meta: n_embd_k_gqa     = 1024
7:26PM DBG GRPC(ggml-model-q4_k.gguf-127.0.0.1:41719): stderr llm_load_print_meta: n_embd_v_gqa     = 1024
7:26PM DBG GRPC(ggml-model-q4_k.gguf-127.0.0.1:41719): stderr llm_load_print_meta: f_norm_eps       = 0.0e+00
7:26PM DBG GRPC(ggml-model-q4_k.gguf-127.0.0.1:41719): stderr llm_load_print_meta: f_norm_rms_eps   = 1.0e-05
7:26PM DBG GRPC(ggml-model-q4_k.gguf-127.0.0.1:41719): stderr llm_load_print_meta: f_clamp_kqv      = 0.0e+00
7:26PM DBG GRPC(ggml-model-q4_k.gguf-127.0.0.1:41719): stderr llm_load_print_meta: f_max_alibi_bias = 0.0e+00
7:26PM DBG GRPC(ggml-model-q4_k.gguf-127.0.0.1:41719): stderr llm_load_print_meta: f_logit_scale    = 0.0e+00
7:26PM DBG GRPC(ggml-model-q4_k.gguf-127.0.0.1:41719): stderr llm_load_print_meta: n_ff             = 14336
7:26PM DBG GRPC(ggml-model-q4_k.gguf-127.0.0.1:41719): stderr llm_load_print_meta: n_expert         = 0
7:26PM DBG GRPC(ggml-model-q4_k.gguf-127.0.0.1:41719): stderr llm_load_print_meta: n_expert_used    = 0
7:26PM DBG GRPC(ggml-model-q4_k.gguf-127.0.0.1:41719): stderr llm_load_print_meta: causal attn      = 1
7:26PM DBG GRPC(ggml-model-q4_k.gguf-127.0.0.1:41719): stderr llm_load_print_meta: pooling type     = 0
7:26PM DBG GRPC(ggml-model-q4_k.gguf-127.0.0.1:41719): stderr llm_load_print_meta: rope type        = 0
7:26PM DBG GRPC(ggml-model-q4_k.gguf-127.0.0.1:41719): stderr llm_load_print_meta: rope scaling     = linear
7:26PM DBG GRPC(ggml-model-q4_k.gguf-127.0.0.1:41719): stderr llm_load_print_meta: freq_base_train  = 10000.0
7:26PM DBG GRPC(ggml-model-q4_k.gguf-127.0.0.1:41719): stderr llm_load_print_meta: freq_scale_train = 1
7:26PM DBG GRPC(ggml-model-q4_k.gguf-127.0.0.1:41719): stderr llm_load_print_meta: n_yarn_orig_ctx  = 32768
7:26PM DBG GRPC(ggml-model-q4_k.gguf-127.0.0.1:41719): stderr llm_load_print_meta: rope_finetuned   = unknown
7:26PM DBG GRPC(ggml-model-q4_k.gguf-127.0.0.1:41719): stderr llm_load_print_meta: ssm_d_conv       = 0
7:26PM DBG GRPC(ggml-model-q4_k.gguf-127.0.0.1:41719): stderr llm_load_print_meta: ssm_d_inner      = 0
7:26PM DBG GRPC(ggml-model-q4_k.gguf-127.0.0.1:41719): stderr llm_load_print_meta: ssm_d_state      = 0
7:26PM DBG GRPC(ggml-model-q4_k.gguf-127.0.0.1:41719): stderr llm_load_print_meta: ssm_dt_rank      = 0
7:26PM DBG GRPC(ggml-model-q4_k.gguf-127.0.0.1:41719): stderr llm_load_print_meta: model type       = 7B
7:26PM DBG GRPC(ggml-model-q4_k.gguf-127.0.0.1:41719): stderr llm_load_print_meta: model ftype      = Q4_K - Medium
7:26PM DBG GRPC(ggml-model-q4_k.gguf-127.0.0.1:41719): stderr llm_load_print_meta: model params     = 7.24 B
7:26PM DBG GRPC(ggml-model-q4_k.gguf-127.0.0.1:41719): stderr llm_load_print_meta: model size       = 4.07 GiB (4.83 BPW)
7:26PM DBG GRPC(ggml-model-q4_k.gguf-127.0.0.1:41719): stderr llm_load_print_meta: general.name     = ..
7:26PM DBG GRPC(ggml-model-q4_k.gguf-127.0.0.1:41719): stderr llm_load_print_meta: BOS token        = 1 '<s>'
7:26PM DBG GRPC(ggml-model-q4_k.gguf-127.0.0.1:41719): stderr llm_load_print_meta: EOS token        = 2 '</s>'
7:26PM DBG GRPC(ggml-model-q4_k.gguf-127.0.0.1:41719): stderr llm_load_print_meta: UNK token        = 0 '<unk>'
7:26PM DBG GRPC(ggml-model-q4_k.gguf-127.0.0.1:41719): stderr llm_load_print_meta: LF token         = 13 '<0x0A>'
7:26PM DBG GRPC(ggml-model-q4_k.gguf-127.0.0.1:41719): stderr ggml_backend_sycl_set_mul_device_mode: true
7:26PM DBG GRPC(ggml-model-q4_k.gguf-127.0.0.1:41719): stderr detect 1 SYCL GPUs: [0] with top Max compute units:24
7:26PM DBG GRPC(ggml-model-q4_k.gguf-127.0.0.1:41719): stderr get_memory_info: [warning] ext_intel_free_memory is not supported (export/set ZES_ENABLE_SYSMAN=1 to support), use total memory as free memory
7:26PM DBG GRPC(ggml-model-q4_k.gguf-127.0.0.1:41719): stdout clip_model_load: model name:   openai/clip-vit-large-patch14-336
7:26PM DBG GRPC(ggml-model-q4_k.gguf-127.0.0.1:41719): stderr llm_load_tensors: ggml ctx size =    0.22 MiB
7:26PM DBG GRPC(ggml-model-q4_k.gguf-127.0.0.1:41719): stdout clip_model_load: description:  image encoder for LLaVA
7:26PM DBG GRPC(ggml-model-q4_k.gguf-127.0.0.1:41719): stdout clip_model_load: GGUF version: 2
7:26PM DBG GRPC(ggml-model-q4_k.gguf-127.0.0.1:41719): stdout clip_model_load: alignment:    32
7:26PM DBG GRPC(ggml-model-q4_k.gguf-127.0.0.1:41719): stdout clip_model_load: n_tensors:    377
7:26PM DBG GRPC(ggml-model-q4_k.gguf-127.0.0.1:41719): stdout clip_model_load: n_kv:         18
7:26PM DBG GRPC(ggml-model-q4_k.gguf-127.0.0.1:41719): stdout clip_model_load: ftype:        f16
7:26PM DBG GRPC(ggml-model-q4_k.gguf-127.0.0.1:41719): stdout
7:26PM DBG GRPC(ggml-model-q4_k.gguf-127.0.0.1:41719): stdout clip_model_load: loaded meta data with 18 key-value pairs and 377 tensors from /models/mmproj-model-f16.gguf
7:26PM DBG GRPC(ggml-model-q4_k.gguf-127.0.0.1:41719): stdout clip_model_load: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
7:26PM DBG GRPC(ggml-model-q4_k.gguf-127.0.0.1:41719): stdout clip_model_load: - kv   0:                       general.architecture str              = clip
7:26PM DBG GRPC(ggml-model-q4_k.gguf-127.0.0.1:41719): stdout clip_model_load: - kv   1:                      clip.has_text_encoder bool             = false
7:26PM DBG GRPC(ggml-model-q4_k.gguf-127.0.0.1:41719): stdout clip_model_load: - kv   2:                    clip.has_vision_encoder bool             = true
7:26PM DBG GRPC(ggml-model-q4_k.gguf-127.0.0.1:41719): stdout clip_model_load: - kv   3:                   clip.has_llava_projector bool             = true
7:26PM DBG GRPC(ggml-model-q4_k.gguf-127.0.0.1:41719): stdout clip_model_load: - kv   4:                          general.file_type u32              = 1
7:26PM DBG GRPC(ggml-model-q4_k.gguf-127.0.0.1:41719): stdout clip_model_load: - kv   5:                               general.name str              = openai/clip-vit-large-patch14-336
7:26PM DBG GRPC(ggml-model-q4_k.gguf-127.0.0.1:41719): stdout clip_model_load: - kv   6:                        general.description str              = image encoder for LLaVA
7:26PM DBG GRPC(ggml-model-q4_k.gguf-127.0.0.1:41719): stdout clip_model_load: - kv   7:                     clip.vision.image_size u32              = 336
7:26PM DBG GRPC(ggml-model-q4_k.gguf-127.0.0.1:41719): stdout clip_model_load: - kv   8:                     clip.vision.patch_size u32              = 14
7:26PM DBG GRPC(ggml-model-q4_k.gguf-127.0.0.1:41719): stdout clip_model_load: - kv   9:               clip.vision.embedding_length u32              = 1024
7:26PM DBG GRPC(ggml-model-q4_k.gguf-127.0.0.1:41719): stdout clip_model_load: - kv  10:            clip.vision.feed_forward_length u32              = 4096
7:26PM DBG GRPC(ggml-model-q4_k.gguf-127.0.0.1:41719): stdout clip_model_load: - kv  11:                 clip.vision.projection_dim u32              = 768
7:26PM DBG GRPC(ggml-model-q4_k.gguf-127.0.0.1:41719): stdout clip_model_load: - kv  12:           clip.vision.attention.head_count u32              = 16
7:26PM DBG GRPC(ggml-model-q4_k.gguf-127.0.0.1:41719): stdout clip_model_load: - kv  13:   clip.vision.attention.layer_norm_epsilon f32              = 0.000010
7:26PM DBG GRPC(ggml-model-q4_k.gguf-127.0.0.1:41719): stdout clip_model_load: - kv  14:                    clip.vision.block_count u32              = 23
7:26PM DBG GRPC(ggml-model-q4_k.gguf-127.0.0.1:41719): stdout clip_model_load: - kv  15:                     clip.vision.image_mean arr[f32,3]       = [0.481455, 0.457828, 0.408211]
7:26PM DBG GRPC(ggml-model-q4_k.gguf-127.0.0.1:41719): stdout clip_model_load: - kv  16:                      clip.vision.image_std arr[f32,3]       = [0.268630, 0.261303, 0.275777]
7:26PM DBG GRPC(ggml-model-q4_k.gguf-127.0.0.1:41719): stdout clip_model_load: - kv  17:                              clip.use_gelu bool             = false
7:26PM DBG GRPC(ggml-model-q4_k.gguf-127.0.0.1:41719): stdout clip_model_load: - type  f32:  235 tensors
7:26PM DBG GRPC(ggml-model-q4_k.gguf-127.0.0.1:41719): stdout clip_model_load: - type  f16:  142 tensors
7:26PM DBG GRPC(ggml-model-q4_k.gguf-127.0.0.1:41719): stdout clip_model_load: CLIP using CPU backend
7:26PM DBG GRPC(ggml-model-q4_k.gguf-127.0.0.1:41719): stdout clip_model_load: text_encoder:   0
7:26PM DBG GRPC(ggml-model-q4_k.gguf-127.0.0.1:41719): stdout clip_model_load: vision_encoder: 1
7:26PM DBG GRPC(ggml-model-q4_k.gguf-127.0.0.1:41719): stdout clip_model_load: llava_projector:  1
7:26PM DBG GRPC(ggml-model-q4_k.gguf-127.0.0.1:41719): stdout clip_model_load: model size:     595.49 MB
7:26PM DBG GRPC(ggml-model-q4_k.gguf-127.0.0.1:41719): stdout clip_model_load: metadata size:  0.14 MB
7:26PM DBG GRPC(ggml-model-q4_k.gguf-127.0.0.1:41719): stdout clip_model_load: params backend buffer size =  595.49 MB (377 tensors)
7:26PM DBG GRPC(ggml-model-q4_k.gguf-127.0.0.1:41719): stdout clip_model_load: compute allocated memory: 32.89 MB
7:26PM DBG GRPC(ggml-model-q4_k.gguf-127.0.0.1:41719): stderr llm_load_tensors: offloading 32 repeating layers to GPU
7:26PM DBG GRPC(ggml-model-q4_k.gguf-127.0.0.1:41719): stderr llm_load_tensors: offloading non-repeating layers to GPU
7:26PM DBG GRPC(ggml-model-q4_k.gguf-127.0.0.1:41719): stderr llm_load_tensors: offloaded 33/33 layers to GPU
7:26PM DBG GRPC(ggml-model-q4_k.gguf-127.0.0.1:41719): stderr llm_load_tensors:      SYCL0 buffer size =  4095.05 MiB
7:26PM DBG GRPC(ggml-model-q4_k.gguf-127.0.0.1:41719): stderr llm_load_tensors:        CPU buffer size =    70.31 MiB
7:26PM DBG GRPC(ggml-model-q4_k.gguf-127.0.0.1:41719): stderr ..............................................................................................
7:26PM DBG GRPC(ggml-model-q4_k.gguf-127.0.0.1:41719): stderr llama_new_context_with_model: n_ctx      = 4096
7:26PM DBG GRPC(ggml-model-q4_k.gguf-127.0.0.1:41719): stderr llama_new_context_with_model: n_batch    = 512
7:26PM DBG GRPC(ggml-model-q4_k.gguf-127.0.0.1:41719): stderr llama_new_context_with_model: n_ubatch   = 512
7:26PM DBG GRPC(ggml-model-q4_k.gguf-127.0.0.1:41719): stderr llama_new_context_with_model: freq_base  = 10000.0
7:26PM DBG GRPC(ggml-model-q4_k.gguf-127.0.0.1:41719): stderr llama_new_context_with_model: freq_scale = 1
7:26PM DBG GRPC(ggml-model-q4_k.gguf-127.0.0.1:41719): stderr llama_kv_cache_init:      SYCL0 KV buffer size =   512.00 MiB
7:26PM DBG GRPC(ggml-model-q4_k.gguf-127.0.0.1:41719): stderr llama_new_context_with_model: KV self size  =  512.00 MiB, K (f16):  256.00 MiB, V (f16):  256.00 MiB
7:26PM DBG GRPC(ggml-model-q4_k.gguf-127.0.0.1:41719): stderr llama_new_context_with_model:  SYCL_Host  output buffer size =    62.50 MiB
7:26PM DBG GRPC(ggml-model-q4_k.gguf-127.0.0.1:41719): stderr llama_new_context_with_model:      SYCL0 compute buffer size =   296.00 MiB
7:26PM DBG GRPC(ggml-model-q4_k.gguf-127.0.0.1:41719): stderr llama_new_context_with_model:  SYCL_Host compute buffer size =    16.00 MiB
7:26PM DBG GRPC(ggml-model-q4_k.gguf-127.0.0.1:41719): stderr llama_new_context_with_model: graph splits: 2
7:27PM DBG GRPC(ggml-model-q4_k.gguf-127.0.0.1:41719): stdout {"timestamp":1711394835,"level":"INFO","function":"initialize","line":502,"message":"initializing slots","n_slots":1}
7:27PM DBG GRPC(ggml-model-q4_k.gguf-127.0.0.1:41719): stdout {"timestamp":1711394835,"level":"INFO","function":"initialize","line":514,"message":"new slot","slot_id":0,"n_ctx_slot":4096}
7:27PM DBG GRPC(ggml-model-q4_k.gguf-127.0.0.1:41719): stdout {"timestamp":1711394835,"level":"INFO","function":"launch_slot_with_data","line":887,"message":"slot is processing task","slot_id":0,"task_id":0}
7:27PM DBG GRPC(ggml-model-q4_k.gguf-127.0.0.1:41719): stdout {"timestamp":1711394835,"level":"INFO","function":"update_slots","line":1787,"message":"kv cache rm [p0, end)","slot_id":0,"task_id":0,"p0":0}
7:27PM DBG GRPC(ggml-model-q4_k.gguf-127.0.0.1:41719): stderr The number of work-items in each dimension of a work-group cannot exceed {256, 256, 256} for this device -54 (PI_ERROR_INVALID_WORK_GROUP_SIZE)Exception caught at file:/build/backend/cpp/llama/llama.cpp/ggml-sycl.cpp, line:14577
7:27PM DBG GRPC(ggml-model-q4_k.gguf-127.0.0.1:41719): stdout encode_image_with_clip: image embedding created: 576 tokens
7:27PM DBG GRPC(ggml-model-q4_k.gguf-127.0.0.1:41719): stdout
7:27PM DBG GRPC(ggml-model-q4_k.gguf-127.0.0.1:41719): stdout encode_image_with_clip: image encoded in 23718.28 ms by CLIP (   41.18 ms per image patch)
[10.42.3.127]:36326 500 - POST /v1/chat/completions

1 reply

fakezeta Apr 22, 2024
Collaborator

Hi,

sorry for the late reply. I had the same error and I opened the issue ggerganov/llama.cpp#5467 to the upstream llama.cpp and it's fixed now.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support for Intel iGPU? #1725

{{title}}

Replies: 3 comments 1 reply

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

Select a reply

Support for Intel iGPU? #1725

DenuxPlays Feb 19, 2024

Replies: 3 comments · 1 reply

mudler Feb 19, 2024 Maintainer

Jamy-L Mar 2, 2024

lenaxia Mar 25, 2024

fakezeta Apr 22, 2024 Collaborator

DenuxPlays
Feb 19, 2024

Replies: 3 comments 1 reply

mudler
Feb 19, 2024
Maintainer

Jamy-L
Mar 2, 2024

lenaxia
Mar 25, 2024

fakezeta Apr 22, 2024
Collaborator