Can kyuubi engine set different resources?

[xcl1102]

Can I set up unused yarn resources for engines of different users?
For example, using the spark engine, can I set different executor numbers and memory for user1 and user2?

[pan3793]

I'm not sure what you mean "unused yarn resources".

By default, Kyuubi uses the USER share level, which means each user creates their own Spark application.

bin/beeline -u jdbc:kyuubi://kyuubi:10009 -n tom \
  --hiveconf spark.executor.memory=2g

bin/beeline -u jdbc:kyuubi://kyuubi:10009 -n jerry \
  --hiveconf spark.executor.memory=8g

With the above commands, Tom and Jerry create their own Spark application with specific executor memory.

[pan3793]

BTW, to utilize the resource of your YARN cluster, DRA is recommended instead of using the fixed executors.

[xcl1102]

Is it possible to set different spark.dynamicAllocation.maxExecutors for each engine? Set up more executors for users who perform large tasks, and set up a small number of executors for users who perform short jobs to reduce resource utilization.

[pan3793]

you can pass any legal Spark configurations on creating engines.

[xcl1102]

thanks you