Dask is simply the best way to implement task parallelism in Python today - if you are going anything that vaguely looks like task farming then you probably should consider using it. There is a lot of material already online discussing its general awesomeness, so it is left as an exercise to the reader to use their favourite search engine to find domain specific examples of its use.
While you can use Dask to parallelise your code on a single multi-core computer you, you really want to use it distributed mode on a cluster if you are processing large datasets where there is lots of parallelism.
The recommend route for running Dask on Azure in distributed mode is deploying Dask with Kubernetes and Helm.
- Create a learning path for Dask with Kubernetes and Helm on Azure
The following resource was created see Cloud Cloud Dask Deployment
Please follow https://github.com/research-software-reactor/cyclecloud/blob/master/QuickStarts/SettingUpCycleCloud.md to setup Azure Cycle Cloud