Better parallelization using Dask?

[jgostick]

We've had some great success with dask for simple single-machine parallelization. I think the current version uses something a bit problematic right, requiring the script to be in a main section right? Dask does not require any of this.

[TomTranter]

Yup, can you post an example? I also started looking at shared memory arrays for something else but only works in linux

[jgostick]

We've been using it to do operations in chunks, like this:

import numpy as np
from dask import delayed, compute

# This decorator tells dask to delay computation of this function
@delayed
def sumnums(arr, num):
    arr = arr + num
    return arr


arr = np.zeros([20, 20])
res = []
for chunk in np.split(arr, 2):
    # This loop creates a list of 'delayed' functions
    res.append(sumnums(chunk, 1))
temp = compute(res)  # Here we tell dask to actually do the calc, in parallel
new_arr = np.vstack(temp[0])  # Now we massage the result back into an array
assert np.all(new_arr.shape == arr.shape)