Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support Multiprocessing #189

Open
nreinicke opened this issue Jul 22, 2024 · 2 comments
Open

Support Multiprocessing #189

nreinicke opened this issue Jul 22, 2024 · 2 comments

Comments

@nreinicke
Copy link
Collaborator

As discovered in #187, pickling the road map rtree is currently failing which disables us from leveraging multiprocessing for a batch of traces. Until the pickling issue is resolved we could explore workarounds to allow traces to be processed in parallel. Road maps could be very large and so a solution that allows the road network data to be shared would be ideal.

@nreinicke
Copy link
Collaborator Author

@jhoshiko - I seem to recall that you've run into a similar issue in the past. Do you have any notes from trying to map match large road networks in parallel that might be useful in this context?

@jhoshiko
Copy link
Collaborator

@nreinicke In the past, I would initialize a copy of the road network on each worker, scaling the number of workers based on available memory and the size of the road network. I haven't personally experimented with matching to a road network in shared memory, but I agree that it would be ideal. I think multiprocessing/joblib might have options for instantiating objects into shared memory, but I haven't personally tested anything.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants