Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

decomposing foldxt #19

Open
ExpandingMan opened this issue Jul 19, 2023 · 0 comments
Open

decomposing foldxt #19

ExpandingMan opened this issue Jul 19, 2023 · 0 comments

Comments

@ExpandingMan
Copy link
Contributor

One may write pseudocode for foldxt and foldxd as something like:

function foldxt(step, xf, xs)
    parts = split_into_parts(xs)
    output = init_output()
    @threads for part  parts
        insert!(output, foldxl(step, xf, part))
    end
    combine(xf, output)
end

This is fairly intuitive, however currently Transducers.jl doesn't really expose any of these intermediate steps, and the internals of it are such that it might not be all that obvious what's going on until close inspection (at least it wasn't to me).

I propose that we create a public API for each of these pieces such that a user could easily write its own foldxt. As I see it, some benefits of this would be:

  • The semantics of foldxt might be somewhat more obvious. This is important because the combine step can cause it to behave quite differently from foldxl.
  • Users could create custom combine. Currently users have to rely on Transducers to provide combine functionality.
  • Users could more easily create custom splitting strategies.

On the other hand, exposing these details could make it significantly easier to write unsafe code, though note that a custom combine needn't rely on parallelism at all.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant