Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[DAPHNE-#525] DaphneDSL map(): UDFs in Other Languages (e.g., Python, C++) #608

Open
wants to merge 107 commits into
base: main
Choose a base branch
from

Conversation

StoeckOverflow
Copy link
Contributor

This PR introduces a new kernel for invoking map kernels from other programming languages and provides methods for the data transfer between DAPHNE and Python (see Issue #525 ).

Features:

  • New kernel for invoking map kernels of other programming languages.
  • Four Python map kernels offering different data transfer methods: Binary file, CSV file, Copy, Shared Memory.

Details:

  • New Kernels:
    - Kernel for invoking map kernels: src/runtime/local/kernels/MapExternalPL.h
    - Four Python map kernels: src/runtime/local/kernels/MAP_EXTERNAL/..
  • Python Interpreter Singleton:
    - Integrated a global PythonInterpreter singleton class to manage Python interactions: src/util/PythonInterpreter
  • Tests
    - Basic functionality tests for different map kernels: test/runtime/local/kernels/AlternativeMapTest
  • Scripts:
    - General use-case scripts:
    scripts/examples/map/map-GeneralExample.daph
    scripts/examples/map/mapExternalPL-UtilityExample.daph
    - Performance testing scripts: scripts/examples/map/performanceTest/..
  • IR Extensions:
    - New operation (MapOpExternal) for Lowering of new Kernel: src/ir/daphneir/DaphneOps.td
  • Documentation Updates:
    - New map entry: doc/DaphneDSL/Builtins.md
    - Guidance for extending the map kernel: doc/developement/ImplementBuiltinKernel.md

@pdamme pdamme self-requested a review September 11, 2023 18:14
@pdamme
Copy link
Collaborator

pdamme commented Sep 11, 2023

Thanks for this contribution @StoeckOverflow! Enabling DaphneDSL map() with UDFs written in other programming languages, such as Python, is a very interesting feature that can be used to do value-based processing beyond what DaphneDSL is currently able to do.

The two of us have already discussed this matter and I will provide some feedback on your code soon.

@pdamme pdamme added the LDE summer 2023 Student project in the course Large-scale Data Engineering at TU Berlin (summer 2023). label Sep 13, 2023
@StoeckOverflow StoeckOverflow changed the title DaphneDSL map(): UDFs in Other Languages (e.g., Python, C++) #525 [DAPHNE-#525] DaphneDSL map(): UDFs in Other Languages (e.g., Python, C++) Sep 17, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
LDE summer 2023 Student project in the course Large-scale Data Engineering at TU Berlin (summer 2023).
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants