DAPHNE 0.3

Hereby we release the third development snapshot of DAPHNE containing many improvements and bug fixes that have been contributed by members of the DAPHNE consortium as well as external contributors (mentioned below). Find some of the more notable (bigger) changes below in the detailed release notes. Besides the artifact downloads on this page and the Docker images, we are releasing the python bindings daphne-lib on PyPI

Release Highlights

• DaphneDSL/DaphneLib:
  • Additional ready-to-use data science algorithms: decision trees, random forests, PageRank
  • Various DaphneDSL language improvements (e.g., UDFs with multiple return values, more built-in functions, bounds-checks for left/right indexing, ...)
  • Support for complex control flow in DaphneLib
  • DaphneLib as a Python package
  • Efficient data exchange with pandas, TensorFlow, PyTorch
• DAPHNE Compiler:
  • Initial MLIR-based codegen pipeline
  • Introduction of a kernel catalog to make the compiler aware of available pre-compiled kernels
• DAPHNE Runtime:
  • CUDA 12.2.2
  • CUDA-support for more kernels
  • Support for synchronous gRPC and chunked data transfer in distributed runtime
  • AUTO option to the DAPHNE’s scheduling algorithms
• Infrastructure and general:
  • Initial extensibility for custom kernels
  • More consistent and actionable error messages
  • Numerous little improvements and bug fixes

Contributors to this release

• AlexRTer
• Aristotelis Vontzalidis
• Benjamin Steinwender
• Constantin Pestka
• DamianDinoiu
• Daniel Wetzel
• Garic
• Henri Willems
• Jonas Henrique Muller Korndorfer
• Lachezar Nikolov
• Lorenz Dirry
• Marcus Paradies
• Marius Birkenbach
• Mark Dokter
• Patrick Damme
• Philipp Ortner
• Quentin Guilloteau
• Samin
• StoeckOverflow
• Stratos Psomadakis
• Tom Schwarzburg
• inikokali

Instructions to download and verify

# The signing keys
wget -qO- https://raw.githubusercontent.com/daphne-eu/daphne/main/KEYS.txt | gpg --import

# The plain DAPHNE release (ARMv8 aka 64 bit ARM)
wget https://github.com/daphne-eu/daphne/releases/download/0.3/daphne-ARMV8-0.3-bin.tgz.sha512sum
wget https://github.com/daphne-eu/daphne/releases/download/0.3/daphne-ARMV8-0.3-bin.tgz.asc
wget https://github.com/daphne-eu/daphne/releases/download/0.3/daphne-ARMV8-0.3-bin.tgz
sha512sum -c daphne-ARMV8-0.3-bin.tgz.sha512sum
gpg --verify daphne-ARMV8-0.3-bin.tgz.asc
tar xf daphne-ARMV8-0.3-bin.tgz
# add lib directory to environment
export LD_LIBRARY_PATH=$PWD/daphne-ARMV8-0.3-bin/lib:$LD_LIBRARY_PATH

# The plain DAPHNE release (X86-64)
wget https://github.com/daphne-eu/daphne/releases/download/0.3/daphne-X86-64-0.3-bin.tgz.sha512sum
wget https://github.com/daphne-eu/daphne/releases/download/0.3/daphne-X86-64-0.3-bin.tgz.asc
wget https://github.com/daphne-eu/daphne/releases/download/0.3/daphne-X86-64-0.3-bin.tgz
sha512sum -c daphne-X86-64-0.3-bin.tgz.sha512sum
gpg --verify daphne-X86-64-0.3-bin.tgz.asc
tar xf daphne-X86-64-0.3-bin.tgz
# add lib directory to environment
export LD_LIBRARY_PATH=$PWD/daphne-X86-64-0.3-bin/lib:$LD_LIBRARY_PATH

# The DAPHNE release with CUDA support (X86-64)
wget https://github.com/daphne-eu/daphne/releases/download/0.3/daphne-cuda-X86-64-0.3-bin.tgz.sha512sum
wget https://github.com/daphne-eu/daphne/releases/download/0.3/daphne-cuda-X86-64-0.3-bin.tgz.asc
wget https://github.com/daphne-eu/daphne/releases/download/0.3/daphne-cuda-X86-64-0.3-bin.tgz
sha512sum -c daphne-cuda-X86-64-0.3-bin.tgz.sha512sum
gpg --verify daphne-cuda-X86-64-0.3-bin.tgz.asc
tar xf daphne-cuda-X86-64-0.3-bin.tgz
# add lib directory to environment
export LD_LIBRARY_PATH=$PWD/daphne-cuda-X86-64-0.3-bin/lib:$LD_LIBRARY_PATH

What's Changed

• Fix erroneous special treatment of default logger by @MarcusParadies in #594
• Cc cuda ops by @corepointer in #596
• Misc improvements by @corepointer in #604
• Fix passing str variable to readFrame/readMatrix by @MarcusParadies in #598
• [DAPHNE-#554] Use hwloc to query CPU topology by @psomas in #563
• [doc] add more docs regarding script arguments by @m-birke in #620
• [DAPHNE-#606] Some INT/FLOAT literal suffixes from C++ in DaphneDSL by @corepointer in #606
• 522 DaphneLib complex control flow and lazy evaluated functions by @lachezar-n in #574
• 615-Constant_folding_EwlogOp by @inikokali in #630
• [MINOR] Enable shape inference for frame type by @DamianDinoiu in #626
• [MINOR] Parsing IR fix. by @aristotelis96 in #634
• MLIR-based code generation pipeline by @philipportner in #633
• SQLParser: Introduce DISTINCT clause by @tomschw in #564
• SQLParser: Support for SELECT , SELECT f., GROUP BY , GROUP BY f. by @tomschw in #588
• 613-Support missing aggregation functions in aggregation kernels by @inikokali in #638
• Integration of mpi_testing into test.sh by @inikokali in #639
• Close #614 Support missing elementwise unary functions in elementwise unary kernels by @AlexRTer in #644
• [doc] fix typo in links within Readme by @AlexRTer in #645
• Automated task grain size calculation (related to issue 567). by @jhmkorndorfer in #647
• #650 fix get topology by @GuilloteauQ in #651
• [DAPHNE-#572] Bounds checks for left and right indexing by @AlexRTer in #648
• Distributed worker: Insert IR arguments to body. by @aristotelis96 in #623
• [DAPHNE-#669] properly handle views in WriteCsv kernel by @AlexRTer in #678
• Feature/daphnelib python pkg by @m-birke in #649
• Pandas, Pytorch & Tensorflow Shared Memory Support (Issue #499) by @danielwetzel in #585
• [DAPHNE-#687] OneHot bound check by @AlexRTer in #695
• Code Generation for Matrix Multiplication by @resting-dove in #653
• Improve error handling during compilation + runtime by @philipportner in #706
• Basic extension catalog for kernels, plus demo on early extensibility. by @pdamme in #456
• Generic Matrix kernel specializations by @AlexRTer in #722
• [BUGFIX] Build target dependencies. by @pdamme in #728
• [DAPHNE-#531] Simplified GettingStarted.md. by @pdamme in #725
• [DAPHNE-#659] Scientific notation for float literals and digit separators by @AlexRTer in #677
• Tensor Support - 1: Data structure by @CPestka in #694
• Non-deterministic test fail during Github CI action by @philipportner in #733
• Enable MathJax for documentation by @AlexRTer in #726
• [DAPHNE-#747] Use test.sh in CI again. by @pdamme in #748
• [DAPHNE-#523] DaphneDSL flexible matrix/frame literals by @AlexRTer in #675
• daphnelib: improve pyproject.toml and readme by @m-birke in #743
• Update docs.yml by @auge in #750
• Update main.yml by @auge in #751
• Add missing (local) kernel tests by @AlexRTer in #730
• [DAPHNE-#658] stop() built-in/op/kernel by @Garic152 in #753
• [DAPHNE-#666] optional inc for seq function by @saminbassiri in #756
• mac support ongoing by @auge in #764
• 752 one hot and bin functions in daphne lib by @saminbassiri in #760
• [DAPHNE-#680] bug fix for signed integer casts by @AlexRTer in #744
• [security] Remove polyfill.io dependency in docs by @AlexRTer in #769
• #720 replace asserts in codebase with exceptions by @AlexRTer in #732
• Add AggAll kernels for missing value types by @resting-dove in #674
• Address Wrong results using Integer Matmul Kernels after Matrix resizing by @resting-dove in #701
• Refactor getLine() to avoid potential memory leak by @psomas in #770
• [DAPHNE-#560] Parsing unary minus operator by @corepointer in #607
• [DAPHNE-#595] Fix crash when calling print(f()) with empty function by @corepointer in #605
• [DAPHNE-#768] isNan: elementwise check for NaN elements ...

DAPHNE 0.3-rc1

Second release candidate for v0.3 containing a number of bug fixes and improvements.
This time making use of the "generate release notes" feature.
Please test and report (in the form of GitHub issues)

What's Changed

• 657 DaphneLib implement indexing by @saminbassiri in #771
• Implementation of new SparsityOp by @Garic152 in #790
• Improve performance of the transposition of CSR matrices by @GuilloteauQ in #798
• 788 closing mem leaks by @corepointer in #791
• [DAPHNE-#774] Simplification rewrites for elementwise unary minus (ad… by @ldirry in #796
• [DAPHNE-#775] Support for unary minus in DaphneLib by @ldirry in #792
• [DAPHNE-#755] Initial support for lists in DaphneDSL. by @pdamme in #806
• Implementation of CSR/CSR Matrix Multiplication (@) by @GuilloteauQ in #799
• [DAPHNE-#773] Undetected invalid script args by @ldirry in #800

New Contributors

• @ldirry made their first contribution in #796

Full Changelog: 0.3-rc0...0.3-rc1

DAPHNE 0.3-rc0

The first release candidate of version 0.3. Please test and report.

DAPHNE v0.2

Hereby we release the second development snapshot of DAPHNE containing many improvements and bug fixes that have been contributed by members of the DAPHNE consortium as well as external contributors (mentioned below). Find some of the more notable (bigger) changes below in the detailed release notes. Besides the artifact downloads on this page, new Docker images are available on dockerhub.

Release Highlights

• DaphneDSL: various little additions and improvements
  • second-order function map()
  • elementwise conditional operator
• system internals
  • initial OpenMPI backend for distributed runtime
  • some more FPGA kernels
  • improvements to the vectorized engine
  • logging facility based on spdlog
  • initial profiling features
• build and deployment: various improvements
  • containers for simple deployment
  • refactoring of the build scripts
  • continuous integration
• external dependencies
  • libeigen is used for calculating Eigen values/vectors and for integer matrix multiply
  • libpapi for profiling
  • OpenMPI for distributed operation
  • hwloc for detecting processor details
  • spdlog for loggin
• miscellaneous changes
  • based on a new snapshot of LLVM/MLIR
  • CUDA version 12.1.1
  • support for 64 bit ARM (CPU only)

Contributors

This is an alphabetically sorted list (based on git history) of people who contributed to this release:

• Ahmed Eleliemy
• Aleš Zamuda
• Aristotelis Vontzalidis
• Benjamin Steinwender
• Damian Dinoiu
• Dzevad Coralic
• Eric Mier
• Kostas Bitsakos
• Marius Birkenbach
• Mark Dokter
• Patrick Damme
• Piotr Ratuszniak
• Simeon
• Stratos Psomadakis
• Tom Schwarzburg

What's Changed

• [DAHPNE-#198] Second order function map by @sigmaeon in #407
• Fpga operators by @ratusz in #451
• 459: enable CI by @auge in #460
• Upgraded MLIR. by @pdamme in #468
• Cached GitHub docker action by @corepointer in #470
• Build script improvements by @corepointer in #471
• Docker improvements by @corepointer in #472
• Draft: [DAPHNE-#132] Frame column labels after group-join by @DamianDinoiu in #478
• Duplicate inputs in pipeline input array. by @aristotelis96 in #476
• Stripping et al by @corepointer in #490
• Docker cuda by @corepointer in #491
• [DAPHNE-#492] Creating Subframe of Frame with zero rows by @tomschw in #496
• Metadata sharing gpu fixes by @corepointer in #503
• Fix: Force set the cached variable ANTLR4_JAR_LOCATION, if old path not exists by @EricMier in #506
• Added load partioning for distributed runtime. by @aristotelis96 in #489
• DaphneLib Frames implementation by @daphne-eu in #318
• Eigen op 391 by @KostasBitsakos in #463
• CUDA fill by @daphne-eu in #313
• Create MPI-Usage.md by @AhmedEleliemy in #532
• [DAPHNE-#427] Initial logging facility using spdlog by @corepointer in #535
• [DAPHNE-#530] Docker Improvements by @corepointer in #534
• Add profiling support by @psomas in #479
• Dev-issue-507-dependency-path-changes by @EricMier in #508
• Eigen op 391 by @KostasBitsakos in #539
• [DAPHNE-#493] SQLParser: Run Aggregation without grouping by @tomschw in #509
• [DAPHNE-#540] Support for the ARMV8 CPU architecture by @corepointer in #540
• [DAPHNE-#399] Aggregation kernels can return different value types by @aristotelis96 in #402
• Initial implementation of DaphneSerializer by @aristotelis96 in #467
• [DAPHNE-#494] SQLParser: Group by without aggregation clause not working by @tomschw in #543
• Logger fixes by @corepointer in #547
• Integer matrix multiplication #352 by @KostasBitsakos in #384
• Feature/docs mvp by @m-birke in #550
• [DOC] Improvements to quickstart guide. by @pdamme in #566
• Pass errors are not only logged, but also printed again. by @pdamme in #571
• [doc] fix md flaws and add newely added docs to html page by @m-birke in #570
• [DAPHNE-#575] Error logger, meaningful default log level, cleanup by @corepointer in #575

New Contributors

• @sigmaeon made their first contribution in #407
• @auge made their first contribution in #460
• @DamianDinoiu made their first contribution in #478
• @m-birke made their first contribution in #550

Full Changelog: 0.1...0.2

Instructions to download and verify

# The signing keys
wget -qO- https://raw.githubusercontent.com/daphne-eu/daphne/main/KEYS.txt | gpg --import

# The plain DAPHNE X86-64 release
wget https://github.com/daphne-eu/daphne/releases/download/0.2/daphne-X86-64-v0.2-bin.tgz.sha512sum
wget https://github.com/daphne-eu/daphne/releases/download/0.2/daphne-X86-64-v0.2-bin.tgz.asc
wget https://github.com/daphne-eu/daphne/releases/download/0.2/daphne-X86-64-v0.2-bin.tgz
sha512sum -c daphne-X86-64-v0.2-bin.tgz.sha512sum
gpg --verify daphne-X86-64-v0.2-bin.tgz.asc
tar xf daphne-X86-64-v0.2-bin.tgz
# add lib directory to environment
export LD_LIBRARY_PATH=$PWD/daphne-X86-64-v0.2-bin/lib:$LD_LIBRARY_PATH

# The DAPHNE release with CUDA support
wget https://github.com/daphne-eu/daphne/releases/download/0.2/daphne-cuda-X86-64-v0.2-bin.tgz.sha512sum
wget https://github.com/daphne-eu/daphne/releases/download/0.2/daphne-cuda-X86-64-v0.2-bin.tgz.asc
wget https://github.com/daphne-eu/daphne/releases/download/0.2/daphne-cuda-X86-64-v0.2-bin.tgz
sha512sum -c daphne-cuda-X86-64-v0.2-bin.tgz.sha512sum
gpg --verify daphne-cuda-X86-64-v0.2-bin.tgz.asc
tar xf daphne-cuda-X86-64-v0.2-bin.tgz
# add lib directory to environment
export LD_LIBRARY_PATH=$PWD/daphne-cuda-X86-64-v0.2-bin/lib:$LD_LIBRARY_PATH

# The DAPHNE release with ARM support
wget https://github.com/daphne-eu/daphne/releases/download/0.2/daphne-ARMV8-v0.2-bin.tgz.sha512sum
wget https://github.com/daphne-eu/daphne/releases/download/0.2/daphne-ARMV8-v0.2-bin.tgz.asc
wget https://github.com/daphne-eu/daphne/releases/download/0.2/daphne-ARMV8-v0.2-bin.tgz
sha512sum -c daphne-ARMV8-v0.2-bin.tgz.sha512sum
gpg --verify daphne-ARMV8-v0.2-bin.tgz.asc
tar xf daphne-ARMV8-v0.2-bin.tgz
# add lib directory to environment
export LD_LIBRARY_PATH=$PWD/daphne-ARMV8-v0.2-bin/lib:$LD_LIBRARY_PATH

Release 0.1

Initial Release of DAPHNE

This first release contains basic initial implementations of most components of the DAPHNE system infrastructure, from a domain-specific language (DaphneDSL) as the main user frontend so far (a Python API is under development), over an MLIR-based intermediate representation (DaphneIR) and compilation chain to runtime components for the distributed execution using a stand-alone backend based on gRPC and the local, optionally vectorized, execution on CPU, GPU, and FPGA, for a certain subset of operations and data/value types.

Release Notes:

• The focus of this release is on providing an early preview of the DAPHNE system prototype for interested researchers/developers/users.
• We are aware of several bugs and issues. Note also that several features of the system are still experimental.
• Likewise, there is still a lot of room for performance improvements in various components.
• The binary release contains precompiled DAPHNE libraries, executables, scripts, and documentation.
• Binaries were generated on Ubuntu 20.04 LTS on an Intel Xeon machine compiled for x86_64.
• GPU operations were compiled against CUDA 11.7.1.
• As this is the first release, the auto-generated full changelog contains all commits to date.
• Experimental features (FPGA, Arrow) which are not available in the binary release can be compiled from source. Follow the instructions in the documentation section.
• To avoid problems with library dependencies, two binary artifacts are provided for this release. One compiled with CUDA support and one without.

Contributors:

This is an alphabetically sorted list (based on git history) of people who contributed to this release:

• Ahmed Eleliemy
• Alexander Hiebl
• Aleš Zamuda
• Aristotelis Vontzalidis
• Artem Kroviakov
• Constantin Pestka
• Dominic Schablas
• Dževad Ćoralić
• Eric Mier
• Gabrielle Poerwawinata
• Izajasz Wrosz
• Jonathan Giger
• Kevin Innerebner
• Konstantinos Bitsakos
• Lennart Schmidt
• Mark Dokter
• Matej Moravec
• Matthias Boehm
• Morten Tychsen Clausen
• Niclas Hedam
• Patrick Damme
• Philipp Ortner
• Philippe Bonnet
• Piotr Ratuszniak
• Stratos Psomadakis

Instructions to download and verify

# The signing keys
wget -qO- https://raw.githubusercontent.com/daphne-eu/daphne/main/KEYS.txt | gpg --import
# The plain DAPHNE release
wget https://github.com/daphne-eu/daphne/releases/download/0.1/daphne-0.1-bin.tgz.sha512sum
wget https://github.com/daphne-eu/daphne/releases/download/0.1/daphne-0.1-bin.tgz.asc
wget https://github.com/daphne-eu/daphne/releases/download/0.1/daphne-0.1-bin.tgz
sha512sum -c daphne-0.1-bin.tgz.sha512sum
gpg --verify daphne-0.1-bin.tgz.asc
tar xf daphne-0.1-bin.tgz
# add lib directory to environment
export LD_LIBRARY_PATH=$PWD/daphne-0.1-bin/lib:$LD_LIBRARY_PATH
# The DAPHNE release with CUDA support
wget https://github.com/daphne-eu/daphne/releases/download/0.1/daphne--cuda-0.1-bin.tgz.sha512sum
wget https://github.com/daphne-eu/daphne/releases/download/0.1/daphne--cuda-0.1-bin.tgz.asc
wget https://github.com/daphne-eu/daphne/releases/download/0.1/daphne--cuda-0.1-bin.tgz
sha512sum -c daphne--cuda-0.1-bin.tgz.sha512sum
gpg --verify daphne--cuda-0.1-bin.tgz.asc
tar xf daphne--cuda-0.1-bin.tgz
# add lib directory to environment
export LD_LIBRARY_PATH=$PWD/daphne--cuda-0.1-bin/lib:$LD_LIBRARY_PATH

What's Changed

• [DAPHNE-#216] ANTLR not found while building by @daphne-eu in #332
• [DAPHNE-#128] Kernel for CastOp from DenseMatrix of one value type to DenseMatrix of another value type. by @akroviakov in #337
• Added reading the prototype configuration from JSON file by @matejmoravec in #336
• [DAPHNE-340] Fix FISS chunk size calculation by @jonnygiger in #341
• 236 copying deploy files to merge by @aleszamuda in #335
• New IO kernels: Matrix Market, Parquet and DAPHNE Binary by @daphne-eu in #331
• [DAPHNE-#234] extractCol-kernel for Frame with positions in DenseMatr… by @Hiebl in #338
• [DAPHNE-#217]: rowBind-kernel for CSRMatrix by @akroviakov in #344
• [DAPHNE-#218]: colBind-kernel for CSRMatrix by @akroviakov in #347
• [DAPHNE-#224]: reshape-kernel for non-zero-copy cases by @akroviakov in #348
• [DAPHNE-#215]: Ternary/conditional operator ?: in DaphneDSL by @akroviakov in #349
• [DAPHNE-#31] Kernel for diagMatrix-operation by @psomas in #362
• [DAPHNE-#350]: DaphneDSL: rbind checks number of rows instead the number of columns by @aristotelis96 in #360
• [DAPHNE-#356]: type casting of DenseMatrix by @aristotelis96 in #361
• Support for mean operation for all-aggregation and row-aggregation. by @aristotelis96 in #398
• [DAPHNE-#358]: DaphneDSL: Matrix literals by @akroviakov in #387
• [DAPHNE-#390]: Kernel for ctable operation by @akroviakov in #392
• Draft: [DAPHNE-#242] Kernel for GroupOp on Frame by @Hiebl in #363
• [DAPHNE-#403]: CLI args for explaining/printing the IR at various points by @philipportner in #404
• CSRMatrix to Dense Matrix CastOpt and vise versa by @KostasBitsakos in #366
• Enable Arrow by flag + arrow docs by @niclashedam in #345
• [DAPHNE-#230]: String as value type of DenseMatrix by @akroviakov in #381
• [DAPHNE-#388] order-kernel for DenseMatrix by @Hiebl in #400
• [DAPHNE-#389] order-kernel: return indexes instead of ordered data object by @Hiebl in #411
• [DAPHNE-196] Multiple task queues and work stealing by @jonnygiger in #342
• [DAPHNE-191] Refactor CUDA buffer mgmt (aka introducing Object Meta D… by @corepointer in #334
• [DAPHNE-#420]: ANTLR Any Conversion. by @DerSchmidt in #422
• [DAPHNE-#200] Transposition-aware matrix multiplication by @Hiebl in #406
• Distributed pipelines by @aristotelis96 in #428
• [DAPHNE-#419] Guidelines on handling pull requests by @pdamme in #430
• Fpga operators by @ratusz in #416
• [DAPHNE-#214]: Import mechanism in DaphneDSL by @akroviakov in #410
• [DAPHNE-#55]: Casts in DaphneDSL by @akroviakov in #408
• [DAPHNE-#417] Incorrect GROUP BY result when all keys are the same by @Hiebl in #426
• [DAPHNE-#385] Connected JSON-based file meta data parser to file IO by @matejmoravec in #394
• Documentation of scheduling options and extensions by @AhmedEleliemy in #435
• Integration of ThetaJoin and OrderBy kernels into SQL Parser + Lowering by @EricMier in #440
• [DAPHNE-#118] Initial write-up of the DaphneDSL documentation. by @pdamme in #443
• Distributed runtime documentation. by @aristotelis96 in #431
• [DAPHNE-#437] Deployment documentation (containerized packaging, virtualized installation, managed deployment, and distributed execution) by @aleszamuda in #441
• SQL tutorial by @DerSchmidt in #442
• Release scripts by @corepointer in #444
• [DAPHNE-#452] Release script feature parameter by @corepointer in #452

New Contributors

• @daphne-eu made their first contribution in #332
• @akroviakov made their first contribution in #337
• @matejmoravec made their first contribution in #336
• @jonnygiger made their first contribution in #341
• @aleszamuda made their first contribution in #335
• @Hiebl made their first contribution in #338
• @psomas made their first contribution in #362
• @aristotelis96 made their first contribution in #360
• @philipportner made their first contribution in #404
• @KostasBitsakos made their first contribution in #366
• @niclashedam made their first contribution in #345
• @corepointer made their first contribution in #334
• @DerSchmidt made their first contribution in #422
• @pdamme made their first contribution in #430
• @ratusz made their first contribution in #416
• @AhmedEleliemy made their fir...