Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[ADD] Interval Join Operator Implementation #20

Open
wants to merge 70 commits into
base: master
Choose a base branch
from
Open

[ADD] Interval Join Operator Implementation #20

wants to merge 70 commits into from

Conversation

DropB1t
Copy link

@DropB1t DropB1t commented Oct 14, 2024

The pull request features full implementation of the Interval Join Operator.

Interval Join Operator Building Methods

  • withBoundaries() sets the lower and upper bounds in milliseconds to compute relative intervals.
  • withKPMode() sets Key Parallelism mode. Each replica will hold and join a subset of keys.
  • withDPSMode() sets Data Parallelism mode. Each replica will hold an exclusive subset of data elements of each key.
  • Default building methods ...

Additional Flags

  • WF_JOIN_MEASUREMENT enables the statistical measurement of how uniformly the tuples are distributed among the replica archives of the Interval Join Operator

DropB1t and others added 30 commits December 11, 2023 11:47
@DropB1t
[ADD] Added new operator - Interval Join ( Key Parallelism )
8066b5a
@DropB1t
[FIX] Fixed nondeterministic behavior of get_MergedNodes1() function
559f5c0
@DropB1t
[FIX] Added a condition to increment the index pointer more securely
d8676c7
@DropB1t
Merge pull request #1 from DropB1t/deterministic-merge-order
c1dd0ae 
Deterministic merge order
@DropB1t
[FIX] Fixed interval evaluation logic
4cea4c4 
Fixed watermark based purge function
@DropB1t
[ADD] Added get/set StreamTag methods to all batch variants
4268f60
@DropB1t
[FIX] Fixed join logic function signature
c7e4b9d
@DropB1t
Merge pull request #2 from DropB1t/optional-result-signature
2b22340 
[FIX] Fixed join logic function signature
@DropB1t
[FIX] Removed GPU's batch JoinStream tag attribute
206057d
@DropB1t
[FIX] Refactored collectors logic
7f67372 
Refactored join buffer purge logic
Join operator can now run in Deterministic & Probabilistic mode
Added purge with punctuations for join operators
@DropB1t
[FIX] Fixed tagging streams in ordering/kslack collectors
962bab9
@DropB1t
[ADD] Added Data Parallelism join execution mode
4b8c9c2
@DropB1t
[ADD] Added synthetic join tests
6c2eb42
@DropB1t
[FIX] Fixed join tests 4 and 5
d617c46
@DropB1t
[ADD] Swappable archive's buffer structure
779492a
@DropB1t
[FIX] Fixed constructor signature
9a89bab
@DropB1t
[FIX] minor fixes
c22c7f2
@DropB1t
Added buffer A stats gathering
5fe135e
@DropB1t
Added buffer A balance measurement
340685f
@DropB1t
Added stats flag macro + buffer stats structure
09a1c3f
@DropB1t
type fix
0cd06f5
@DropB1t
minor
da7e315
@DropB1t
minor fix
81503b7
@DropB1t
cleaned comments
5ecc075
@DropB1t
Merge pull request #3 from DropB1t/join-buffer-stats
c86c50d 
Join Buffers Stats Monitoring
@DropB1t
[FIX] refactored type of tuple wrapper scruct
3b5f357
@DropB1t
[FIX] fixed join archive redundant typename
85dce0a
@DropB1t
join logic refactor
531ceef
@DropB1t
Added interval join tests to regression
536d95e 
Fixed bounds parameters
@DropB1t
Bug fixes, minor refactor, test correctness check fixed
5e7c891
DropB1t and others added 30 commits March 8, 2024 16:08
@DropB1t
Merge branch 'upstream/master' into interval-join
ee60a5a
@DropB1t
Removed deprecated code
82165e0
@DropB1t
Mix refactor
90346b2
@DropB1t
Debug commit
fdbae79
@DropB1t
fixed if condition
214fb2e
@DropB1t
Removed all deprecated debug features
e1ae30e
@DropB1t
bug fix
5032831
@DropB1t
[ADD] join_collector for data parallelism mode in interval join opera…
0508204 
…tors
@DropB1t
Fixed join_collector eosnotify logic
37058fa
@DropB1t
Fixed RR for Single_t tuple dispatching in join_collector.hpp
37c4d4d
@DropB1t
[FIX] New round robin logic implemented
3b612e4
@DropB1t
[FIX] Fixed skewed batch bug (dp mode)
fbe8dc6
@DropB1t
Fixed fnv-1a hashing method, used uint64_t type
f62b7a8
@DropB1t
feat: new fnv1-a hash function
39688f5
@DropB1t
Merge branch 'interval-join' into join-collector
34166ec
@DropB1t
Merge pull request #6 from DropB1t/join-collector
a1ed948 
Join collector
@DropB1t
Fix mean buffer size calculation in printBufferStats
805caee
@DropB1t
[FIX] Enable measurement of tuple distribution in data partitioning f…
fb835bc 
…or interval join operator
@DropB1t
fix: Refactored whole buffer size measurement logic to be more fine-g…
e3ad596 
…rained
@DropB1t
[FIX] Changed boundaries inclusion logic, fixed ij mode comments
495e748
@DropB1t
feat: Refactor archive size measurement logic for interval join
9a8ae98 
Made more coincise functions and structures naming

The commit refactors the archive size measurement logic in the interval join code to provide more fine-grained measurements. This change improves the accuracy of archive size calculations and ensures more precise tracking of archive metrics
@DropB1t
Merge pull request #7 from DropB1t/buffer-metrics-refactor
e02583e 
[ADD] Archive metrics refactor
@DropB1t
[MERGE] 'upstream/master' updates into interval-join
9706a27 
Update headers license URL in all files
Update emitter logic in Interval Join replicas
@DropB1t
[FIX] Fixed join tests files, added interval_join.hpp as doxygen inpu…
9e42d18 
…t files
@DropB1t
[FIX] Redesigned comments
671a6a2
@DropB1t
[ADD] Added size in KB for mean archive size
ad634c8
@DropB1t
[FIX] Added purgeArchives procedure in case of Deterministic mode exe…
474801e 
…cution
@DropB1t
[FIX] Fixed .gitignore
e35eb42
@DropB1t
[FIX] Remove WF_JOIN_MEASUREMENT macro from CMakeLists
c9e657e
@DropB1t
[FIX] incorrect compare function type in assert evaluation in Win Arc…
b3cc8b7 
…hive
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@DropB1t