fixed: offset replay mechanism #2
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Contributor
devansh42
commented
Aug 5, 2023
devansh42
commented
- previously we were seeking for byte offset but now we'll seek entry offset
- changed wal file naming convention to ease offset replaying
…ffset but now we'll seek for entry offset, also changed wal file naming convention
|
Instead of fixing the byte size of the file, we can set a limit that we will store maximum 1000 lines in one segment. We can name the file with the starting segment. This will help us read from a particular monotonically incrementing counter for the entire wal.( the counter will increment once for every entry into the wal) |
|
The problem, here is the random seeks we need to make to go to exact
offset. With 1000 lines or entries max solution we would have the same
issue that is jumping on a given offset
…On Sun, 6 Aug, 2023, 2:14 pm Aarthik Rao, ***@***.***> wrote:
Instead of fixing the byte size of the file, we can set a limit that we
will store maximum 1000 lines in one segment. We can name the file with the
starting segment. This will help us read from a particular monotonically
incrementing counter for the entire wal.( the counter will increment once
for every entry into the wal)
—
Reply to this email directly, view it on GitHub
<#2 (comment)>, or
unsubscribe
<https://github.com/notifications/unsubscribe-auth/AFADX7VMYPRS55NSH62JHILXT5KQBANCNFSM6AAAAAA3FOYXSI>
.
You are receiving this because you authored the thread.Message ID:
***@***.***>
|
|
aarthikrao
approved these changes
Aug 8, 2023
| func (wal *WriteAheadLog) Replay(offset int64, f func([]byte) error) error { | ||
| logFiles, err := filepath.Glob(wal.logFileName + "*") | ||
| if err != nil { | ||
| return err | ||
| } | ||
|
|
||
| for _, logFile := range logFiles { | ||
| index, startingOffset, err := wal.findStartingLogFile(offset, logFiles) |
There was a problem hiding this comment.
Can we use sort the files and then start reading. It will be simpler.
wal.go
Outdated
| file, err := os.Open(logFile) | ||
| if err != nil { | ||
| return err | ||
| } | ||
| defer file.Close() | ||
|
|
||
| if _, err := file.Seek(offset, io.SeekStart); err != nil { | ||
| if i > 0 { | ||
| if _, err := file.Seek(0, io.SeekStart); err != nil { |
There was a problem hiding this comment.
I'm a little unclear on how this logic works. Can you please add comments.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.