Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ZSTD_compressSequencesAndLiterals #4217

Open
wants to merge 67 commits into
base: dev
Choose a base branch
from
Open

ZSTD_compressSequencesAndLiterals #4217

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
67 commits
Select commit Hold shift + click to select a range
125f052
publish new symbol ZSTD_compressSequencesAndLiterals()
Cyan4973 Dec 10, 2024
a00f45a
created ZSTD_storeSeqOnly()
Cyan4973 Dec 10, 2024
b4a40a8
move Sequences definition to zstd_compress_internal.h
Cyan4973 Dec 10, 2024
9671813
codemod: seqDef -> SeqDef
Cyan4973 Dec 10, 2024
a224572
codemod: seqStore_t -> SeqStore_t
Cyan4973 Dec 10, 2024
8d4506b
codemod: ZSTD_sequenceLength -> ZSTD_SequenceLength
Cyan4973 Dec 10, 2024
477a010
codemod: symbolEncodingType_e -> SymbolEncodingType_e
Cyan4973 Dec 10, 2024
0442e43
codemod: ZSTD_defaultPolicy_e -> ZSTD_DefaultPolicy_e
Cyan4973 Dec 10, 2024
e9f8a11
ZSTD_entropyCompressSeqStore_internal() can accept an externally defi…
Cyan4973 Dec 10, 2024
0165eeb
created ZSTD_entropyCompressSeqStore_wExtLitBuffer()
Cyan4973 Dec 10, 2024
c97522f
codemod: ZSTD_sequenceFormat_e -> ZSTD_SequenceFormat_e
Cyan4973 Dec 10, 2024
894ea31
codemod: ZSTD_sequenceCopier -> ZSTD_SequenceCopier_f
Cyan4973 Dec 10, 2024
1ac79ba
minor: simplify ZSTD_selectSequenceCopier
Cyan4973 Dec 10, 2024
76dd3a9
scope: ZSTD_copySequencesToSeqStore*() are private to ZSTD_compress.c
Cyan4973 Dec 10, 2024
03d95f9
fix proper type for .forceNonContiguous
Cyan4973 Dec 10, 2024
5359d16
enable proper type
Cyan4973 Dec 10, 2024
30671d7
codemod: ZSTD_sequencePosition -> ZSTD_SequencePosition
Cyan4973 Dec 10, 2024
fa46894
codemod: ZSTD_buildSeqStore_e -> ZSTD_BuildSeqStore_e
Cyan4973 Dec 10, 2024
5df80ac
codemod: ZSTD_matchState_t -> ZSTD_MatchState_t
Cyan4973 Dec 10, 2024
41c667c
codemod: repcodes_t -> Repcodes_t
Cyan4973 Dec 11, 2024
25bef24
codemod: rawSeqStore_t -> RawSeqStore_t
Cyan4973 Dec 11, 2024
08edecb
codemod: ZSTD_blockCompressor -> ZSTD_BlockCompressor_f
Cyan4973 Dec 11, 2024
e0f3aae
doc: add mention of frame checksum incompatibility
Cyan4973 Dec 11, 2024
2503b64
fix minor artifact error in single_file_lib
Cyan4973 Dec 11, 2024
13b9296
minor simplification
Cyan4973 Dec 11, 2024
56cfb78
codemod: ZSTD_paramSwitch_e -> ZSTD_ParamSwitch_e
Cyan4973 Dec 11, 2024
4ef9d7d
codemod: ZSTD_cParamMode_e -> ZSTD_CParamMode_e
Cyan4973 Dec 11, 2024
047db4f
ZSTD_SequenceCopier_f no returns the nb of bytes consumed from input
Cyan4973 Dec 11, 2024
bcb1509
minor: more accurate variable scope
Cyan4973 Dec 11, 2024
14a21e4
produced ZSTD_compressSequencesAndLiterals() as a separate pipeline
Cyan4973 Dec 11, 2024
0b013b2
added unit tests to ZSTD_compressSequencesAndLiterals()
Cyan4973 Dec 12, 2024
09964c6
prepare fullbench for a refactor
Cyan4973 Dec 13, 2024
4c097b4
refactor fullbench
Cyan4973 Dec 13, 2024
c540976
minor: test reordering
Cyan4973 Dec 13, 2024
8b7e1b7
minor variable renaming
Cyan4973 Dec 13, 2024
c050ae4
fullbench: preparation functions are now in charge of allocating buffers
Cyan4973 Dec 14, 2024
8ab0409
add the compressSequences() benchmark scenario
Cyan4973 Dec 14, 2024
ac05ea8
fullbench: switch default generator to lorem ipsum
Cyan4973 Dec 14, 2024
f281497
fullbench: new scenario: compressSequencesAndLiterals()
Cyan4973 Dec 15, 2024
0a5c080
minor conversion warning fix
Cyan4973 Dec 15, 2024
1c8f5b0
minor optimization for ZSTD_compressSequencesAndLiterals()
Cyan4973 Dec 16, 2024
f176514
minor doc update
Cyan4973 Dec 16, 2024
a288751
minor optimization: only track seqPos->posInSrc when validateSequence…
Cyan4973 Dec 16, 2024
1f6d681
optimization: instantiate specialized version without Sequence checki…
Cyan4973 Dec 16, 2024
d2d0fda
updated documentation on validateSequence
Cyan4973 Dec 16, 2024
ca8bd83
minor: cleaner function parameter repcodeResolution
Cyan4973 Dec 16, 2024
5164d44
change advanced parameter name: ZSTD_c_repcodeResolution
Cyan4973 Dec 16, 2024
31b5ef2
ZSTD_compressSequencesAndLiterals() now supports multi-blocks frames.
Cyan4973 Dec 17, 2024
f0d0d95
added tests
Cyan4973 Dec 17, 2024
6f8c104
update Visual Studio solutions
Cyan4973 Dec 17, 2024
47edd0a
removed fullbench-dll project from visual solutions
Cyan4973 Dec 17, 2024
f617e86
fixed incorrect assert
Cyan4973 Dec 17, 2024
61ac831
attempt to silence Visual Studio warning about fopen()
Cyan4973 Dec 17, 2024
d48e330
change name to ZSTD_convertSequences*()
Cyan4973 Dec 17, 2024
95ad9e4
added benchmark for ZSTD_convertBlockSequences_wBlockDelim()
Cyan4973 Dec 18, 2024
12c47d3
improved speed of the Sequences converter
Cyan4973 Dec 18, 2024
b7b4e86
fixed minor conversion warning
Cyan4973 Dec 18, 2024
ad023b3
fixed minor error in one benchmark scenario
Cyan4973 Dec 18, 2024
0a54f6f
ZSTD_compressSequencesAndLiterals requires srcSize as parameter
Cyan4973 Dec 19, 2024
a80f55f
added a test for ZSTD_compressSequencesAndLiterals
Cyan4973 Dec 19, 2024
b339eff
add dedicated error code for special case
Cyan4973 Dec 19, 2024
ab0f179
ensure that srcSize is controlled
Cyan4973 Dec 19, 2024
52a9bc6
fixed minor error in preparation of one fullbench scenario
Cyan4973 Dec 19, 2024
76445bb
add a check, to return an error if Sequence validation is enabled
Cyan4973 Dec 19, 2024
b7a9e69
added parameter litCapacity
Cyan4973 Dec 19, 2024
522adc3
minor: use MEM_writeLE24()
Cyan4973 Dec 20, 2024
47cbfc8
restore invocation of ZSTD_entropyCompressSeqStore()
Cyan4973 Dec 20, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
189 changes: 0 additions & 189 deletions build/VS2010/fullbench-dll/fullbench-dll.vcxproj
View file
Open in desktop

This file was deleted.

1 change: 1 addition & 0 deletions build/VS2010/fullbench/fullbench.vcxproj
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -184,6 +184,7 @@
<ClCompile Include="..\..\..\programs\util.c" />
<ClCompile Include="..\..\..\programs\timefn.c" />
<ClCompile Include="..\..\..\programs\datagen.c" />
<ClCompile Include="..\..\..\programs\lorem.c" />
<ClCompile Include="..\..\..\programs\benchfn.c" />
<ClCompile Include="..\..\..\tests\fullbench.c" />
</ItemGroup>
Expand Down
5 changes: 0 additions & 5 deletions build/VS2010/zstd.sln
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,11 +7,6 @@ Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "fuzzer", "fuzzer\fuzzer.vcx
EndProject
Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "fullbench", "fullbench\fullbench.vcxproj", "{61ABD629-1CC8-4FD7-9281-6B8DBB9D3DF8}"
EndProject
Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "fullbench-dll", "fullbench-dll\fullbench-dll.vcxproj", "{00000000-1CC8-4FD7-9281-6B8DBB9D3DF8}"
ProjectSection(ProjectDependencies) = postProject
{00000000-94D5-4BF9-8A50-7BD9929A0850} = {00000000-94D5-4BF9-8A50-7BD9929A0850}
EndProjectSection
EndProject
Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "datagen", "datagen\datagen.vcxproj", "{037E781E-81A6-494B-B1B3-438AB1200523}"
EndProject
Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "libzstd", "libzstd\libzstd.vcxproj", "{8BFD8150-94D5-4BF9-8A50-7BD9929A0850}"
Expand Down
1 change: 1 addition & 0 deletions build/single_file_libs/.gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ zstddeclib.c
zstdenclib.c
zstd.c
zstd.h
zstd_errors.h

# test artifacts
temp*
2 changes: 1 addition & 1 deletion contrib/largeNbDicts/largeNbDicts.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1025,7 +1025,7 @@ int main (int argc, const char** argv)
unsigned nbBlocks = 0; /* determine nbBlocks automatically, from source and blockSize */
ZSTD_dictContentType_e dictContentType = ZSTD_dct_auto;
ZSTD_dictAttachPref_e dictAttachPref = ZSTD_dictDefaultAttach;
ZSTD_paramSwitch_e prefetchCDictTables = ZSTD_ps_auto;
ZSTD_ParamSwitch_e prefetchCDictTables = ZSTD_ps_auto;
metricAggregatePref_e metricAggregatePref = fastest;

for (int argNb = 1; argNb < argc ; argNb++) {
Expand Down
Loading
Loading