Skip to content

Commit

Permalink
Merge pull request #4184 from facebook/ZSTD_getErrorCode
Browse files Browse the repository at this point in the history 
elevated ZSTD_getErrorCode() to stable status
  • Loading branch information
@Cyan4973
Cyan4973 authored Nov 5, 2024
2 parents 15c2916 + 2e02cd3 commit 51eb7da
Show file tree
Hide file tree
Showing 5 changed files with 141 additions and 101 deletions.
1 change: 1 addition & 0 deletions build/single_file_libs/build_library_test.sh
View file
Original file line number Diff line number Diff line change
Expand Up @@ -70,6 +70,7 @@ echo "Single file library creation script: PASSED"

# Copy the header to here (for the tests)
cp "$ZSTD_SRC_ROOT/zstd.h" examples/zstd.h
cp "$ZSTD_SRC_ROOT/zstd_errors.h" examples/zstd_errors.h

# Compile the generated output
cc -Wall -Wextra -Werror -Wshadow -pthread -I. -Os -g0 -o $OUT_FILE zstd.c examples/roundtrip.c
Expand Down
1 change: 1 addition & 0 deletions contrib/gen_html/gen_html.cpp
View file
Original file line number Diff line number Diff line change
Expand Up @@ -211,6 +211,7 @@ int main(int argc, char *argv[]) {

ostream << "<html>\n<head>\n<meta http-equiv=\"Content-Type\" content=\"text/html; charset=ISO-8859-1\">\n<title>" << version << "</title>\n</head>\n<body>" << endl;
ostream << "<h1>" << version << "</h1>\n";
ostream << "Note: the content of this file has been automatically generated by parsing \"zstd.h\" \n";

ostream << "<hr>\n<a name=\"Contents\"></a><h2>Contents</h2>\n<ol>\n";
for (size_t i=0; i<chapters.size(); i++)
Expand Down
186 changes: 111 additions & 75 deletions doc/zstd_manual.html
View file
Original file line number Diff line number Diff line change
@@ -1,16 +1,17 @@
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1">
<title>zstd 1.5.6 Manual</title>
<title>zstd 1.5.7 Manual</title>
</head>
<body>
<h1>zstd 1.5.6 Manual</h1>
<h1>zstd 1.5.7 Manual</h1>
Note: the content of this file has been automatically generated by parsing "zstd.h"
<hr>
<a name="Contents"></a><h2>Contents</h2>
<ol>
<li><a href="#Chapter1">Introduction</a></li>
<li><a href="#Chapter2">Version</a></li>
<li><a href="#Chapter3">Simple API</a></li>
<li><a href="#Chapter3">Simple Core API</a></li>
<li><a href="#Chapter4">Explicit context</a></li>
<li><a href="#Chapter5">Advanced compression API (Requires v1.4.0+)</a></li>
<li><a href="#Chapter6">Advanced decompression API (Requires v1.4.0+)</a></li>
Expand Down Expand Up @@ -74,7 +75,7 @@ <h1>zstd 1.5.6 Manual</h1>
</b><p> Return runtime library version, like "1.4.5". Requires v1.3.0+.
</p></pre><BR>

<a name="Chapter3"></a><h2>Simple API</h2><pre></pre>
<a name="Chapter3"></a><h2>Simple Core API</h2><pre></pre>

<pre><b>size_t ZSTD_compress( void* dst, size_t dstCapacity,
const void* src, size_t srcSize,
Expand All @@ -88,38 +89,42 @@ <h1>zstd 1.5.6 Manual</h1>

<pre><b>size_t ZSTD_decompress( void* dst, size_t dstCapacity,
const void* src, size_t compressedSize);
</b><p> `compressedSize` : must be the _exact_ size of some number of compressed and/or skippable frames.
`dstCapacity` is an upper bound of originalSize to regenerate.
If user cannot imply a maximum upper bound, it's better to use streaming mode to decompress data.
@return : the number of bytes decompressed into `dst` (<= `dstCapacity`),
or an errorCode if it fails (which can be tested using ZSTD_isError()).
</b><p> `compressedSize` : must be the _exact_ size of some number of compressed and/or skippable frames.
Multiple compressed frames can be decompressed at once with this method.
The result will be the concatenation of all decompressed frames, back to back.
`dstCapacity` is an upper bound of originalSize to regenerate.
First frame's decompressed size can be extracted using ZSTD_getFrameContentSize().
If maximum upper bound isn't known, prefer using streaming mode to decompress data.
@return : the number of bytes decompressed into `dst` (<= `dstCapacity`),
or an errorCode if it fails (which can be tested using ZSTD_isError()).
</p></pre><BR>

<h3>Decompression helper functions</h3><pre></pre><b><pre></pre></b><BR>
<pre><b>#define ZSTD_CONTENTSIZE_UNKNOWN (0ULL - 1)
#define ZSTD_CONTENTSIZE_ERROR (0ULL - 2)
unsigned long long ZSTD_getFrameContentSize(const void *src, size_t srcSize);
</b><p> `src` should point to the start of a ZSTD encoded frame.
`srcSize` must be at least as large as the frame header.
hint : any size >= `ZSTD_frameHeaderSize_max` is large enough.
@return : - decompressed size of `src` frame content, if known
- ZSTD_CONTENTSIZE_UNKNOWN if the size cannot be determined
- ZSTD_CONTENTSIZE_ERROR if an error occurred (e.g. invalid magic number, srcSize too small)
note 1 : a 0 return value means the frame is valid but "empty".
note 2 : decompressed size is an optional field, it may not be present, typically in streaming mode.
When `return==ZSTD_CONTENTSIZE_UNKNOWN`, data to decompress could be any size.
In which case, it's necessary to use streaming mode to decompress data.
Optionally, application can rely on some implicit limit,
as ZSTD_decompress() only needs an upper bound of decompressed size.
(For example, data could be necessarily cut into blocks <= 16 KB).
note 3 : decompressed size is always present when compression is completed using single-pass functions,
such as ZSTD_compress(), ZSTD_compressCCtx() ZSTD_compress_usingDict() or ZSTD_compress_usingCDict().
note 4 : decompressed size can be very large (64-bits value),
potentially larger than what local system can handle as a single memory segment.
In which case, it's necessary to use streaming mode to decompress data.
note 5 : If source is untrusted, decompressed size could be wrong or intentionally modified.
Always ensure return value fits within application's authorized limits.
Each application can set its own limits.
note 6 : This function replaces ZSTD_getDecompressedSize()
</b><p> `src` should point to the start of a ZSTD encoded frame.
`srcSize` must be at least as large as the frame header.
hint : any size >= `ZSTD_frameHeaderSize_max` is large enough.
@return : - decompressed size of `src` frame content, if known
- ZSTD_CONTENTSIZE_UNKNOWN if the size cannot be determined
- ZSTD_CONTENTSIZE_ERROR if an error occurred (e.g. invalid magic number, srcSize too small)
note 1 : a 0 return value means the frame is valid but "empty".
note 2 : decompressed size is an optional field, it may not be present (typically in streaming mode).
When `return==ZSTD_CONTENTSIZE_UNKNOWN`, data to decompress could be any size.
In which case, it's necessary to use streaming mode to decompress data.
Optionally, application can rely on some implicit limit,
as ZSTD_decompress() only needs an upper bound of decompressed size.
(For example, data could be necessarily cut into blocks <= 16 KB).
note 3 : decompressed size is always present when compression is completed using single-pass functions,
such as ZSTD_compress(), ZSTD_compressCCtx() ZSTD_compress_usingDict() or ZSTD_compress_usingCDict().
note 4 : decompressed size can be very large (64-bits value),
potentially larger than what local system can handle as a single memory segment.
In which case, it's necessary to use streaming mode to decompress data.
note 5 : If source is untrusted, decompressed size could be wrong or intentionally modified.
Always ensure return value fits within application's authorized limits.
Each application can set its own limits.
note 6 : This function replaces ZSTD_getDecompressedSize()
</p></pre><BR>

<pre><b>ZSTD_DEPRECATED("Replaced by ZSTD_getFrameContentSize")
Expand All @@ -140,50 +145,54 @@ <h1>zstd 1.5.6 Manual</h1>
or an error code if input is invalid
</p></pre><BR>

<h3>Helper functions</h3><pre></pre><b><pre></b>/* ZSTD_compressBound() :<b>
* maximum compressed size in worst case single-pass scenario.
* When invoking `ZSTD_compress()` or any other one-pass compression function,
* it's recommended to provide @dstCapacity >= ZSTD_compressBound(srcSize)
* as it eliminates one potential failure scenario,
* aka not enough room in dst buffer to write the compressed frame.
* Note : ZSTD_compressBound() itself can fail, if @srcSize > ZSTD_MAX_INPUT_SIZE .
* In which case, ZSTD_compressBound() will return an error code
* which can be tested using ZSTD_isError().
*
* ZSTD_COMPRESSBOUND() :
* same as ZSTD_compressBound(), but as a macro.
* It can be used to produce constants, which can be useful for static allocation,
* for example to size a static array on stack.
* Will produce constant value 0 if srcSize too large.
*/
#define ZSTD_MAX_INPUT_SIZE ((sizeof(size_t)==8) ? 0xFF00FF00FF00FF00ULL : 0xFF00FF00U)
<h3>Compression helper functions</h3><pre></pre><b><pre></pre></b><BR>
<pre><b>#define ZSTD_MAX_INPUT_SIZE ((sizeof(size_t)==8) ? 0xFF00FF00FF00FF00ULL : 0xFF00FF00U)
#define ZSTD_COMPRESSBOUND(srcSize) (((size_t)(srcSize) >= ZSTD_MAX_INPUT_SIZE) ? 0 : (srcSize) + ((srcSize)>>8) + (((srcSize) < (128<<10)) ? (((128<<10) - (srcSize)) >> 11) </b>/* margin, from 64 to 0 */ : 0)) /* this formula ensures that bound(A) + bound(B) <= bound(A+B) as long as A and B >= 128 KB */<b>
size_t ZSTD_compressBound(size_t srcSize); </b>/*!< maximum compressed size in worst case single-pass scenario */<b>
</b><p> maximum compressed size in worst case single-pass scenario.
When invoking `ZSTD_compress()`, or any other one-pass compression function,
it's recommended to provide @dstCapacity >= ZSTD_compressBound(srcSize)
as it eliminates one potential failure scenario,
aka not enough room in dst buffer to write the compressed frame.
Note : ZSTD_compressBound() itself can fail, if @srcSize > ZSTD_MAX_INPUT_SIZE .
In which case, ZSTD_compressBound() will return an error code
which can be tested using ZSTD_isError().

ZSTD_COMPRESSBOUND() :
same as ZSTD_compressBound(), but as a macro.
It can be used to produce constants, which can be useful for static allocation,
for example to size a static array on stack.
Will produce constant value 0 if srcSize is too large.

</p></pre><BR>

<h3>Error helper functions</h3><pre></pre><b><pre>#include "zstd_errors.h" </b>/* list of errors */<b>
</b>/* ZSTD_isError() :<b>
* Most ZSTD_* functions returning a size_t value can be tested for error,
* using ZSTD_isError().
* @return 1 if error, 0 otherwise
*/
unsigned ZSTD_isError(size_t code); </b>/*!< tells if a `size_t` function result is an error code */<b>
const char* ZSTD_getErrorName(size_t code); </b>/*!< provides readable string from an error code */<b>
int ZSTD_minCLevel(void); </b>/*!< minimum negative compression level allowed, requires v1.4.0+ */<b>
int ZSTD_maxCLevel(void); </b>/*!< maximum compression level available */<b>
int ZSTD_defaultCLevel(void); </b>/*!< default compression level, specified by ZSTD_CLEVEL_DEFAULT, requires v1.5.0+ */<b>
unsigned ZSTD_isError(size_t result); </b>/*!< tells if a `size_t` function result is an error code */<b>
ZSTD_ErrorCode ZSTD_getErrorCode(size_t functionResult); </b>/* convert a result into an error code, which can be compared to error enum list */<b>
const char* ZSTD_getErrorName(size_t result); </b>/*!< provides readable string from a function result */<b>
int ZSTD_minCLevel(void); </b>/*!< minimum negative compression level allowed, requires v1.4.0+ */<b>
int ZSTD_maxCLevel(void); </b>/*!< maximum compression level available */<b>
int ZSTD_defaultCLevel(void); </b>/*!< default compression level, specified by ZSTD_CLEVEL_DEFAULT, requires v1.5.0+ */<b>
</pre></b><BR>
<a name="Chapter4"></a><h2>Explicit context</h2><pre></pre>

<h3>Compression context</h3><pre> When compressing many times,
it is recommended to allocate a context just once,
it is recommended to allocate a compression context just once,
and reuse it for each successive compression operation.
This will make workload friendlier for system's memory.
This will make the workload easier for system's memory.
Note : re-using context is just a speed / resource optimization.
It doesn't change the compression ratio, which remains identical.
Note 2 : In multi-threaded environments,
use one different context per thread for parallel execution.
Note 2: For parallel execution in multi-threaded environments,
use one different context per thread .

</pre><b><pre>typedef struct ZSTD_CCtx_s ZSTD_CCtx;
ZSTD_CCtx* ZSTD_createCCtx(void);
size_t ZSTD_freeCCtx(ZSTD_CCtx* cctx); </b>/* accept NULL pointer */<b>
size_t ZSTD_freeCCtx(ZSTD_CCtx* cctx); </b>/* compatible with NULL pointer */<b>
</pre></b><BR>
<pre><b>size_t ZSTD_compressCCtx(ZSTD_CCtx* cctx,
void* dst, size_t dstCapacity,
Expand All @@ -194,7 +203,7 @@ <h3>Compression context</h3><pre> When compressing many times,
this function compresses at the requested compression level,
__ignoring any other advanced parameter__ .
If any advanced parameter was set using the advanced API,
they will all be reset. Only `compressionLevel` remains.
they will all be reset. Only @compressionLevel remains.

</p></pre><BR>

Expand Down Expand Up @@ -394,7 +403,8 @@ <h3>Decompression context</h3><pre> When decompressing many times,
* ZSTD_c_stableOutBuffer
* ZSTD_c_blockDelimiters
* ZSTD_c_validateSequences
* ZSTD_c_useBlockSplitter
* ZSTD_c_blockSplitterLevel
* ZSTD_c_splitAfterSequences
* ZSTD_c_useRowMatchFinder
* ZSTD_c_prefetchCDictTables
* ZSTD_c_enableSeqProducerFallback
Expand All @@ -421,7 +431,8 @@ <h3>Decompression context</h3><pre> When decompressing many times,
ZSTD_c_experimentalParam16=1013,
ZSTD_c_experimentalParam17=1014,
ZSTD_c_experimentalParam18=1015,
ZSTD_c_experimentalParam19=1016
ZSTD_c_experimentalParam19=1016,
ZSTD_c_experimentalParam20=1017
} ZSTD_cParameter;
</b></pre><BR>
<pre><b>typedef struct {
Expand Down Expand Up @@ -718,7 +729,7 @@ <h3>Streaming compression functions</h3><pre></pre><b><pre>typedef enum {
<a name="Chapter9"></a><h2>Streaming decompression - HowTo</h2><pre>
A ZSTD_DStream object is required to track streaming operations.
Use ZSTD_createDStream() and ZSTD_freeDStream() to create/release resources.
ZSTD_DStream objects can be reused multiple times.
ZSTD_DStream objects can be re-employed multiple times.

Use ZSTD_initDStream() to start a new decompression operation.
@return : recommended first input size
Expand All @@ -728,16 +739,21 @@ <h3>Streaming compression functions</h3><pre></pre><b><pre>typedef enum {
The function will update both `pos` fields.
If `input.pos < input.size`, some input has not been consumed.
It's up to the caller to present again remaining data.

The function tries to flush all data decoded immediately, respecting output buffer size.
If `output.pos < output.size`, decoder has flushed everything it could.
But if `output.pos == output.size`, there might be some data left within internal buffers.,

However, when `output.pos == output.size`, it's more difficult to know.
If @return > 0, the frame is not complete, meaning
either there is still some data left to flush within internal buffers,
or there is more input to read to complete the frame (or both).
In which case, call ZSTD_decompressStream() again to flush whatever remains in the buffer.
Note : with no additional input provided, amount of data flushed is necessarily <= ZSTD_BLOCKSIZE_MAX.
@return : 0 when a frame is completely decoded and fully flushed,
or an error code, which can be tested using ZSTD_isError(),
or any other value > 0, which means there is still some decoding or flushing to do to complete current frame :
the return value is a suggested next input size (just a hint for better latency)
that will never request more than the remaining frame size.
that will never request more than the remaining content of the compressed frame.

<BR></pre>

Expand All @@ -763,9 +779,10 @@ <h3>Streaming decompression functions</h3><pre></pre><b><pre></pre></b><BR>
Function will update both input and output `pos` fields exposing current state via these fields:
- `input.pos < input.size`, some input remaining and caller should provide remaining input
on the next call.
- `output.pos < output.size`, decoder finished and flushed all remaining buffers.
- `output.pos == output.size`, potentially uncflushed data present in the internal buffers,
call ZSTD_decompressStream() again to flush remaining data to output.
- `output.pos < output.size`, decoder flushed internal output buffer.
- `output.pos == output.size`, unflushed data potentially present in the internal buffers,
check ZSTD_decompressStream() @return value,
if > 0, invoke it again to flush remaining data to output.
Note : with no additional input, amount of data flushed <= ZSTD_BLOCKSIZE_MAX.

@return : 0 when a frame is completely decoded and fully flushed,
Expand Down Expand Up @@ -1311,19 +1328,37 @@ <h3>Streaming decompression functions</h3><pre></pre><b><pre></pre></b><BR>

</p></pre><BR>

<pre><b></b><p> Generate sequences using ZSTD_compress2(), given a source buffer.
<pre><b>ZSTD_DEPRECATED("For debugging only, will be replaced by ZSTD_extractSequences()")
ZSTDLIB_STATIC_API size_t
ZSTD_generateSequences(ZSTD_CCtx* zc,
ZSTD_Sequence* outSeqs, size_t outSeqsSize,
const void* src, size_t srcSize);
</b><p> WARNING: This function is meant for debugging and informational purposes ONLY!
Its implementation is flawed, and it will be deleted in a future version.
It is not guaranteed to succeed, as there are several cases where it will give
up and fail. You should NOT use this function in production code.

This function is deprecated, and will be removed in a future version.

Generate sequences using ZSTD_compress2(), given a source buffer.

@param zc The compression context to be used for ZSTD_compress2(). Set any
compression parameters you need on this context.
@param outSeqs The output sequences buffer of size @p outSeqsSize
@param outSeqsSize The size of the output sequences buffer.
ZSTD_sequenceBound(srcSize) is an upper bound on the number
of sequences that can be generated.
@param src The source buffer to generate sequences from of size @p srcSize.
@param srcSize The size of the source buffer.

Each block will end with a dummy sequence
with offset == 0, matchLength == 0, and litLength == length of last literals.
litLength may be == 0, and if so, then the sequence of (of: 0 ml: 0 ll: 0)
simply acts as a block delimiter.

@zc can be used to insert custom compression params.
This function invokes ZSTD_compress2().

The output of this function can be fed into ZSTD_compressSequences() with CCtx
setting of ZSTD_c_blockDelimiters as ZSTD_sf_explicitBlockDelimiters
@return : number of sequences generated
@returns The number of sequences generated, necessarily less than
ZSTD_sequenceBound(srcSize), or an error code that can be checked
with ZSTD_isError().

</p></pre><BR>

Expand Down Expand Up @@ -1512,13 +1547,14 @@ <h3>Streaming decompression functions</h3><pre></pre><b><pre></pre></b><BR>
#ifdef __GNUC__
__attribute__((__unused__))
#endif
ZSTD_customMem const ZSTD_defaultCMem = { NULL, NULL, NULL }; </b>/**< this constant defers to stdlib's functions */<b>
</b><p> These prototypes make it possible to pass your own allocation/free functions.
ZSTD_customMem is provided at creation time, using ZSTD_create*_advanced() variants listed below.
All allocation/free operations will be completed using these custom variants instead of regular <stdlib.h> ones.

</p></pre><BR>

<pre><b>ZSTD_customMem const ZSTD_defaultCMem = { NULL, NULL, NULL }; </b>/**< this constant defers to stdlib's functions */<b>
</b></pre><BR>
<pre><b>typedef struct POOL_ctx_s ZSTD_threadPool;
ZSTDLIB_STATIC_API ZSTD_threadPool* ZSTD_createThreadPool(size_t numThreads);
ZSTDLIB_STATIC_API void ZSTD_freeThreadPool (ZSTD_threadPool* pool); </b>/* accept NULL pointer */<b>
Expand Down
Loading

0 comments on commit 51eb7da

Please sign in to comment.