README.md

SPIRVSmith

SPIRVSmith

SPIRVSmith is a differential testing tool that leverages structured fuzzing techniques to find bugs in producers and consumers of SPIRV shaders.

SPIRVSmith attempts to find bugs in the following projects:

• SPIRV-Tools - mostly the validator and reducer components.
• SPIRV-Cross
• Amber
• SwiftShader
• MoltenVK
• ...and more to come!

Table of Contents

• Installation
• Usage
• How does it work?
  • Differential testing
  • Program Reconditioning
• SPIR-V Language Coverage
• Contributing
• Authors
• License

Installation

SPIRVSmith uses poetry to manage Python dependencies and ships with scripts to install external dependencies.

1. Follow the poetry installation instructions for your platform.

2. Grab a local copy of SPIRVSmith:

$ git clone https://github.com/rayanht/SPIRVSmith.git && cd SPIRVSmith

3. Install Python dependencies using poetry and start a poetry shell:

$ poetry install && poetry shell

4. Install the external dependencies:

$ mkdir bin
$ sh scripts/get_spirv_tools.sh <platform>

Replace <platform> by either linux or macos (No support for Windows 😭)

Usage

The first step is to run SPIRVSmith is to head to config.py. That file contains the various parameters that are used to dictate the behaviour of the fuzzer.

SPIRVSmith is highly parametrizable, you can choose to disable certain features of the SPIR-V language (e.g. do not emit any control flow operation), limit the number of global constants generated, favour the generation of certain kinds of instructions etc.

Once you are happy with your parametrization, make sure you are in a poetry virtual environment ($ poetry shell) and run SPIRVSmith:

$ sh scripts/run.sh

SPIR-V assembly files will be saved as they are generated to the out/ directory and the fuzzer can be stopped at any time by pressing Ctrl+C.

How does it work?

Differential Testing

The bread and butter of SPIRVSmith is differential testing (sometimes called differential fuzzing), in which we provide the same SPIRV shader to similar consumers (say three different SPIRV compilers for example), execute the three resulting programs and compare the values contained inside all buffers at the end of exeuction.

In a fully deterministic program (== synchronous && free of undefined behaviour), we expect all these buffers to be exactly the same at the end of execution, regardless of what compiler was used to generate said program. If one program ends up with different buffers than the other two, we have a strong reason to believe that the compiler that generated it has a bug.

This concept can be extended further by varying the platform that executes the shader. If we get different results by running the same shader on an Nvidia GPU, an AMD GPU, and an Intel integrated graphics chip then there is a good chance that either the underlying Vulkan engine or the GPU driver has a bug (possibly both).

The constraint on determinism creates an interesting problem, how can we ensure that the randomly generated programs are free of undefined behaviour? Unlike existing differential testing tools, SPIRVSmith does not perform any static analysis or backtracking at generation-time to enforce this constraint, we rather implement the idea of program reconditioning by Donaldson et al.

Program Reconditioning

Donaldson et al. introduce the idea of program reconditioning as "a method for leveraging off-the-shelf test case reducers to simplify programs that expose miscompilation bugs during randomised differential testing".

This approach solves the issues raised by Yang et al. in the case of CSmith where test case reducers couldn't be used to provide concise, actionable bug reports since they would themselves often introduce undefined behaviour.

Program reconditioning works by decoupling the process of generating a program and the process of ensuring that said program is free of undefined behaviour. This is in contrast to Livinskii et al. in YARPGen where code generation steps and static analysis steps are interleaved.

Donaldson et al. describe a rule-based reconditioning approach where transforms are applied to constructs that could exhibit undefined behaviour. Transforms are applied to all eligible construct in a blanket manner, no static analysis is performed to determine which constructs are in fact worth reconditioning. Here is example of reconditioning a GLSL shader:

Original

float A [3]; // Not initialised
void main () {
    int i = int (A[0]);
    float f = 2000000.0;
    while (i != -42) { // Might not terminate
        A[i] = f; // Out of  bounds ?
        f = f + f; // Roundoff
        int j = i ++ + ( i / ( i - 1)); // Order of side effects, divide by zero?
        i = j;
    }
}

Reconditioned

// [ Declarations of SAFE_ABS , MAKE_IN_RANGE and SAFE_DIV ]
uint _loop_count = 0u;
const uint _loop_limit = 100u;
float A [3] = float [3](1.0 , 1.0 , 1.0);
void main () {
    int i = int (A[0]);
    float f = 2000000.0;
    while (i != -42) {
        if (_loop_count >= _loop_limit) break;
        _loop_count ++;
        A[SAFE_ABS(i) % 3] = f ;
        f = MAKE_IN_RANGE(f + f);
        int _t = SAFE_DIV(i , i - 1);
        int j = i ++ + _t;
        i = j;
    }
}

SPIRV Language Coverage

Expand

Instructions

Miscellanous

Expand

OpCode	Status
OpNop	🔴
OpUndef	🔴
OpSizeOf	🔴

Debug

Expand

OpCode	Status
OpSourceContinued	🔴
OpSource	🔴
OpSourceExtension	🔴
OpName	🔴
OpMemberName	🔴
OpString	🔴
OpLine	🔴
OpNoLine	🔴
OpModuleProcessed	🔴

Annotation

Expand

OpCode	Status
OpDecorate	✅
OpMemberDecorate	✅
OpDecorationGroup	🔴
OpGroupDecorate	🔴
OpGroupMemberDecorate	🔴
OpDecorateId	🔴
OpDecorateString	🔴
OpMemberDecorateString	🔴

Extension

Expand

OpCode	Status
OpExtension	✅
OpExtInstImport	✅
OpExtInst	✅

Mode-Setting

Expand

OpCode	Status
OpMemoryModel	✅
OpEntryPoint	✅
OpExecutionMode	✅
OpCapability	✅
OpExecutionModeId	🔴

Type-Declaration

Expand

OpCode	Status
OpTypeVoid	✅
OpTypeBool	✅
OpTypeInt	✅
OpTypeFloat	✅
OpTypeVector	✅
OpTypeMatrix	✅
OpTypeImage	🔴
OpTypeSampler	🔴
OpTypeSampledImage	🔴
OpTypeArray	✅
OpTypeRuntimeArray	🔴
OpTypeStruct	✅
OpTypeOpaque	🔴
OpTypePointer	✅
OpTypeFunction	✅

Constant-Creation

Expand

OpCode	Status
OpConstantTrue	✅
OpConstantFalse	✅
OpConstant	✅
OpConstantComposite	✅
OpConstantSampler	🔴
OpConstantNull	🔴
OpSpecConstantTrue	🔴
OpSpecConstantFalse	🔴
OpSpecConstant	🔴
OpSpecConstantComposite	🔴
OpSpecConstantOp	🔴

Memory

Expand

OpCode	Status
OpVariable	✅
OpImageTexelPointer	🔴
OpLoad	✅
OpStore	✅
OpCopyMemory	🔴
OpCopyMemorySized	🔴
OpAccessChain	✅
OpInBoundsAccessChain	🔴
OpPtrAccessChain	🔴
OpPtrEqual	🔴
OpPtrNotEqual	🔴
OpPtrDiff	🔴

Function

Expand

OpCode	Status
OpFunction	✅
OpFunctionParameter	✅
OpFunctionEnd	✅
OpFunctionCall	🔴

Image

Expand

OpCode	Status
OpSampledImage	🔴
OpImageSampleImplicitLod	🔴
OpImageSampleExplicitLod	🔴
OpImageSampleDrefImplicitLod	🔴
OpImageSampleDrefExplicitLod	🔴
OpImageSampleProjImplicitLod	🔴
OpImageSampleProjExplicitLod	🔴
OpImageSampleProjDrefImplicitLod	🔴
OpImageSampleProjDrefExplicitLod	🔴
OpImageFetch	🔴
OpImageGather	🔴
OpImageDrefGather	🔴
OpImageRead	🔴
OpImageWrite	🔴
OpImage	🔴
OpImageQueryFormat	🔴
OpImageQueryOrder	🔴
OpImageQuerySizeLod	🔴
OpImageQuerySize	🔴
OpImageQueryLod	🔴
OpImageQueryLevels	🔴
OpImageQuerySamples	🔴
OpImageSparseSampleImplicitLod	🔴
OpImageSparseSampleExplicitLod	🔴
OpImageSparseSampleDrefImplicitLod	🔴
OpImageSparseSampleDrefExplicitLod	🔴
OpImageSparseFetch	🔴
OpImageSparseGather	🔴
OpImageSparseDrefGather	🔴
OpImageSparseTexelsResident	🔴
OpImageSparseRead	🔴

Conversion

Expand

OpCode	Status
OpConvertFToU	✅
OpConvertFToS	✅
OpConvertSToF	✅
OpConvertUToF	✅
OpUConvert	🔴
OpSConvert	🔴
OpFConvert	🔴
OpQuantizeToF16	🔴
OpConvertPtrToU	🔴
OpSatConvertSToU	🔴
OpSatConvertUToS	🔴
OpConvertUToPtr	🔴
OpPtrCastToGeneric	🔴
OpGenericCastToPtr	🔴
OpGenericCastToPtrExplicit	🔴
OpBitcast	🔴

Composite

Expand

OpCode	Status
OpVectorExtractDynamic	✅
OpVectorInsertDynamic	✅
OpVectorShuffle	✅
OpCompositeConstruct	🔴
OpCompositeExtract	✅
OpCompositeInsert	✅
OpCopyObject	✅
OpTranspose	✅
OpCopyLogical	🔴

Arithmetic

Expand

OpCode	Status
OpSNegate	✅
OpFNegate	✅
OpIAdd	✅
OpFAdd	✅
OpISub	✅
OpFSub	✅
OpIMul	✅
OpFMul	✅
OpUDiv	✅
OpSDiv	✅
OpFDiv	✅
OpUMod	✅
OpSRem	✅
OpSMod	✅
OpFRem	✅
OpFMod	✅
OpVectorTimesScalar	✅
OpMatrixTimesScalar	✅
OpVectorTimesMatrix	✅
OpMatrixTimesVector	✅
OpMatrixTimesMatrix	✅
OpOuterProduct	✅
OpDot	✅
OpIAddCarry	🔴
OpISubBorrow	🔴
OpUMulExtended	🔴
OpSMulExtended	🔴

Bit

Expand

OpCode	Status
OpShiftRightLogical	✅
OpShiftRightArithmetic	✅
OpShiftLeftLogical	✅
OpBitwiseOr	✅
OpBitwiseXor	✅
OpBitwiseAnd	✅
OpNot	✅
OpBitFieldInsert	✅
OpBitFieldSExtract	✅
OpBitFieldUExtract	✅
OpBitReverse	✅
OpBitCount	✅

Relational and Logical

Expand

OpCode	Status
OpAny	✅
OpAll	✅
OpIsNan	✅
OpIsInf	✅
OpIsFinite	🔴
OpIsNormal	🔴
OpSignBitSet	🔴
OpOrdered	🔴
OpUnordered	🔴
OpLogicalEqual	✅
OpLogicalNotEqual	✅
OpLogicalOr	✅
OpLogicalAnd	✅
OpLogicalNot	✅
OpSelect	✅
OpIEqual	✅
OpINotEqual	✅
OpUGreaterThan	✅
OpSGreaterThan	✅
OpUGreaterThanEqual	✅
OpSGreaterThanEqual	✅
OpULessThan	✅
OpSLessThan	✅
OpULessThanEqual	✅
OpSLessThanEqual	✅
OpFOrdEqual	✅
OpFUnordEqual	✅
OpFOrdNotEqual	✅
OpFUnordNotEqual	✅
OpFOrdLessThan	✅
OpFUnordLessThan	✅
OpFOrdGreaterThan	✅
OpFUnordGreaterThan	✅
OpFOrdLessThanEqual	✅
OpFUnordLessThanEqual	✅
OpFOrdGreaterThanEqual	✅
OpFUnordGreaterThanEqual	✅

Control Flow

Expand

OpCode	Status
OpPhi	🔴
OpLoopMerge	✅
OpSelectionMerge	✅
OpLabel	✅
OpBranch	✅
OpBranchConditional	✅
OpSwitch	🔴
OpReturn	🔴
OpReturnValue	🔴

Contributing

Encountered a bug? Have an idea for a new feature? This project is open to all sorts of contribution! Feel free to head to the Issues tab and describe your request!

Authors

• Rayan Hatout - GitHub | Twitter | LinkedIn

License

This project is licensed under the Apache 2.0 license.