Skip to content

Commit

Permalink
Merge pull request #251 from sammsaski/formalise25v2
Browse files Browse the repository at this point in the history 
FormaliSE 2025 AE Submission Fixes
  • Loading branch information
@mldiego
mldiego authored Dec 7, 2024
2 parents 1ecabc6 + f656198 commit 4abdc23
Show file tree
Hide file tree
Showing 6 changed files with 365 additions and 325 deletions.
51 changes: 32 additions & 19 deletions code/nnv/examples/Submission/FORMALISE2025/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2,20 +2,20 @@

This artifact is used to reproduce the results in _Robustness Verification of Video Classification Neural Networks_.

All results from _Robustness Verification of Video Classification Neural Networks_ were captured using an `Apple M1 Max 10-core CPU@3.20GHz×10` with 64GB of RAM. The times reported to run the smoke test and reproduce results are approximated based on this computer.
All results from _Robustness Verification of Video Classification Neural Networks_ were captured using an `Apple M1 Max 10-core CPU@3.20GHz×10` with 64GB of RAM.

# Requirements

The following resources are required to run this artifact:

- MATLAB 2024a with NNV and npy-matlab installed and added to the path.
- MATLAB 2024a with NNV and `npy-matlab` installed and added to the path.
- A conda environment with Python v3.11. Install rquirements from `requirements.txt`. Make sure to install the source files.

# Installation

This section describes all of the necessary steps for installing tools, dependencies, etc. needed to reproduce the artifact. For the remainder of these instructions, when the `FORMALISE2025` directory is referred to, this really means the directory at the following path: `nnv/code/nnv/examples/Submission/FORMALISE2025`.
This section describes all of the necessary steps for installing tools, dependencies, etc. needed to reproduce the artifact. For the remainder of these instructions when the `FORMALISE2025` directory is referred to we are really referring to the directory at the following path: `nnv/code/nnv/examples/Submission/FORMALISE2025`.

1. Clone NNV and npy-matlab and install them ensuring that both have been added to the MATLAB path.
1. Clone NNV and `npy-matlab` and install them ensuring that both have been added to the MATLAB path.

```
# Clone NNV and npy-matlab
Expand All @@ -33,7 +33,7 @@ Either comment it out or remove it.

For `npy-matlab`, add and save the path to MATLAB. Again, instructions are available on [`npy-matlab`'s README](https://github.com/kwikteam/npy-matlab/blob/master/README.md).

For installing both of these tools, it will be necessary to run with administrator privileges so that you can save them to the MATLAB path.
> NOTE: It will be necessary to perform installations with administrator privileges for both of these tools so that it is possible to `savepath` in MATLAB after completing their respective installation instructions.
2. Download the following dataset files from [here](https://drive.google.com/drive/folders/1sXRtSObHLBTeKVss2IA-NGPKljirgD8D?usp=drive_link) into the `FORMALISE2025/data` folder so that the file tree now looks like

Expand All @@ -47,25 +47,26 @@ FORMALISE2025
...
```

Note that the `GTSRB` data folder is quite large, so it will likely be downloaded in separate parts. Please make sure the organization of the folder is the same as in the download link.
Note that the `GTSRB` data folder is quite large, so it will likely be downloaded in separate parts. Please make sure the organization of the folder is the same as in the download link and that all filenames match.

3. Create a conda environment with `Python v3.11` and install the requirements from `requirements.txt`. Additionally, install the source files to the environment. Both can be done by running the following commands from the root directory (`FORMALISE2025`):
3. Create a conda environment with `Python v3.11` and install the requirements from `requirements.txt`. Instructions for installing conda can be found [here](https://docs.conda.io/projects/conda/en/latest/user-guide/install/) with links under `# Regular installation` to instructions for various operating systems. After confirming that conda has installed successfully by creating a conda environment and making it active, proceed to install the requirements as detailed below. In addition to installing from `requirements.txt`, you must install the source files to the environment. Both of these can be done by running the following commands from the root directory (`FORMALISE2025`):

```bash
pip install -r requirements.txt
```bash
# installing requirements
pip install -r requirements.txt

# before installing source files, make sure to navigate to this src directory, e.g.
cd /path/to/FORMALISE2025/src
pip install -e .
```
# before installing source files, make sure to navigate to this src directory, e.g.
cd /path/to/FORMALISE2025/src
pip install -e .
```

4. Modify the `.env` file to add the path to your NNV and npy-matlab directories (the repositories that were cloned earlier). For the `npy-matlab` repository, you'll want to reference the subfolder in the directory also called `npy-matlab`, e.g. `/some/path/to/npy-matlab/npy-matlab`.

5. With all of these steps done, you are now ready to begin reproducing the artifacts.

# Smoke Test Instructions (~30 minutes)
# Smoke Test Instructions (~1min)

Instructions to quickly test the software needed to reproduce the artifacts of the paper. If no issues arise during the smoke test, you can safely proceed to reproducing all artifacts as described in the below section.
Instructions to quickly test the software needed to reproduce the artifacts of the paper. If no issues arise during the smoke test, you can safely proceed to reproducing all artifacts as described in the below sections.

1. Open a terminal and navigate to the `FORMALISE2025` directory.
2. Make sure the conda environment with the installed dependencies is activated.
Expand All @@ -75,17 +76,17 @@ Instructions to quickly test the software needed to reproduce the artifacts of t
chmod +x run_smoketest.sh && ./run_smoketest.sh
```

4. The smoketest will verify a single sample from each of the different datasets and with the different verification algorithms. The results will be output to a `smoketest_outputs.txt` file. Assuming everything ran smoothly, you should see:
4. The smoke test will verify a single sample. If the smoke test is successful, then the following message will be output.

```
**********************************************
Smoke test complete.
**********************************************
```

# Reproducing a Subset of the Results
# Reproducing a Subset of the Results (~1-2 hours)

Assuming the average runtime for the experiments remains as shown in the paper, then it will take approximately 9-10 days to reproduce the results. For that reason, this set of instructions is for reproducing a subset of the results. In this case, we verify every tenth sample from what was done for the original verification. Generating the reachable output range plots remains the same.
Assuming the average runtime for the experiments remains as shown in the paper, then it will take approximately 9-10 days to reproduce the results. For that reason, this set of instructions is for reproducing a subset of the results. More specifically, we reproduce the first row of Table 2 from the paper, e.g. the verification results for the 4-frame variation of the Zoom In dataset with the relax verification algorithm. The results will be output to the console after the script completes. Additionally, this script will generate the reachable output range plots used in Figure 7.

1. Open a terminal and navigate to the `FORMALISE2025` directory.
2. Make sure the conda environment with the installed dependencies is activated.
Expand All @@ -95,9 +96,21 @@ Assuming the average runtime for the experiments remains as shown in the paper,
chmod +x run_subset_vvn.sh && run_subset_vvn.sh
```

# Reproducing a Subset of the Results pt. 2

There is additionally a script to reproduce a single sample from all variations of the datasets. To run this subset, please perform the following:

1. Open a terminal and navigate to the `FORMALISE25` directory.
2. Make sure the conda environment with the installed dependencies is activated.
3. Run the following command to begin reproducing a subset of the artifacts:

```bash
chmod +x run_single_sample_vvn.sh && run_single_sample_vvn.sh
```

# Reproducing the Full Results

This set of instructions describes how to reproduce the full results. The instructions will be just like the previous parts only we use a different script.
This set of instructions describes how to reproduce the full results.

1. Open a terminal and navigate to the `FORMALISE2025` directory.
2. Make sure the conda environment with the installed dependencies is activated.
Expand Down
95 changes: 95 additions & 0 deletions code/nnv/examples/Submission/FORMALISE2025/run_single_sample_vvn.sh
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,95 @@
#!/bin/bash

OUTPUT_FILE="test_single_all_samples.txt"

# Clear the file if it exists
> $OUTPUT_FILE

echo "Starting test..."
echo "Starting test..." >> $OUTPUT_FILE

#######################
### ZOOM IN RESULTS ###
#######################
# Zoom In 4f, relax
python src/vvn/verify.py zoom_in relax 1 4 1800 751 >> $OUTPUT_FILE 2>&1

# Zoom In 8f, relax
python src/vvn/verify.py zoom_in relax 1 8 1800 751 >> $OUTPUT_FILE 2>&1

# Zoom In 16f, relax
python src/vvn/verify.py zoom_in relax 1 16 1800 751 >> $OUTPUT_FILE 2>&1

# Zoom In 4f, approx
python src/vvn/verify.py zoom_in approx 1 4 1800 751 >> $OUTPUT_FILE 2>&1

# Zoom In 8f, approx
python src/vvn/verify.py zoom_in approx 1 8 1800 751 >> $OUTPUT_FILE 2>&1


########################
### ZOOM OUT RESULTS ###
########################
# Zoom Out 4f, relax
python src/vvn/verify.py zoom_out relax 1 4 1800 624 >> $OUTPUT_FILE 2>&1

# Zoom Out 8f, relax
python src/vvn/verify.py zoom_out relax 1 8 1800 624 >> $OUTPUT_FILE 2>&1

# Zoom Out 16f, relax
python src/vvn/verify.py zoom_out relax 1 16 1800 624 >> $OUTPUT_FILE 2>&1

# Zoom Out 4f, approx
python src/vvn/verify.py zoom_out approx 1 4 1800 624 >> $OUTPUT_FILE 2>&1

# Zoom Out 8f, approx
python src/vvn/verify.py zoom_out approx 1 8 1800 624 >> $OUTPUT_FILE 2>&1


#######################
#### GTSRB RESULTS ####
#######################
# GTSRB 4f, relax
python src/vvn/verify.py gtsrb relax 1 4 1800 12597 >> $OUTPUT_FILE 2>&1

# GTSRB 8f, relax
python src/vvn/verify.py gtsrb relax 1 8 1800 12597 >> $OUTPUT_FILE 2>&1

# GTSRB 16f, relax
python src/vvn/verify.py gtsrb relax 1 16 1800 12597 >> $OUTPUT_FILE 2>&1

# GTSRB 4f, approx
python src/vvn/verify.py gtsrb approx 1 4 1800 12597 >> $OUTPUT_FILE 2>&1

# GTSRB 8f, approx
python src/vvn/verify.py gtsrb approx 1 8 1800 12597 >> $OUTPUT_FILE 2>&1

# GTSRB 16f, approx
python src/vvn/verify.py gtsrb approx 1 16 1800 12597 >> $OUTPUT_FILE 2>&1


#######################
### STMNIST RESULTS ###
#######################
# STMNIST 16f, relax
python src/vvn/verify.py stmnist relax 1 16 1800 311 >> $OUTPUT_FILE 2>&1

# STMNIST 32f, relax
python src/vvn/verify.py stmnist relax 1 32 1800 311 >> $OUTPUT_FILE 2>&1

# STMNIST 64f, relax
python src/vvn/verify.py stmnist relax 1 64 1800 311 >> $OUTPUT_FILE 2>&1

# STMNIST 16f, approx
python src/vvn/verify.py stmnist approx 1 16 1800 311 >> $OUTPUT_FILE 2>&1

# STMNIST 32f, approx
python src/vvn/verify.py stmnist approx 1 32 1800 311 >> $OUTPUT_FILE 2>&1

# STMNIST 64f, approx
python src/vvn/verify.py stmnist approx 1 64 1800 311 >> $OUTPUT_FILE 2>&1


echo -e "\n\n**********************************************"
echo " Test all samples complete. "
echo -e "**********************************************\n\n"
87 changes: 1 addition & 86 deletions code/nnv/examples/Submission/FORMALISE2025/run_smoketest.sh
View file
Original file line number Diff line number Diff line change
@@ -1,94 +1,9 @@
#!/bin/bash

OUTPUT_FILE="smoketest_outputs.txt"

# Clear the file if it exists
> $OUTPUT_FILE

echo "Starting smoke test..."
echo "Starting smoke test..." >> $OUTPUT_FILE

#######################
### ZOOM IN RESULTS ###
#######################
# Zoom In 4f, relax
python src/vvn/verify.py zoom_in relax 1 4 1800 751 >> $OUTPUT_FILE 2>&1

# Zoom In 8f, relax
python src/vvn/verify.py zoom_in relax 1 8 1800 751 >> $OUTPUT_FILE 2>&1

# Zoom In 16f, relax
python src/vvn/verify.py zoom_in relax 1 16 1800 751 >> $OUTPUT_FILE 2>&1

# Zoom In 4f, approx
python src/vvn/verify.py zoom_in approx 1 4 1800 751 >> $OUTPUT_FILE 2>&1

# Zoom In 8f, approx
python src/vvn/verify.py zoom_in approx 1 8 1800 751 >> $OUTPUT_FILE 2>&1


########################
### ZOOM OUT RESULTS ###
########################
# Zoom Out 4f, relax
python src/vvn/verify.py zoom_out relax 1 4 1800 624 >> $OUTPUT_FILE 2>&1

# Zoom Out 8f, relax
python src/vvn/verify.py zoom_out relax 1 8 1800 624 >> $OUTPUT_FILE 2>&1

# Zoom Out 16f, relax
python src/vvn/verify.py zoom_out relax 1 16 1800 624 >> $OUTPUT_FILE 2>&1

# Zoom Out 4f, approx
python src/vvn/verify.py zoom_out approx 1 4 1800 624 >> $OUTPUT_FILE 2>&1

# Zoom Out 8f, approx
python src/vvn/verify.py zoom_out approx 1 8 1800 624 >> $OUTPUT_FILE 2>&1


#######################
#### GTSRB RESULTS ####
#######################
# GTSRB 4f, relax
python src/vvn/verify.py gtsrb relax 1 4 1800 12597 >> $OUTPUT_FILE 2>&1

# GTSRB 8f, relax
python src/vvn/verify.py gtsrb relax 1 8 1800 12597 >> $OUTPUT_FILE 2>&1

# GTSRB 16f, relax
python src/vvn/verify.py gtsrb relax 1 16 1800 12597 >> $OUTPUT_FILE 2>&1

# GTSRB 4f, approx
python src/vvn/verify.py gtsrb approx 1 4 1800 12597 >> $OUTPUT_FILE 2>&1

# GTSRB 8f, approx
python src/vvn/verify.py gtsrb approx 1 8 1800 12597 >> $OUTPUT_FILE 2>&1

# GTSRB 16f, approx
python src/vvn/verify.py gtsrb approx 1 16 1800 12597 >> $OUTPUT_FILE 2>&1


#######################
### STMNIST RESULTS ###
#######################
# STMNIST 16f, relax
python src/vvn/verify.py stmnist relax 1 16 1800 311 >> $OUTPUT_FILE 2>&1

# STMNIST 32f, relax
python src/vvn/verify.py stmnist relax 1 32 1800 311 >> $OUTPUT_FILE 2>&1

# STMNIST 64f, relax
python src/vvn/verify.py stmnist relax 1 64 1800 311 >> $OUTPUT_FILE 2>&1

# STMNIST 16f, approx
python src/vvn/verify.py stmnist approx 1 16 1800 311 >> $OUTPUT_FILE 2>&1

# STMNIST 32f, approx
python src/vvn/verify.py stmnist approx 1 32 1800 311 >> $OUTPUT_FILE 2>&1

# STMNIST 64f, approx
python src/vvn/verify.py stmnist approx 1 64 1800 311 >> $OUTPUT_FILE 2>&1

python src/vvn/verify.py zoom_in relax 1 4 1800 751

echo -e "\n\n**********************************************"
echo " Smoke test complete. "
Expand Down
11 changes: 3 additions & 8 deletions code/nnv/examples/Submission/FORMALISE2025/run_subset_vvn.sh
View file
Original file line number Diff line number Diff line change
@@ -1,14 +1,9 @@
#!/bin/bash
# TABLE II (GET ALL VERIFICATION RESULTS)
python src/run.py subset # MNIST Video
python src/run_gtsrb.py subset # GTSRB
python src/run_stmnist.py subset # STMNIST
python src/analysis/make_table2.py


# FIGURE 8 (COMPARISON OF AVERAGE RUNTIME)
python src/analysis/make_plots.py
# Reproduce the first row of table 2.

# TABLE II (GET ALL VERIFICATION RESULTS)
python src/run.py --subset # MNIST Video

# FIGURE 7 (REACHABLE OUTPUT PLOTS)
python src/generate_output_plots.py
Loading

0 comments on commit 4abdc23

Please sign in to comment.