Skip to content

Commit

Permalink
Polish README
Browse files Browse the repository at this point in the history
  • Loading branch information
ryanxingql committed Feb 9, 2021
1 parent 85a0151 commit 2eab883
Showing 1 changed file with 28 additions and 19 deletions.
47 changes: 28 additions & 19 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,24 +13,24 @@

## 0. Background

Official repository of [[*MFQE 2.0: A New Approach for Multi-frame Quality Enhancement on Compressed Video*]](http://arxiv.org/abs/1902.09707), TPAMI 2019. [[速览 (中文)]](https://github.com/RyanXingQL/Blog/blob/master/posts/mfqev2.md)
Official repository of [*MFQE 2.0: A New Approach for Multi-frame Quality Enhancement on Compressed Video*](http://arxiv.org/abs/1902.09707), TPAMI 2019. [[速览 (中文)]](https://github.com/RyanXingQL/Blog/blob/master/posts/mfqev2.md)

- The first **multi-frame** quality enhancement approach for compressed videos.
- The first to consider and utilize the **quality fluctuation** feature of compressed videos.
- Enhance low-quality frames using **neighboring high-quality** frames.

![Demo](https://user-images.githubusercontent.com/34084019/105737566-10a31b00-5f71-11eb-9d2c-19780ab94ab1.png)

Feel free to contact: ryanxingql@gmail.com.
Feel free to contact: <ryanxingql@gmail.com>.

## 1. Pre-request

### 1.1. Environment

- Python 3.5
- TensorFlow 1.8 (1.13/14 is ok but with warnings)
- TFLearn
- Scikit-image (for calculating PSNR and SSIM)
- PYTHON 3.5
- TENSORFLOW 1.8 (1.13/14 is ok but with warnings)
- TFLEARN
- SCIKIT-IMAGE (for calculating PSNR and SSIM)

### 1.2. Dataset

Expand All @@ -39,8 +39,8 @@ We open-source our lossless video dataset, including 108 videos for training and
Download link: [[DropBox]](https://www.dropbox.com/sh/d04222pwk36n05b/AAC9SJ1QypPt79MVUZMosLk5a?dl=0) [[百度网盘 (mfqe)]](https://pan.baidu.com/s/1oBZf75bFGRanLmQQLAg4Ew)

<details>

<summary><b>How to compress videos</b></summary>
<p>

We have also provided the video compression toolbox in the dataset link.

Expand Down Expand Up @@ -71,6 +71,7 @@ MFQEv2_dataset/
└── README.md
```

</p>
</details>

## 2. Test
Expand Down Expand Up @@ -117,23 +118,24 @@ You can also train your own model by fine-tuning the open-source pre-trained mod

Unfortunately, the above training codes are written in different times and devices. To run these code properly, you may have to change some paths.

I'm sorry about the coarse training code of my first scientific work MFQEv2. If you're finding a more robust work for practical use, see my implementation of [[STDF (AAAI 2020)]](https://github.com/RyanXingQL/STDF-PyTorch).
I'm sorry about the coarse training code of my first scientific work MFQEv2. If you're finding a more robust work for practical use, see my implementation of [STDF (AAAI 2020)](https://github.com/RyanXingQL/STDF-PyTorch).

## 4. Q&A

<details>

<summary><b>How to enhanced PQFs</b></summary>
<p>

In MFQEv2, PQFs are also enhanced using their neighboring PQFs. Note that the models for PQF enhancement and non-PQF enhancement are trained separately.

In MFQEv1, PQFs are enhanced using other image enhancement approaches.

</p>
</details>

<details>

<summary><b>How to detect PQFs</b></summary>
<p>

In the training stage, we use **ground truth PQF labels**, i.e., labels that generated by PSNR values. See our paper for more details.

Expand Down Expand Up @@ -163,21 +165,23 @@ As we can see, the whole process is a bit complicated:

Therefore, we omit the compression, IQA and detection processes, but instead provide you with the pre-generated 18 PQF labels as well as two simpler options as mentioned above.

</p>
</details>

<details>

<summary><b>Assign approximate QP label (If needed)</b></summary>
<p>

There may exist frames with different QPs in one video. You can prepare a `npy` file that **records the QP of each frame in one video**, and store it in folder `data/PQF_label` as `ApprQP_VideoName.npy`.

Notice that we have only 5 models with QP22, 27, 32, 37, 42, so we should record the nearest QP for each frame. For example, if the QPs for 4 frames are: `21,28,25,33`, then we should record: `22,27,27,32`. That's why we call it "approximate QP label".

</p>
</details>

<details>

<summary><b>Modify unqualified label</b></summary>
<p>

In our MFQE approach, each non-PQF should be enhanced with the help of its neighboring two PQFs (previous one + subsequent one).

Expand All @@ -190,41 +194,46 @@ Our solution: we simply let themselves to be the pre-PQF and sub-PQF, to manage

Similarly, the first PQF has no previous PQF, and the last PQF has no subsequent PQF. The first PQF serves as the previous PQF for itself, and the last PQF serves as the subsequent PQF for itself.

</p>
</details>

<details>

<summary><b>Two MF-CNN models are similar</b></summary>
<p>

There are two different models in `net_MFCNN.py`. `network2` is for QP = 22, 27, 32 and `network1` for QP = 37, 42. Correspondingly, there are two types of pre-trained model. The performances of these two networks are close. Feel free to use them.

</p>
</details>

<details>

<summary><b>OOM when testing class A videos</b></summary>
<p>

Even with a 2080Ti GPU, we cannot process `2560x1600` frames (i.e., test sequences *Traffic* and *PeopleOnStreet*) directly. We simply cut them into 4 patches for enhancement, combine the enhanced patches, and then calculate PSNR and SSIM. For simplicity, the patching and combination processes are omitted in the test code.

</p>
</details>

<details>

<summary><b>Deal with scene switch</b></summary>
<p>

There may exist multiple scenes in one video. Frames in different scenes should not be fused. In this case, we can use SSIM to detect scene switch, and then cut the video into a few clips. Luckily, it seems that no scene switch exists in the 18 test sequences.

</p>
</details>

<details>

<summary><b>Enhance black frame</b></summary>
<p>

Enhancing black frames or other "plane" frames (all pixel values are the same) may lead to inf PSNR. Our solution:

1. If the middle frame is plane, skip it (do not enhance it).
2. If the pre- or sub-PQF is plane, simply let the middle frame itself to be its pre-PQF and sub-PQF for enhancement.

</p>
</details>

## 5. License & Citation
Expand All @@ -249,11 +258,11 @@ You can **use, redistribute, and adapt** the material for **non-commercial purpo

## 6. See more

- [[PyTorch implementation of STDF (AAAI 2020)]](https://github.com/RyanXingQL/STDF-PyTorch)
- A **simple** yet **effective** video quality enhancement network.
- [PyTorch implementation of STDF (AAAI 2020)](https://github.com/RyanXingQL/STDF-PyTorch)
- A **simple** and **effective** video quality enhancement network.
- Adopt **feature alignment** by multi-frame **deformable convolutions**, instead of motion estimation and motion compensation.

- [[RBQE (ECCV 2020)]](https://github.com/RyanXingQL/RBQE)
- [RBQE (ECCV 2020)](https://github.com/RyanXingQL/RBQE)
- A **single blind** enhancement model for HEVC/JPEG-compressed images with a **wide range** of Quantization Parameters (QPs) or Quality Factors (QFs).
- A **multi-output dynamic** network with **early-exit** mechanism for easy input.
- A **Tchebichef-moments** based **NR-IQA** approach for early-exit decision. This IQA approach is highly interpretable and sensitive to blocking energy detection.

0 comments on commit 2eab883

Please sign in to comment.