From 2eab883d802258fd29edb46a87765d0a253c1ae8 Mon Sep 17 00:00:00 2001 From: ryanxing Date: Tue, 9 Feb 2021 16:50:53 +0800 Subject: [PATCH] Polish README --- README.md | 47 ++++++++++++++++++++++++++++------------------- 1 file changed, 28 insertions(+), 19 deletions(-) diff --git a/README.md b/README.md index 3925635..a5638ab 100644 --- a/README.md +++ b/README.md @@ -13,7 +13,7 @@ ## 0. Background -Official repository of [[*MFQE 2.0: A New Approach for Multi-frame Quality Enhancement on Compressed Video*]](http://arxiv.org/abs/1902.09707), TPAMI 2019. [[速览 (中文)]](https://github.com/RyanXingQL/Blog/blob/master/posts/mfqev2.md) +Official repository of [*MFQE 2.0: A New Approach for Multi-frame Quality Enhancement on Compressed Video*](http://arxiv.org/abs/1902.09707), TPAMI 2019. [[速览 (中文)]](https://github.com/RyanXingQL/Blog/blob/master/posts/mfqev2.md) - The first **multi-frame** quality enhancement approach for compressed videos. - The first to consider and utilize the **quality fluctuation** feature of compressed videos. @@ -21,16 +21,16 @@ Official repository of [[*MFQE 2.0: A New Approach for Multi-frame Quality Enhan ![Demo](https://user-images.githubusercontent.com/34084019/105737566-10a31b00-5f71-11eb-9d2c-19780ab94ab1.png) -Feel free to contact: ryanxingql@gmail.com. +Feel free to contact: . ## 1. Pre-request ### 1.1. Environment -- Python 3.5 -- TensorFlow 1.8 (1.13/14 is ok but with warnings) -- TFLearn -- Scikit-image (for calculating PSNR and SSIM) +- PYTHON 3.5 +- TENSORFLOW 1.8 (1.13/14 is ok but with warnings) +- TFLEARN +- SCIKIT-IMAGE (for calculating PSNR and SSIM) ### 1.2. Dataset @@ -39,8 +39,8 @@ We open-source our lossless video dataset, including 108 videos for training and Download link: [[DropBox]](https://www.dropbox.com/sh/d04222pwk36n05b/AAC9SJ1QypPt79MVUZMosLk5a?dl=0) [[百度网盘 (mfqe)]](https://pan.baidu.com/s/1oBZf75bFGRanLmQQLAg4Ew)
- How to compress videos +

We have also provided the video compression toolbox in the dataset link. @@ -71,6 +71,7 @@ MFQEv2_dataset/ └── README.md ``` +

## 2. Test @@ -117,23 +118,24 @@ You can also train your own model by fine-tuning the open-source pre-trained mod Unfortunately, the above training codes are written in different times and devices. To run these code properly, you may have to change some paths. -I'm sorry about the coarse training code of my first scientific work MFQEv2. If you're finding a more robust work for practical use, see my implementation of [[STDF (AAAI 2020)]](https://github.com/RyanXingQL/STDF-PyTorch). +I'm sorry about the coarse training code of my first scientific work MFQEv2. If you're finding a more robust work for practical use, see my implementation of [STDF (AAAI 2020)](https://github.com/RyanXingQL/STDF-PyTorch). ## 4. Q&A
- How to enhanced PQFs +

In MFQEv2, PQFs are also enhanced using their neighboring PQFs. Note that the models for PQF enhancement and non-PQF enhancement are trained separately. In MFQEv1, PQFs are enhanced using other image enhancement approaches. +

- How to detect PQFs +

In the training stage, we use **ground truth PQF labels**, i.e., labels that generated by PSNR values. See our paper for more details. @@ -163,21 +165,23 @@ As we can see, the whole process is a bit complicated: Therefore, we omit the compression, IQA and detection processes, but instead provide you with the pre-generated 18 PQF labels as well as two simpler options as mentioned above. +

- Assign approximate QP label (If needed) +

There may exist frames with different QPs in one video. You can prepare a `npy` file that **records the QP of each frame in one video**, and store it in folder `data/PQF_label` as `ApprQP_VideoName.npy`. Notice that we have only 5 models with QP22, 27, 32, 37, 42, so we should record the nearest QP for each frame. For example, if the QPs for 4 frames are: `21,28,25,33`, then we should record: `22,27,27,32`. That's why we call it "approximate QP label". +

- Modify unqualified label +

In our MFQE approach, each non-PQF should be enhanced with the help of its neighboring two PQFs (previous one + subsequent one). @@ -190,41 +194,46 @@ Our solution: we simply let themselves to be the pre-PQF and sub-PQF, to manage Similarly, the first PQF has no previous PQF, and the last PQF has no subsequent PQF. The first PQF serves as the previous PQF for itself, and the last PQF serves as the subsequent PQF for itself. +

- Two MF-CNN models are similar +

There are two different models in `net_MFCNN.py`. `network2` is for QP = 22, 27, 32 and `network1` for QP = 37, 42. Correspondingly, there are two types of pre-trained model. The performances of these two networks are close. Feel free to use them. +

- OOM when testing class A videos +

Even with a 2080Ti GPU, we cannot process `2560x1600` frames (i.e., test sequences *Traffic* and *PeopleOnStreet*) directly. We simply cut them into 4 patches for enhancement, combine the enhanced patches, and then calculate PSNR and SSIM. For simplicity, the patching and combination processes are omitted in the test code. +

- Deal with scene switch +

There may exist multiple scenes in one video. Frames in different scenes should not be fused. In this case, we can use SSIM to detect scene switch, and then cut the video into a few clips. Luckily, it seems that no scene switch exists in the 18 test sequences. +

- Enhance black frame +

Enhancing black frames or other "plane" frames (all pixel values are the same) may lead to inf PSNR. Our solution: 1. If the middle frame is plane, skip it (do not enhance it). 2. If the pre- or sub-PQF is plane, simply let the middle frame itself to be its pre-PQF and sub-PQF for enhancement. +

## 5. License & Citation @@ -249,11 +258,11 @@ You can **use, redistribute, and adapt** the material for **non-commercial purpo ## 6. See more -- [[PyTorch implementation of STDF (AAAI 2020)]](https://github.com/RyanXingQL/STDF-PyTorch) - - A **simple** yet **effective** video quality enhancement network. +- [PyTorch implementation of STDF (AAAI 2020)](https://github.com/RyanXingQL/STDF-PyTorch) + - A **simple** and **effective** video quality enhancement network. - Adopt **feature alignment** by multi-frame **deformable convolutions**, instead of motion estimation and motion compensation. -- [[RBQE (ECCV 2020)]](https://github.com/RyanXingQL/RBQE) +- [RBQE (ECCV 2020)](https://github.com/RyanXingQL/RBQE) - A **single blind** enhancement model for HEVC/JPEG-compressed images with a **wide range** of Quantization Parameters (QPs) or Quality Factors (QFs). - A **multi-output dynamic** network with **early-exit** mechanism for easy input. - A **Tchebichef-moments** based **NR-IQA** approach for early-exit decision. This IQA approach is highly interpretable and sensitive to blocking energy detection.