MagicInpainter 3.0 is image processing tool for inpaint, mask selection and objects removal of small and medium sized objects from photos. Starting with version 3.0 GPU optimized image inpaint algorithms are used.
Inpaint (or image reconstruction) is a process of filling in the missing parts of images in a natural way, preserving textures and background so that a person is not able to spot the difference.
Historically, there are two main groups of modern image inpaint methods:
- Image processing gradient algorithms (called also traditional)
- Generative Deep Learning using neural networks with CNNs, etc. (called also AI methods)
As shown below gradient methods in OpenCV are good only for small linear scratches, DL also has quality issues. MagicInpainter, on the other hand, employs statistical methods, which can offer great flexibility in solving various problems. However, compared to the legacy gradient-based methods, one drawback of statistical approaches is the potentially high computational complexity. Therefore, significant challenge lies in optimizing performance.
MagicInpainter 3.0 fills in the pixels from the missing noisy regions using the data from the undamaged area in the same image, also called low noise area. Training, preprocessing or additional images are not necessary. Inpain\t is done only with extracting valid image data from the image area with low noise and then using this data to fill in areas with high noise. This is done with assigning each pixel or group of several pixels to the so-called image keys or as some researchers call them image features. Image keys are extracted from the non-noisy pixels neighborhood and then used to find the best match for the corresponding noisy pixels. Thus, like many other problems for inpaint we use a feature space composed from the collected before valid image keys. So the process of extracting image keys can be called constructing the feature space and inpaint is then can be called the process of decreasing image noise using this feature space.
Some of the parameters of the inpaint (available in Settings):
-
Low noise and high noise area We first separate image into two regions – low noise and high noise, then we take pixels from the low noise area and fills in the pixels from the high noise area going from outside to inside. Choosing the objects for removal is done manually with applying a mask, MagicInpainter 3.0 has Eraser Button allowing to apply such mask over any part of the image.
-
Max. Noise In some cases pixels close to the image edges or with too many noise around can distort results. This parameter is given in percents and controls which feature keys from the low noise area can be used during the inpaint, valid image keys are these for which is fullfiled:
-
Zoom In/Out For images with higher resolution generated feature keys are too many, it makes it more difficult and too computationally expensive to find a best match. To resolve this MagicInpainter provides the Zoom In and Zoom Out buttons and one parameter called Max. Distance.
-
Inpaint Radius This radius controls the size of the area around each noise pixel, it would be used from the inpaint filter. Usually this is called filter kernel radius and is given in pixels. Filter has kernel matrix with size [2R+1] X [2R+1]. Using larger inpaint radius means that pixels further away will participate in the reconstruction but just increasing R does not always means better quality.
MagicInpainter 3.0 uses inpaint filters to find the best replacement candidates for the noisy pixels . For the calculation of inpaint pixels replacements sometimes are used gradient based methods, the most well-known of these is the Navier-Stokes method, described in the paper "Navier-Stokes, Fluid Dynamics, and Image and Video Inpainting", by Bertalmio, Marcelo, Andrea L. Bertozzi, and Guillermo Sapiro, published in 2001. This was further improved with the so called Fast Marching Method, representing successive fill-in from outside to inside of the noisy region, described in another famous paper "An Image Inpainting Technique Based on the Fast Marching Method" by Alexandru Telea, published in 2004 [See]. These are the only inpaint algorithms provided currently by OpenCV (See https://docs.opencv.org/3.4/df/d3d/tutorial_py_inpainting.html). My tests however showed that these methods, even as sufficiently fast are good only for small linear scratches and failed to reproduce the picture patterns even for small rectangular inpaint regions.
One other approach is to use GPU optimized variations of the Normalized Cross Correlation calculation methods. These methods are widely used in the Forensic Tampering detection and elsewhere, [See]. The idea is to divide the image in blocks and to estimate the cross correlation as described in the paper "A Scaling Robust Copy-Paste Tampering Detection for Digital Image Forensics" [See], with the normalized cross correlation coefficient as given by:
Similar approach can be applied but in reverse, instead of searching for forged content to reconstruct already known masked regions. However, one drawback is that it is quite computational expensive, even with the proposed optimizations. One other optimization is called the Fast Template Matching method as described in this paper , but there are many others. An open research in MagicInpainter 3.0 is how to further improve the inpaint algorithms to work with bigger resolutions and inpaint radius (see Future Work section below).
For removing medium sized object from some real-life photos like beach photos, inpaint quality is comparable with the AI apps available on net (after several passes):
 |
 |
 |
 |
|---|---|---|---|
| Original Image, 862x400 | MagicInpainter 3.0, R15 | SnapEdit, pass 2 | Adobe PS 2023 |
 |
 |
 |
 |
|---|---|---|---|
| Original Image, 750x500 | MagicInpainter 3.0, R25 | SnapEdit, pass 2 | Adobe PS 2023 |
See Also:
Tests with Textures
Image Tampering Test
MagicInpainter 3.0 works very well with textures and other pictures but data from other photos is not used and reconstruction of complex features, especially in photos with large sizes, may not be correct. For images with higher resolution, when noise area is large, reconstruction would be too slow, several repeats of “zoom and mask” would be necessary to fill in the missing content.
In comming releases there would be several improvements:
-
GPU performance - several optimizations are possible for the GPU to work even faster, also search algorithms can be greately improved
-
Images with larger resolution Optimization algorithms used in MagicInpainter 3.0 become sometimes too slow and unstable for images with high resolution (in these cases Zoom In/Out buttons can be used). One other limitation is that inpaint radius can not exceed 64 pixels.
-
Processing several images and video MagicInpainter 3.0 is currently limited to single photos.
-
Use AI and deep learning to speed up algorithms and optimal inpaint radius selection Assist inpaint algorithms with AI for classification, automation of algorithms type and optimal radius size selection. Used here inpaint algorithms can be combined with various neural networks models, especially in the cases when inpaint involves complicated features and big resolution, speed can be also significantly improved.
Following research areas are also interesting for me:
- Video reconstruction
- Textures and artificial images generation
- Noise removal
- Noise Detection
- Deblur
- Image Tampering Analysis
- Superposition
- Superresolution
- Stable Diffusion
Noise removal for example can be done with improving image quality using partial inpaint and weights coefficients rather than filling in the missing data with masks having only two values.
There is also a Python library exporting the used inpaint methods:
https://github.com/antonmilev/magicinpaintpython
Version 3.1.190 (from July 2023)
- Bugfixes
- Limitation of the image sizes is removed!
- The new GPU Fast patch-matching algorithm is up to 5 times faster which allows processing of HD images :
MagicInpainter 3.0 can be installed from:
| CPU Version (3.1.190) | MagicInpainter.msi |
|---|---|
| GPU Version (3.1.190) | MagicInpainterGpu.msi |
| User Manual | Magic Inpainter 3.0.pdf |
MagicInpainter 3.0 is free for personal and non-comercial use. It is provided in two versions for CPU amd GPU. I strongly recommend using the GPU version, it works better and faster, But for this you need NVIDIA GTX or RTX video card with CUDA support (10x,11x,12x). I tested the GPU app successfully on (GTX1050,RTX2070,RTX3050TI and RTX3060), with CUDA 10.1, 10.2 and 11.8 runtime. If you install the GPU version there is option in Settings to switch back to CPU.
If you do not have NVIDIA GPU card and installed CUDA driver (GTX or RTX) the GPU version of the application will not work!
MagicInpainter 3.0 and all related source and documentation is provided under GNU GPLv3 public license.
Supported OS: Window 7/Vista/10/11 x64
For introduction please see: https://youtu.be/RXDDdDnvNPY