Background Removal Quality Evaluation
How do we know if an algorithm is doing a good job of removing the background from an image? Let’s explore it qualitatively and quantitatively.
Evaluating Qualitatively
There are a few factors you can consider to evaluate the quality of an image background removal. Here are some things to look for:
- Check if the edges of the foreground object(s) clean and well-defined.
- Check if the the transparency is preserved, if the foreground object has such features. Example: Car windows, glasses.
- Check if the fine details are preserved (like hairs, fur).
A good method would be adding a solid background color which has a strong contrast with the foreground object. By doing so, the flaws would be more visible.
Evaluating Quantitatively
To come up with a numeric value which is representing the overall quality, we build a dataset (images) and corresponding alpha channels (ground truth, provided by an expert) and we compare the resulting alpha channel and the ground truth alpha channel.
To be able to report the quality of an alpha matte, we can do a pixel-wise comparison, we can compare the fine details, or check the structural similarities. Here are the corresponding metrics:
Alpha Matte Error
This metric measures the difference between the predicted alpha matte and the ground-truth alpha matte for each pixel in the image. A lower alpha error indicates that the model is more accurate in its predictions of the transparency of each pixel.
Structural Similarity (SSIM)
This metric measures the similarity between the predicted alpha matte and the ground-truth alpha matte by comparing the structures, textures, and patterns in the images. A higher SSIM score indicates that the model is better able to preserve the structural information in the alpha matte.
Gradient Error
This metric measures the difference between the gradient of the predicted alpha matte and the gradient of the ground-truth alpha matte. A lower gradient error indicates that the model is better able to preserve the edges and transitions in the alpha matte. A research (A Perceptually Motivated Online Benchmark for Image Matting) shows that this metric has a high positive correlation with human perception.
Intuitively, these metrics evaluate how well the model is able to accurately predict the transparency of each pixel in the image while preserving the structures, textures, and edges in the alpha matte. A good model will have a low alpha error, high SSIM, and low gradient error, indicating that it is able to accurately predict the transparency of each pixel and preserve the important features of the alpha matte.
Summary
The evaluation of image background removal techniques can be approached from both a qualitative and quantitative perspective. Qualitatively, the visual appearance and overall effectiveness of the removal technique can be assessed by human reviewers. Quantitatively, metrics such as the alpha matte error, gradient error, and SSIM metrics provide good insights. Both of these methods are important for a comprehensive evaluation.
References
Christoph Rhemann, Carsten Rother, Jue Wang, Margrit Gelautz, Pushmeet Kohli, Pamela Rott (2009).