Image Background Removal Quality Evaluation

Author: Imran Kocabiyik

Image without background

How do we know if we've done a good job of removing the background from an image? In this blog post, we'll explore some ways to evaluate the quality of image background removal, both qualitatively and quantitatively.

Evaluating Qualitatively

There are a few factors you can consider to evaluate the quality of an image background removal. Here are some things to look for:

Checking the edges

Are the edges of the foreground object(s) clean and well-defined, or are there jagged or blurry edges? A good background removal should have clean and smooth edges around the foreground object(s).

Checking the Transparent Area

If the foreground object has transparency, it should be handled carefully. A good way to see is by compositing it into another background or adding a suitable solid color background.

Checking Foreground Object Details

It might be difficult to remove the background of an image when the foreground object has fine details (like hairs, and fur).

Evaluating Quantitatively

You need a quantitative evaluation if you are developing a tool for background removal. In this case, we focus on the alpha channels. An alpha channel represents the transparency of each pixel.

To come up with a numeric value which is representing the overall quality, you build a dataset (images) and corresponding alpha channels (ground truth, provided by an expert) and you compare the resulting alpha channel and the ground truth alpha channel.

To be able to report the quality of an alpha matte, we can do a pixel-wise comparison, we can compare the fine details, or check the structural similarities. Here are the corresponding metrics:

Alpha Matte Error

This metric measures the difference between the predicted alpha matte and the ground-truth alpha matte for each pixel in the image. A lower alpha error indicates that the model is more accurate in its predictions of the transparency of each pixel.

Structural Similarity (SSIM)

This metric measures the similarity between the predicted alpha matte and the ground-truth alpha matte by comparing the structures, textures, and patterns in the images. A higher SSIM score indicates that the model is better able to preserve the structural information in the alpha matte.

Gradient Error

This metric measures the difference between the gradient of the predicted alpha matte and the gradient of the ground-truth alpha matte. A lower gradient error indicates that the model is better able to preserve the edges and transitions in the alpha matte. Researches show that this metric is a high positive correlation with human perception.

An illustration of image gradients, after the Sobel filters were applied on an alpha matte

An illustration of image gradients, after the Sobel filters were applied on an alpha matte

Intuitively, these metrics evaluate how well the model is able to accurately predict the transparency of each pixel in the image while preserving the structures, textures, and edges in the alpha matte. A good model will have a low alpha error, high SSIM, and low gradient error, indicating that it is able to accurately predict the transparency of each pixel and preserve the important features of the alpha matte.

Summary

The evaluation of image background removal techniques can be approached from both a qualitative and quantitative perspective. Qualitatively, the visual appearance and overall effectiveness of the removal technique can be assessed by human reviewers. Quantitatively, metrics such as the alpha matte error, gradient error, and SSIM metrics provide good insights. Both of these methods are important for a comprehensive evaluation.