Alpha Matting Algorithm Performance

Performance comparison of alpha matting algorithms across different metrics. Lower values are better for all metrics.

Data sourced from alphamatting.com, a comprehensive benchmark for alpha matting algorithms.

Evaluation Metrics for Image Matting#

The paper "A Perceptually Motivated Online Benchmark for Image Matting" introduces four evaluation metrics for assessing image matting quality. They are:

1. Sum of Absolute Differences (SAD)#

  • Traditional metric that computes the pixel-wise absolute error between the estimated alpha matte and the ground truth.
  • Measures the overall deviation but is not always correlated with perceived visual quality.

2. Mean Squared Error (MSE)#

  • Another pixel-based metric that calculates the average squared difference between predicted and ground truth alpha values.
  • Like SAD, it may fail to reflect visual artifacts (e.g., ConnectivityError issues).

3. Gradient Error#

  • A perceptually motivated metric focusing on the preservation of alpha matte gradients (important for fine details like hair or edges).
  • It measures the difference between the gradients of the estimated and ground truth mattes:
    Gradient Error=∑i(∇αi−∇αi∗)q\text{Gradient Error} = \sum_i (\nabla \alpha_i - \nabla \alpha^*_i)^q
  • Shown to correlate better with human perception of smoothness and edge quality.

4. ConnectivityError#

  • Designed to penalize disconnects or fragmentation of the foreground object.
  • It evaluates how well the ConnectivityError of regions in the alpha matte matches the ground truth using a ConnectivityError function Ï•(αi,Ω)\phi(\alpha_i, \Omega).
  • Particularly important for assessing structural integrity (e.g., connected hair strands).

Key Findings:#

  • SAD and MSE have weak correlation with human judgment (correlation ~0.28–0.51).
  • Gradient and ConnectivityError measures correlate much better (correlation ~0.75 with human ratings).
  • Different algorithms rank differently depending on the metric (e.g., Random Walk matting excels in ConnectivityError, Closed-form matting scores best for SAD/MSE/Gradient).
Algorithm / Model
SAD
MSE
Gradient
Connectivity
TransMatting
3.1
2.3
2.8
6.2
TMFNet
3.4
4.0
3.9
15.0
IamAlpha
4.4
4.8
4.8
12.8
LFPNet
4.7
4.1
2.8
11.2
TIMI-Net
4.9
5.5
6.2
8.8
SIM
5.8
6.3
6.2
9.6
PIIAMatting
9.1
11.3
11.8
17.7
HDMatt
10.5
10.3
8.7
29.9
AdaMatting
12.6
12.8
12.3
22.3
A2U Matting
12.8
14.6
11.3
26.3
SampleNet Matting
13.0
13.2
14.2
25.4
FGI Matting
13.1
N/A
N/A
N/A
GCA Matting
14.2
14.5
12.8
21.7
ATNet Matting
15.4
14.9
18.1
24.9
VDRN Matting
15.8
17.4
18.7
22.4
Deep Matting
16.1
18.1
22.5
20.0
Information-flow matting
17.9
18.7
25.3
30.2
IndexNet Matting
19.3
22.0
17.6
24.7
DCNN Matting
20.0
19.2
23.7
26.7
AlphaGAN
20.9
23.2
22.4
35.8
Context-aware Matting
22.8
16.7
13.7
25.2
Three-layer graph matting
25.0
24.4
27.3
26.3
ATPM Matting
27.3
29.4
31.5
34.0
Three Stages Matting
28.5
25.7
31.4
25.9
CSC Matting
30.0
34.1
34.8
37.1
LNSP Matting
30.1
26.3
28.9
19.4
Graph-based sparse matting
31.1
30.7
28.1
40.0
KL-Divergence Based Sparse Sampling
31.2
29.8
28.2
40.0
Patch-based Matting
31.3
28.1
28.1
33.2
TSPS-RV Matting
32.6
31.6
35.4
31.5
Iterative Transductive Matting
33.2
37.1
41.3
50.0
SVR Matting
33.8
30.4
32.3
26.5
Comprehensive sampling
33.8
31.4
29.4
42.8
Comprehensive Weighted Color and Texture
34.3
32.2
34.4
37.8
Sparse coded matting
34.8
34.7
31.8
38.5
LocalSamplingAndKnnClassification
36.3
34.5
38.7
27.8
Weighted Color and Texture Matting
37.0
36.4
40.4
38.4
CCM
37.3
30.3
32.5
18.7
LNCLM matting
37.3
37.1
39.7
37.4
Shared Matting
37.5
37.5
33.5
37.7
Global Sampling Matting
39.8
36.3
32.8
34.5
SRLO Matting
40.8
42.5
42.6
46.3
Segmentation-based matting
41.1
40.4
32.6
34.3
Improved color matting
42.3
40.8
33.3
27.6
KNN Matting
42.6
37.5
44.3
33.9
Local Spline Regression (LSR)
42.7
44.0
44.2
28.9
Global Sampling Matting (filter version)
42.9
44.5
38.5
46.8
Learning Based Matting
44.0
43.7
43.3
30.8
LMSPIR
45.2
44.9
43.5
43.2
Shared Matting (Real Time)
45.6
47.0
48.5
45.5
Closed-Form Matting
46.0
44.6
44.1
19.2
Improving Sampling Criterion
49.7
41.6
42.5
34.2
Cell-based matting Laplacian
50.3
49.8
51.0
29.0
Large Kernel Matting
50.5
48.2
46.3
25.6
Robust Matting
51.2
49.6
45.3
45.0
SPS matting
53.3
44.5
40.7
42.1
High-res matting
54.2
51.5
46.9
35.3
Transfusive Weights
54.7
51.3
54.2
12.8
Random Walk Matting
57.9
57.4
55.5
8.7
Geodesic Matting
59.1
N/A
N/A
52.2
Iterative BP Matting
60.2
N/A
N/A
52.5
Easy Matting
60.6
N/A
N/A
N/A
Improved Bayesian
61.1
N/A
N/A
N/A
Bayesian Matting
62.4
N/A
N/A
N/A
Poisson Matting
64.8
N/A
N/A
N/A

Showing 65 algorithms with their performance metrics.

Metrics: SAD (Sum of Absolute Differences), MSE (Mean Squared Error), Gradient (Gradient Error), Connectivity (Connectivity Error). Lower values indicate better performance.

Color coding: Values are colored from green (best) to red (worst) within each column, with yellow and orange representing intermediate performance levels.