Alpha matting vs image segmentation#

Segmentation decides which pixels belong to the subject. Alpha matting decides how much each pixel belongs to it. That difference is invisible on a product with a sharp outline and obvious on hair, fur, glass and smoke.

Binary mask vs alpha matte#

A segmentation mask has two values per pixel: foreground or background. An alpha matte has a value between 0 and 1: 0 is background, 1 is foreground, and anything in between is a pixel that mixes both. Placing a cutout on a new background is the compositing equation below.

Compositing

# C: composited pixel, F: foreground colour, B: new background colour
# alpha is in [0, 1] for matting, and only 0 or 1 for a binary mask
C = alpha * F + (1 - alpha) * B

With a binary mask the in-between pixels have to be rounded to 0 or 1. On a hair strand that gives a hard, jagged edge or a halo of the old background. Many matting methods take a trimap (foreground, background, unknown) as a hint for where to estimate opacity; automatic background removal models predict the matte without one.

How edge quality is measured#

Bench v0 compares each model's predicted alpha matte with a ground-truth matte on 60 real photos, both scored at a 1024 px shorter side (never upscaled). Three numbers, all lower-is-better:

  • MAE: mean absolute alpha error over every pixel. Large flat areas (solid background, solid object interior) are easy, so they dominate this average.
  • MGE: error between the Sobel gradient magnitudes of the prediction and the ground truth. It reacts to edges that are too hard, too soft or misplaced.
  • Band MAE: MAE only inside a 10 px band (at 1024 px) around the matte edge, where the matting problem actually is.

The band is where errors concentrate: across all 14 models on Bench v0, band MAE is between 1.53× and 2.30× the overall MAE, so edge pixels carry more error per pixel than the image as a whole.

Bench v0 scores, sorted by edge error#

Overall (MAE) and edge (MGE) rankings do not always agree. BiRefNet General Lite is 6th by MAE and 5th by MGE.

ModelMGEBand MAEMAEBand ÷ MAERank: MAE / MGE / band
withoutBG Pro Model API0.02740.05360.02402.24×1 / 1 / 1
withoutBG Pro Model Self-Host0.03690.06080.02762.20×2 / 2 / 2
withoutBG Open Weights0.04520.06720.02922.30×3 / 3 / 3
BiRefNet General0.04950.09250.04262.17×4 / 4 / 4
BiRefNet General Lite0.05380.10900.05222.09×6 / 5 / 6
BiRefNet (legacy)0.05440.09760.04392.22×5 / 6 / 5
BiRefNet Massive0.05590.12040.07141.69×8 / 7 / 8
BiRefNet DIS0.05850.11670.05752.03×7 / 8 / 7
IS-Net0.06120.16140.09401.72×10 / 9 / 10
BiRefNet Portrait0.06470.14550.07142.04×9 / 10 / 9
BiRefNet HRSOD0.07140.16380.10691.53×11 / 11 / 11
U²-Net0.07980.23800.12831.86×12 / 12 / 12
U²-NetP0.08030.25270.14051.80×14 / 13 / 14
Silueta0.08450.24460.13441.82×13 / 14 / 13

BiRefNet DIS is a checkpoint trained for dichotomous (binary) segmentation of fine structures. Our model registry notes to avoid it for soft transparency such as hair and smoke, because its masks are close to binary. On Bench v0 it ranks 7 of 14 by MAE, 8 by MGE and 7 by band MAE.

Router disclosure#

withoutBG Open Weights and Pro Model API are routed pipelines. A classifier sends fine strands, soft detail and transparency to the withoutBG matting model, and hard opaque objects, flat scenes and vehicles to BiRefNet (ZhengPeng7/BiRefNet, MIT), which is part of the pipeline. Pro Model API also runs an edge refiner on the result of either branch, so its BiRefNet-routed images are refined BiRefNet output. Car endpoints add further steps (such as cast shadows) that are not part of this benchmark.

PipelineRouteImagesMAEMGE
withoutBG Pro Model APImatting410.01280.0199
withoutBG Pro Model APIbirefnet170.01230.0260
withoutBG Open Weightsmatting410.01850.0377
withoutBG Open Weightsbirefnet170.01510.0431

Some images have no recorded route and are in neither row (withoutBG Pro Model API: 2 of 60; withoutBG Open Weights: 2 of 60), so the rows do not average to the overall MAE in the table above.

Frequently asked questions#

What is the difference between alpha matting and image segmentation?#

Segmentation labels every pixel as foreground or background, a binary mask. Alpha matting predicts an opacity between 0 and 1 for every pixel, so a strand of hair, a glass edge or a motion-blurred border can be partly foreground. The composite is the foreground over the new background, weighted by that alpha.

Why do binary masks look wrong on hair, fur and glass?#

At those edges a single pixel mixes foreground and background colour. A binary mask has to call it one or the other, which leaves a hard, jagged edge or a halo of the old background once the cutout is placed on a new one. An alpha matte keeps the partial opacity instead.

What do MAE, MGE and band MAE measure?#

MAE is the mean absolute difference between the predicted alpha and the ground-truth alpha over all pixels. MGE compares the Sobel gradient magnitude of the two mattes, so it reacts to edges that are too hard, too soft or misplaced. Band MAE is MAE restricted to a 10 px band (at 1024 px) around the matte edge.

Which model has the lowest edge error on Bench v0?#

On Bench v0 (60 photos with ground-truth alpha mattes), withoutBG Pro Model API has the lowest MGE (0.0274) and withoutBG Pro Model API has the lowest band MAE (0.0536). Lower is better. withoutBG Open Weights, Pro Model API and Pro Model Self-Host are routed pipelines that use BiRefNet for some images; see the router disclosure on this page.