# Open background removal benchmark (Bench v0)

Source: https://withoutbg.com/benchmarks/open-background-removal

Per-image alpha matte scores (MAE, MGE, band MAE) against ground-truth mattes for 14 background removal models on 60 real photos. Every model runs on the same inputs; ground truth and predictions are scored at a 1024 px shorter side (never upscaled). Lower is better.

## Models

- withoutBG Pro Model API
- withoutBG Pro Model Self-Host
- withoutBG Open Weights
- BiRefNet (legacy, BiRefNet_dynamic)
- BiRefNet General
- BiRefNet General Lite
- IS-Net
- U²-Net
- U²-NetP
- Silueta
- BiRefNet Portrait
- BiRefNet DIS
- BiRefNet HRSOD
- BiRefNet Massive

Per-model averages and per-image scores are in the data files below.

## Methodology

- 60 real photos (people, hair, plants, objects, animals, motion), each with a ground-truth alpha matte.
- Every model runs on the same inputs.
- Ground truth and prediction are both resized (Lanczos) to a 1024 px shorter side, aspect kept. Images already at or below that size are not upscaled.
- MAE: mean absolute alpha error. MGE: mean Sobel gradient-magnitude error (edges). Band MAE: MAE inside a 10 px band (at 1024 px) around the matte edge.
- Per-image average: every image weighs the same. Pooled: total error over total pixels.
- Compare pages without ground truth (e.g. remove.bg) are qualitative: side by side only, no scores.

## Disclosure: withoutBG includes BiRefNet

withoutBG Open Weights and Pro Model API are routed pipelines. A classifier sends fine strands, soft detail and transparency to the withoutBG matting model, and hard opaque objects, flat scenes and vehicles to BiRefNet (ZhengPeng7/BiRefNet, MIT), which is part of the pipeline. Pro Model API also runs an edge refiner on the result of either branch, so its BiRefNet-routed images are refined BiRefNet output. Car endpoints add further steps (such as cast shadows) that are not part of this benchmark.

On Bench v0, 41 images went to the withoutBG matting model and 17 to BiRefNet. The data files split every model's scores by this route (`withoutbg_route` column, `summaryByWithoutbgRoute`).

## Data

- Scores (JSON): https://withoutbg.com/benchmarks/bench-v0/scores.json
- Scores (CSV): https://withoutbg.com/benchmarks/bench-v0/scores.csv
- License: CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Creator: Imran Kocabiyik, withoutBG.

Pairwise comparison pages: https://withoutbg.com/compare