Open background removal benchmark#
Bench v0 is a fixed set of 60 real photos (people, hair, plants, objects, animals, motion), each with a ground-truth alpha matte. 14 background removal models run on the same inputs and are scored against that ground truth with MAE and edge quality (MGE). The compare pages show every scored image side by side with its numbers.
Methodology
- Same source images for every model; outputs are stored and rescored, never hand-picked.
- Ground truth and prediction are both resized (Lanczos) to a 1024 px shorter side, aspect kept, never upscaled, so every model is scored at the same scale.
- MAE is the mean absolute alpha error. MGE is the mean difference of Sobel gradient magnitudes (edge sharpness). Both are lower-is-better.
- Reported numbers are per-image averages: every image weighs the same. Each image also stores its error sums and pixel count, so pooled totals or any subset can be recomputed from the data below.
- Comparisons with a model that has no ground-truth run (for example remove.bg) are qualitative: side by side only, no scores.
- Images shown on the site are resized to a 600 px longer side; scoring uses the full outputs.
Disclosure: withoutBG includes BiRefNet
withoutBG Open Weights and Pro Model API are routed pipelines. A classifier sends fine strands, soft detail and transparency to withoutBG's own matting model, and hard opaque objects, flat scenes and vehicles to BiRefNet (MIT license), which is part of the pipeline. On Bench v0, 41 images went to the withoutBG matting model and 17 to BiRefNet. Pro Model API also runs an edge refiner on the result of either branch, so its BiRefNet-routed images are refined BiRefNet output; its car endpoints add further steps (such as cast shadows) that are not part of this benchmark. Comparisons with BiRefNet therefore partly compare BiRefNet with itself. Every compare page with a withoutBG side splits its scores by route, and the data below has the route for each image.
Data
- Per-image scores for every model: JSON · CSV. Each row has MAE, MGE, their error sums and the pixel count.
- License: CC BY 4.0. Credit: Imran Kocabiyik, withoutBG.
- Markdown version of this page: /benchmarks/open-background-removal.md