Background Removal

Unstable API

0.8.0

@project-lakechain/rembg-image-processor

---

The RemBg image processor packages the RemBg algorithm and suite of machine-learning models on AWS for easy background removal on images. This middleware is a good fit when processing images that have a foreground entity you would like to extract from the background.

The algorithm is implemented as different steps which produce a list of binary masks representing the foreground objects in the image, which are then post-processed and combined to create a final cutout image.

Original Image	Processed Image

Credits Silje Midtgård on Unsplash

---

🖼️ Removing Backgrounds

To use this middleware, you import it in your CDK stack and connect it to a data source providing images. You also need to specify a VPC in which the middleware will be deployed as it deploys an EFS file-system to cache the models used by RemBg.

import { RembgImageProcessor } from '@project-lakechain/rembg-image-processor';
import { CacheStorage } from '@project-lakechain/core';

class Stack extends cdk.Stack {
  constructor(scope: cdk.Construct, id: string) {
    // The cache storage.
    const cache = new CacheStorage(this, 'Cache');

    // Create the RemBg image processor.
    const processor = new RembgImageProcessor.Builder()
      .withScope(this)
      .withIdentifier('Processor')
      .withCacheStorage(cache)
      .withVpc(vpc)
      .withSource(source) // 👈 Specify a data source
      .build();
  }
}

---

Alpha Matting

The Rembg algorithm can perform alpha matting on cutout results to improve the quality of the results. This is done by creating a soft transition between the foreground and the background, which can be useful when overlaying the cutout on a different background.

⛔ Important - Alpha matting is a memory intensive process. If you use the CPU compute type which is the default compute type, an exception will be raised when you enable this option. You will be prompted to manually adjust the maximum memory size to 10GB on a CPU compute type using the .withMaxMemorySize method. This is because Lambda functions having 10GB of memory have a cost implication, and this is a safety measure to require users to manually adjust the memory size.

CPU Example

const processor = new RembgImageProcessor.Builder()
  .withScope(this)
  .withIdentifier('Processor')
  .withCacheStorage(cache)
  .withVpc(vpc)
  .withSource(source)
  .withMaxMemorySize(10240) // 👈 Set the memory size to 10GB
  .withAlphaMatting(true) // 👈 Enable alpha matting
  .build();

GPU Example

💁 GPU instances don’t have the same memory limitations as Lambda functions. It is safe to keep the default maximum memory size.

import { ComputeType } from '@project-lakechain/core';

const processor = new RembgImageProcessor.Builder()
  .withScope(this)
  .withIdentifier('Processor')
  .withCacheStorage(cache)
  .withVpc(vpc)
  .withSource(source)
  .withComputeType(ComputeType.GPU)
  .withAlphaMatting(true) // 👈 Enable alpha matting
  .build();

Additional Options

Alpha matting supports 3 additional options you can optionally customize when alpha matting is enabled.

• alphaMattingForegroundThreshold - Foreground threshold for alpha matting. Defaults to 240.
• alphaMattingBackgroundThreshold - Background threshold for alpha matting. Defaults to 10.
• alphaMattingErosionSize - Erosion size for alpha matting. Defaults to 10.

👇 The example below uses the appropriate methods to customize those values.

const processor = new RembgImageProcessor.Builder()
  .withScope(this)
  .withIdentifier('Processor')
  .withCacheStorage(cache)
  .withVpc(vpc)
  .withSource(source)
  .withAlphaMatting(true)
  .withAlphaMattingForegroundThreshold(240)
  .withAlphaMattingBackgroundThreshold(10)
  .withAlphaMattingErosionSize(10)
  .build();

---

Post-processing

To further improve the results of the cutout, you can enable mask post-processing on the Rembg algorithm for a smooth boundary by applying morphological operations.

ℹ️ See this paper on the method used for post-processing.

const processor = new RembgImageProcessor.Builder()
  .withScope(this)
  .withIdentifier('Processor')
  .withCacheStorage(cache)
  .withVpc(vpc)
  .withSource(source)
  .withMaskPostProcessing(true) // 👈 Enable post-processing
  .build();

---

🏗️ Architecture

This middleware supports both CPU and GPU compute types. We implemented 2 different architectures, one that’s GPU based and using ECS, the other which is CPU based and serverless, based on AWS Lambda. You can use the .withComputeType API to select the compute type you want to use.

💁 By default, this implementation will run on the CPU compute type.

GPU Architecture

The GPU architecture leverages AWS ECS to run the summarization process on g4dn.xlarge instances. The GPU instance is part of a ECS cluster, and the cluster is part of a VPC, running within its private subnets.

CPU Architecture

The CPU architecture leverages AWS Lambda to run the background removal process on serverless compute. The Lambda function runs as part of a VPC and is integrated with AWS EFS to cache the models used by RemBg.

---

🏷️ Properties

Supported Inputs

Mime Type	Description
image/jpeg	JPEG images.
image/png	PNG images.

Supported Outputs

Mime Type	Description
image/png	PNG images.

Supported Compute Types

Type	Description
CPU	This middleware supports CPU compute.
GPU	This middleware supports GPU compute.

---

📖 Examples

• Image Background Removal - Builds a pipeline demonstrating automatic image background removal using Rembg.