Trafilatura

Unstable API 0.10.0 @project-lakechain/trafilatura

---

The Trafilatura middleware is based on the Trafilatura library which provides one of the most accurate rule-based engine for HTML article text extraction. It provides capability to analyze and extract the substance of HTML documents on the Web and use that text as an input to other middlewares in your pipeline.

---

📰 Extracting Text

To use this middleware, you import it in your CDK stack and instantiate it as part of a pipeline.

import { TrafilaturaParser } from '@project-lakechain/trafilatura';
import { CacheStorage } from '@project-lakechain/core';

class Stack extends cdk.Stack {
  constructor(scope: cdk.Construct, id: string) {
    // The cache storage.
    const cache = new CacheStorage(this, 'Cache');

    // The trafilatura parser.
    const trafilatura = new TrafilaturaParser.Builder()
      .withScope(this)
      .withIdentifier('Trafilatura')
      .withCacheStorage(cache)
      .withSource(source) // 👈 Specify a data source
      .build();
  }
}

---

🏗️ Architecture

This middleware is based on a Lambda compute running the Trafilatura library packaged as a Docker container.

---

🏷️ Properties

Supported Inputs

Mime Type	Description
text/html	HTML documents.

Supported Outputs

Mime Type	Description
text/plain	Plain text documents.

Supported Compute Types

Type	Description
CPU	This middleware only supports CPU compute.