s3torchconnector.s3reader.constructor

Classes

S3ReaderConstructor

Constructor for creating partial(S3Reader) instances.

Module Contents

class s3torchconnector.s3reader.constructor.S3ReaderConstructor[source]

Constructor for creating partial(S3Reader) instances.

Creates partial S3Reader instances that will be completed by S3Client with the remaining required parameters (e.g. bucket, key, get_object_info, get_stream).

The constructor provides factory methods for different reader types:

  • sequential(): Creates a constructor for sequential readers that buffer the entire object. Best for full reads and repeated access.

  • range_based(): Creates a constructor for range-based readers that fetch specific byte ranges. Suitable for sparse partial reads for large objects.

static sequential() → s3torchconnector.s3reader.protocol.S3ReaderConstructorProtocol[source]

Creates a constructor for sequential readers

Returns:

Partial constructor for SequentialS3Reader

Return type:

S3ReaderConstructorProtocol

Example:

reader_constructor = S3ReaderConstructor.sequential()
static range_based(buffer_size: int | None = None) → s3torchconnector.s3reader.protocol.S3ReaderConstructorProtocol[source]

Creates a constructor for range-based readers

Parameters:

buffer_size – Internal buffer size in bytes. If None, uses default 8MB. Set to 0 to disable buffering.

Returns:

Partial constructor for RangedS3Reader

Return type:

S3ReaderConstructorProtocol

Range-based reader performs byte-range requests to read specific portions of S3 objects without downloading the entire file.

Buffer size affects read performance:

  • Small reads (< buffer_size): Loads buffer_size bytes to buffer to reduce S3 API calls for small, sequential reads

  • Large reads (≥ buffer_size): bypass the buffer for direct transfer from S3

  • Forward overlap reads: Reuses buffered data when reading ranges that extend beyond current buffer, and processes remaining

data according to size with logic above.

Configuration Guide:

  • Use larger buffer sizes for workloads with many small, sequential reads of nearby bytes

  • Use smaller buffer sizes or disable buffering for sparse partial reads

  • Buffer can be disabled by setting buffer_size to 0

  • If buffer_size is None, uses default 8MB buffer

Examples:

# Range-based reader with default 8MB buffer
reader_constructor = S3ReaderConstructor.range_based()

# Range-based reader with custom buffer size
reader_constructor = S3ReaderConstructor.range_based(buffer_size=16*1024*1024)

# Range-based reader with buffering disabled
reader_constructor = S3ReaderConstructor.range_based(buffer_size=0)
static default() → s3torchconnector.s3reader.protocol.S3ReaderConstructorProtocol[source]

Creates default reader constructor (sequential)

Returns:

Partial constructor for SequentialS3Reader

Return type:

S3ReaderConstructorProtocol

static get_reader_type_string(constructor: s3torchconnector.s3reader.protocol.S3ReaderConstructorProtocol | None) → str[source]

Returns the reader type string for the given constructor.