s3torchconnector.s3reader.ranged

Attributes

DEFAULT_BUFFER_SIZE

Classes

RangedS3Reader

Range-based S3 reader implementation with adaptive buffering.

Module Contents

s3torchconnector.s3reader.ranged.DEFAULT_BUFFER_SIZE = 8388608[source]
class s3torchconnector.s3reader.ranged.RangedS3Reader(bucket: str, key: str, get_object_info: Callable[[], s3torchconnectorclient._mountpoint_s3_client.ObjectInfo | s3torchconnectorclient._mountpoint_s3_client.HeadObjectResult], get_stream: Callable[[int | None, int | None], s3torchconnectorclient._mountpoint_s3_client.GetObjectStream], buffer_size: int | None = None)[source]

Bases: s3torchconnector.s3reader.s3reader.S3Reader

Range-based S3 reader implementation with adaptive buffering.

Performs byte-range requests to read specific portions of S3 objects without downloading the entire file. Includes optional adaptive buffer to reduce S3 API calls for small, sequential reads while bypassing buffering for large reads. Optimal for sparse partial reads of large objects.

Buffering behavior:

  • Small reads (< buffer_size): Loads buffer_size bytes to buffer, copies to user

  • Large reads (>= buffer_size): Direct S3 access, bypass buffer

  • Forward overlapping reads: Reuses existing buffer data if possible when read range extends beyond current buffer

  • Buffer can be disabled by setting buffer_size to 0

  • If buffer_size is None, uses default 8MB buffer

Parameters:

  • bucket – S3 bucket name

  • key – S3 object key

  • get_object_info – Callable that returns object metadata

  • get_stream – Callable that returns stream for byte range requests

  • buffer_size – Internal buffer size in bytes, defaults to 8MB

property bucket: str[source]
property key: str[source]
readinto(buf) → int[source]

Read up to len(buf) bytes into a pre-allocated, writable bytes-like object buf. Return the number of bytes read. If no bytes are available, zero is returned.

Parameters:

buf – writable bytes-like object

Returns:

numer of bytes read or zero, if no bytes available

Return type:

int

read(size: int | None = None) → bytes[source]

Read up to size bytes from the current position.

If size is zero or positive, read that many bytes from S3, or until the end of the object. If size is None or negative, read until the end of the object.

Parameters:

size (int | None) – how many bytes to read.

Returns:

Bytes read from specified range.

Return type:

bytes

Raises:

S3Exception – An error occurred accessing S3.

seek(offset: int, whence: int = SEEK_SET, /) → int[source]

Change the stream position to the given byte offset, interpreted relative to whence.

When seeking beyond the end of the file, always stay at EOF. Seeking before the start of the file results in a ValueError.

Parameters:

  • offset (int) – How many bytes to seek relative to whence.

  • whence (int) – One of SEEK_SET, SEEK_CUR, and SEEK_END. Default: SEEK_SET

Returns:

Current position of the stream

Return type:

int

Raises:

S3Exception – An error occurred accessing S3.

tell() → int[source]
Returns:

Current stream position.

Return type:

int