s3torchconnector.dcp.s3_prefix_strategy

Classes

`S3PrefixStrategyBase`	Base class for S3 prefix generation strategies.
`DefaultPrefixStrategy`	Default strategy for generating S3 prefixes.
`NumericPrefixStrategy`	Base class for numeric prefix generation strategies.
`BinaryPrefixStrategy`	Binary (Base2) prefix generation strategy using only 0 and 1.
`HexPrefixStrategy`	Hexadecimal-based prefix generation strategy.
`RoundRobinPrefixStrategy`	Strategy that distributes ranks across user-provided prefixes in round-robin fashion.

Module Contents

class s3torchconnector.dcp.s3_prefix_strategy.S3PrefixStrategyBase[source]

Bases: abc.ABC

Base class for S3 prefix generation strategies.

abstract generate_prefix(rank: int) → str[source]: Generate storage prefix for the given rank.

class s3torchconnector.dcp.s3_prefix_strategy.DefaultPrefixStrategy[source]

Bases: S3PrefixStrategyBase

Default strategy for generating S3 prefixes.

generate_prefix(rank: int) → str[source]: Generate simple rank-based name without prefix.

class s3torchconnector.dcp.s3_prefix_strategy.NumericPrefixStrategy(base: int, epoch_num: int | None = None, min_prefix_length: int = 10, prefix_count: int | None = None)[source]

Bases: S3PrefixStrategyBase

Base class for numeric prefix generation strategies.

base[source]

epoch_num = None[source]

min_prefix_len = 10[source]

prefix_count = 1[source]

prefix_map[source]

generate_prefix(rank: int) → str[source]

Generate numeric-based prefix with optional epoch number.

Parameters:: rank – Process rank in the distributed environment.
Returns:: <pattern>/epoch_<num>/__<rank>_ or <pattern>/__<rank>_ if no epoch number is provided.
Return type:: Prefix string in format

class s3torchconnector.dcp.s3_prefix_strategy.BinaryPrefixStrategy(epoch_num: int | None = None, min_prefix_length: int = 10, prefix_count: int | None = None)[source]

Bases: NumericPrefixStrategy

Binary (Base2) prefix generation strategy using only 0 and 1.

class s3torchconnector.dcp.s3_prefix_strategy.HexPrefixStrategy(epoch_num: int | None = None, min_prefix_length: int = 10, prefix_count: int | None = None)[source]

Bases: NumericPrefixStrategy

Hexadecimal-based prefix generation strategy.

class s3torchconnector.dcp.s3_prefix_strategy.RoundRobinPrefixStrategy(user_prefixes: List[str], epoch_num: int | None = None)[source]

Bases: S3PrefixStrategyBase

Strategy that distributes ranks across user-provided prefixes in round-robin fashion.

user_prefixes[source]

epoch_num = None[source]

generate_prefix(rank: int) → str[source]

Generate prefix for given rank using round-robin distribution.

Parameters:: rank – Process rank in the distributed environment.
Returns:: <user_prefix>/epoch_<num>/__<rank>_ or <user_prefix>/__<rank>_ if no epoch number is provided.
Return type:: Prefix string in format