s3torchconnector.s3checkpoint

Classes

S3Checkpoint

A checkpoint manager for S3.

Module Contents

class s3torchconnector.s3checkpoint.S3Checkpoint(region: str, endpoint: str | None = None, s3client_config: s3torchconnector._s3client.S3ClientConfig | None = None)[source]

A checkpoint manager for S3.

To read a checkpoint from S3, users need to create an S3Reader by providing s3_uri of the checkpoint stored in S3. Similarly, to save a checkpoint to S3, users need to create an S3Writer by providing s3_uri. S3Reader and S3Writer implements io.BufferedIOBase therefore, they can be passed to torch.load, and torch.save.

region[source]

endpoint = None[source]

reader(s3_uri: str) → s3torchconnector.S3Reader[source]

Creates an S3Reader from a given s3_uri.

Parameters:: s3_uri (str) – A valid s3_uri. (i.e. s3://<BUCKET>/<KEY>)
Returns:: a read-only binary stream of the S3 object’s contents, specified by the s3_uri.
Return type:: S3Reader
Raises:: S3Exception – An error occurred accessing S3.

writer(s3_uri: str) → s3torchconnector.S3Writer[source]

Creates an S3Writer from a given s3_uri.

Parameters:: s3_uri (str) – A valid s3_uri. (i.e. s3://<BUCKET>/<KEY>)
Returns:: a write-only binary stream. The content is saved to S3 using the specified s3_uri.
Return type:: S3Writer
Raises:: S3Exception – An error occurred accessing S3.