Package com.amazonaws.athena.connector.lambda.data

Class S3BlockSpiller

java.lang.Object

com.amazonaws.athena.connector.lambda.data.S3BlockSpiller

All Implemented Interfaces:

BlockSpiller, BlockWriter, AutoCloseable

public class S3BlockSpiller
extends Object
implements AutoCloseable, BlockSpiller

Implementation of BlockSpiller which spills Blocks from large results to S3 with optional AES-GCM encryption.

Nested Class Summary

Nested classes/interfaces inherited from interface com.amazonaws.athena.connector.lambda.data.BlockWriter

BlockWriter.RowWriter

Constructor Summary

Constructors

Constructor	Description
S3BlockSpiller(software.amazon.awssdk.services.s3.S3Client amazonS3, SpillConfig spillConfig, BlockAllocator allocator, org.apache.arrow.vector.types.pojo.Schema schema, ConstraintEvaluator constraintEvaluator, int maxRowsPerCall, Map<String,String> configOptions)	Constructs a new S3BlockSpiller.
S3BlockSpiller(software.amazon.awssdk.services.s3.S3Client amazonS3, SpillConfig spillConfig, BlockAllocator allocator, org.apache.arrow.vector.types.pojo.Schema schema, ConstraintEvaluator constraintEvaluator, Map<String,String> configOptions)	Constructor which uses the default maxRowsPerCall.

Method Summary

Modifier and Type	Method	Description
void	close()	Frees any resources held by this BlockSpiller.
Block	getBlock()	If spilled() returns false this can be used to access the block.
ConstraintEvaluator	getConstraintEvaluator()	Provides access to the constraint evaluator used to constrain blocks written via this BlockSpiller.
List<SpillLocation>	getSpillLocations()	If spilled() returns true this can be used to access the spill locations of all blocks.
protected Block	read(S3SpillLocation spillLocation, EncryptionKey key, org.apache.arrow.vector.types.pojo.Schema schema)	Reads a spilled block.
boolean	spilled()	Used to tell if any blocks were spilled or if the response can be inline.
protected SpillLocation	write(Block block)	Writes (aka spills) a Block.
void	writeRows(BlockWriter.RowWriter rowWriter)	Used to write rows via the BlockWriter.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

S3BlockSpiller

public S3BlockSpiller(software.amazon.awssdk.services.s3.S3Client amazonS3,
                      SpillConfig spillConfig,
                      BlockAllocator allocator,
                      org.apache.arrow.vector.types.pojo.Schema schema,
                      ConstraintEvaluator constraintEvaluator,
                      Map<String,String> configOptions)

Constructor which uses the default maxRowsPerCall.

Parameters:

amazonS3 - AmazonS3 client to use for writing to S3.

spillConfig - The spill config for this instance. Includes things like encryption key, s3 path, etc...

allocator - The BlockAllocator to use when creating blocks.

schema - The schema for blocks that should be written.

constraintEvaluator - The ConstraintEvaluator that should be used to constrain writes.

S3BlockSpiller

public S3BlockSpiller(software.amazon.awssdk.services.s3.S3Client amazonS3,
                      SpillConfig spillConfig,
                      BlockAllocator allocator,
                      org.apache.arrow.vector.types.pojo.Schema schema,
                      ConstraintEvaluator constraintEvaluator,
                      int maxRowsPerCall,
                      Map<String,String> configOptions)

Constructs a new S3BlockSpiller.

Parameters:

amazonS3 - AmazonS3 client to use for writing to S3.

spillConfig - The spill config for this instance. Includes things like encryption key, s3 path, etc...

allocator - The BlockAllocator to use when creating blocks.

schema - The schema for blocks that should be written.

constraintEvaluator - The ConstraintEvaluator that should be used to constrain writes.

maxRowsPerCall - The max number of rows to allow callers to write in one call.

Method Detail

getConstraintEvaluator

public ConstraintEvaluator getConstraintEvaluator()

Provides access to the constraint evaluator used to constrain blocks written via this BlockSpiller.

Specified by:

getConstraintEvaluator in interface BlockSpiller

Specified by:

getConstraintEvaluator in interface BlockWriter

Returns:

writeRows

public void writeRows(BlockWriter.RowWriter rowWriter)

Used to write rows via the BlockWriter.

Specified by:

writeRows in interface BlockWriter

Parameters:

rowWriter - The RowWriter that the BlockWriter should use to write rows into the Block(s) it is managing.

See Also:

BlockSpiller

spilled

public boolean spilled()

Used to tell if any blocks were spilled or if the response can be inline.

Specified by:

spilled in interface BlockSpiller

Returns:

True is spill occurred, false otherwise.

getBlock

public Block getBlock()

If spilled() returns false this can be used to access the block.

Specified by:

getBlock in interface BlockSpiller

Returns:

Block to be inlined in the response.

getSpillLocations

public List<SpillLocation> getSpillLocations()

If spilled() returns true this can be used to access the spill locations of all blocks.

Specified by:

getSpillLocations in interface BlockSpiller

Returns:

List of spill locations.

close

public void close()

Frees any resources held by this BlockSpiller.

Specified by:

close in interface AutoCloseable

Specified by:

close in interface BlockSpiller

See Also:

BlockSpiller

write

protected SpillLocation write(Block block)

Writes (aka spills) a Block.

read

protected Block read(S3SpillLocation spillLocation,
                     EncryptionKey key,
                     org.apache.arrow.vector.types.pojo.Schema schema)

Reads a spilled block.

Parameters:

spillLocation - The location to read the spilled Block from.

key - The encryption key to use when reading the spilled Block.

schema - The Schema to use when deserializing the spilled Block.

Returns:

The Block stored at the spill location.