Package com.amazonaws.athena.connectors.datalakegen2

Class DataLakeGen2MetadataHandler

java.lang.Object

com.amazonaws.athena.connector.lambda.handlers.MetadataHandler

com.amazonaws.athena.connectors.jdbc.manager.JdbcMetadataHandler

com.amazonaws.athena.connectors.datalakegen2.DataLakeGen2MetadataHandler

All Implemented Interfaces:

FederationRequestHandler, com.amazonaws.services.lambda.runtime.RequestStreamHandler

public class DataLakeGen2MetadataHandler
extends JdbcMetadataHandler

Field Summary

Fields inherited from class com.amazonaws.athena.connectors.jdbc.manager.JdbcMetadataHandler

caseResolver, jdbcQueryPassthrough, TABLES_AND_VIEWS

Fields inherited from class com.amazonaws.athena.connector.lambda.handlers.MetadataHandler

configOptions, DISABLE_SPILL_ENCRYPTION, KMS_KEY_ID_ENV, SPILL_BUCKET_ENV, SPILL_PREFIX_ENV

Constructor Summary

Constructors

Modifier	Constructor	Description
	DataLakeGen2MetadataHandler(DatabaseConnectionConfig databaseConnectionConfig, JdbcConnectionFactory jdbcConnectionFactory, Map<String,String> configOptions)
	DataLakeGen2MetadataHandler(DatabaseConnectionConfig databaseConnectionConfig, Map<String,String> configOptions)	Used by Mux.
protected	DataLakeGen2MetadataHandler(DatabaseConnectionConfig databaseConnectionConfig, software.amazon.awssdk.services.secretsmanager.SecretsManagerClient secretsManager, software.amazon.awssdk.services.athena.AthenaClient athena, JdbcConnectionFactory jdbcConnectionFactory, Map<String,String> configOptions, JDBCCaseResolver caseResolver)
	DataLakeGen2MetadataHandler(Map<String,String> configOptions)	Instantiates handler to be used by Lambda function directly.

Method Summary

Modifier and Type	Method	Description
protected Optional<org.apache.arrow.vector.types.pojo.ArrowType>	convertDatasourceTypeToArrow(int columnIndex, int precision, Map<String,String> configOptions, ResultSetMetaData metadata)	A method that takes in a JDBC type; and converts it to Arrow Type This can be overriden by other Metadata Handlers extending JDBC
GetDataSourceCapabilitiesResponse	doGetDataSourceCapabilities(BlockAllocator allocator, GetDataSourceCapabilitiesRequest request)	Used to describe the types of capabilities supported by a data source.
GetSplitsResponse	doGetSplits(BlockAllocator blockAllocator, GetSplitsRequest getSplitsRequest)	Used to split-up the reads required to scan the requested batch of partition(s).
protected CredentialsProvider	getCredentialProvider()
void	getPartitions(BlockWriter blockWriter, GetTableLayoutRequest getTableLayoutRequest, QueryStatusChecker queryStatusChecker)	The partitions are being implemented based on the type of data externally in case of Gen 2.
org.apache.arrow.vector.types.pojo.Schema	getPartitionSchema(String catalogName)	Delegates creation of partition schema to database type implementation.
protected org.apache.arrow.vector.types.pojo.Schema	getSchema(Connection jdbcConnection, TableName tableName, org.apache.arrow.vector.types.pojo.Schema partitionSchema)	Appropriate datatype to arrow type conversions will be done by fetching data types of columns

Methods inherited from class com.amazonaws.athena.connectors.jdbc.manager.JdbcMetadataHandler

doGetQueryPassthroughSchema, doGetTable, doListSchemaNames, doListTables, escapeNamePattern, getArrayArrowTypeFromTypeName, getColumns, getDatabaseConnectionConfig, getJdbcConnectionFactory, getSplitClauses, listDatabaseNames, listPaginatedTables, listTables, setupQueryPassthroughSplit, wrapNameWithEscapedCharacter

Methods inherited from class com.amazonaws.athena.connector.lambda.handlers.MetadataHandler

doGetTableLayout, doHandleRequest, doPing, enhancePartitionSchema, getCachableSecretsManager, getRequestOverrideConfig, getSecret, handleRequest, makeEncryptionKey, makeSpillLocation, onPing, resolveSecrets, resolveWithDefaultCredentials

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface com.amazonaws.athena.connector.lambda.handlers.FederationRequestHandler

getAthenaClient, getRequestOverrideConfig, getS3Client, getSessionCredentials

Constructor Detail

DataLakeGen2MetadataHandler

public DataLakeGen2MetadataHandler(Map<String,String> configOptions)

Instantiates handler to be used by Lambda function directly. Recommend using DataLakeGen2MuxCompositeHandler instead.

DataLakeGen2MetadataHandler

public DataLakeGen2MetadataHandler(DatabaseConnectionConfig databaseConnectionConfig,
                                   Map<String,String> configOptions)

Used by Mux.

DataLakeGen2MetadataHandler

public DataLakeGen2MetadataHandler(DatabaseConnectionConfig databaseConnectionConfig,
                                   JdbcConnectionFactory jdbcConnectionFactory,
                                   Map<String,String> configOptions)

DataLakeGen2MetadataHandler

protected DataLakeGen2MetadataHandler(DatabaseConnectionConfig databaseConnectionConfig,
                                      software.amazon.awssdk.services.secretsmanager.SecretsManagerClient secretsManager,
                                      software.amazon.awssdk.services.athena.AthenaClient athena,
                                      JdbcConnectionFactory jdbcConnectionFactory,
                                      Map<String,String> configOptions,
                                      JDBCCaseResolver caseResolver)

Method Detail

doGetDataSourceCapabilities

public GetDataSourceCapabilitiesResponse doGetDataSourceCapabilities(BlockAllocator allocator,
                                                                     GetDataSourceCapabilitiesRequest request)

Description copied from class: MetadataHandler

Used to describe the types of capabilities supported by a data source. An engine can use this to determine what portions of the query to push down. A connector that returns any optimization will guarantee that the associated predicate will be pushed down.

Overrides:

doGetDataSourceCapabilities in class MetadataHandler

Parameters:

allocator - Tool for creating and managing Apache Arrow Blocks.

request - Provides details about the catalog being used.

Returns:

A GetDataSourceCapabilitiesResponse object which returns a map of supported optimizations that the connector is advertising to the consumer. The connector assumes all responsibility for whatever is passed here.

getPartitionSchema

public org.apache.arrow.vector.types.pojo.Schema getPartitionSchema(String catalogName)

Description copied from class: JdbcMetadataHandler

Delegates creation of partition schema to database type implementation.

Specified by:

getPartitionSchema in class JdbcMetadataHandler

Parameters:

catalogName - Athena provided catalog name.

Returns:

schema. See Schema

getPartitions

public void getPartitions(BlockWriter blockWriter,
                          GetTableLayoutRequest getTableLayoutRequest,
                          QueryStatusChecker queryStatusChecker)

The partitions are being implemented based on the type of data externally in case of Gen 2. Considering the ADLS Gen2 data has already been partitioned and distributed within Gen 2 storage system, connector will fetch data as single split.

Specified by:

getPartitions in class JdbcMetadataHandler

Parameters:

blockWriter -

getTableLayoutRequest -

queryStatusChecker -

Throws:

Exception

doGetSplits

public GetSplitsResponse doGetSplits(BlockAllocator blockAllocator,
                                     GetSplitsRequest getSplitsRequest)

Description copied from class: MetadataHandler

Used to split-up the reads required to scan the requested batch of partition(s).

Specified by:

doGetSplits in class JdbcMetadataHandler

Parameters:

blockAllocator - Tool for creating and managing Apache Arrow Blocks.

getSplitsRequest - Provides details of the catalog, database, table, andpartition(s) being queried as well as any filter predicate.

Returns:

A GetSplitsResponse which primarily contains: 1. A Set which represent read operations Amazon Athena must perform by calling your read function. 2. (Optional) A continuation token which allows you to paginate the generation of splits for large queries.

convertDatasourceTypeToArrow

protected Optional<org.apache.arrow.vector.types.pojo.ArrowType> convertDatasourceTypeToArrow(int columnIndex,
                                                                                              int precision,
                                                                                              Map<String,String> configOptions,
                                                                                              ResultSetMetaData metadata)
                                                                                       throws SQLException

Description copied from class: JdbcMetadataHandler

A method that takes in a JDBC type; and converts it to Arrow Type This can be overriden by other Metadata Handlers extending JDBC

Overrides:

convertDatasourceTypeToArrow in class JdbcMetadataHandler

Returns:

Arrow Type

Throws:

SQLException

getSchema

protected org.apache.arrow.vector.types.pojo.Schema getSchema(Connection jdbcConnection,
                                                              TableName tableName,
                                                              org.apache.arrow.vector.types.pojo.Schema partitionSchema)
                                                       throws Exception

Appropriate datatype to arrow type conversions will be done by fetching data types of columns

Overrides:

getSchema in class JdbcMetadataHandler

Parameters:

jdbcConnection -

tableName -

partitionSchema -

Returns:

Throws:

Exception

getCredentialProvider

protected CredentialsProvider getCredentialProvider()

Overrides:

getCredentialProvider in class JdbcMetadataHandler