Package com.amazonaws.athena.connectors.jdbc.manager

Class JdbcMetadataHandler

java.lang.Object

com.amazonaws.athena.connector.lambda.handlers.MetadataHandler

com.amazonaws.athena.connectors.jdbc.manager.JdbcMetadataHandler

All Implemented Interfaces:

com.amazonaws.services.lambda.runtime.RequestStreamHandler

Direct Known Subclasses:

DataLakeGen2MetadataHandler, Db2As400MetadataHandler, Db2MetadataHandler, HiveMetadataHandler, HiveMetadataHandler, ImpalaMetadataHandler, MultiplexingJdbcMetadataHandler, MySqlMetadataHandler, OracleMetadataHandler, PostGreSqlMetadataHandler, SaphanaMetadataHandler, SnowflakeMetadataHandler, SqlServerMetadataHandler, SynapseMetadataHandler, TeradataMetadataHandler, VerticaMetadataHandler

public abstract class JdbcMetadataHandler
extends MetadataHandler

Abstracts JDBC metadata handler and provides common reusable metadata handling.

Field Summary

Fields

Modifier and Type	Field	Description
protected JdbcQueryPassthrough	jdbcQueryPassthrough
static String	TABLES_AND_VIEWS

Fields inherited from class com.amazonaws.athena.connector.lambda.handlers.MetadataHandler

configOptions, DISABLE_SPILL_ENCRYPTION, KMS_KEY_ID_ENV, SPILL_BUCKET_ENV, SPILL_PREFIX_ENV

Constructor Summary

Constructors

Modifier	Constructor	Description
protected	JdbcMetadataHandler(DatabaseConnectionConfig databaseConnectionConfig, JdbcConnectionFactory jdbcConnectionFactory, Map<String,String> configOptions)
protected	JdbcMetadataHandler(DatabaseConnectionConfig databaseConnectionConfig, com.amazonaws.services.secretsmanager.AWSSecretsManager secretsManager, com.amazonaws.services.athena.AmazonAthena athena, JdbcConnectionFactory jdbcConnectionFactory, Map<String,String> configOptions)
protected	JdbcMetadataHandler(String sourceType, Map<String,String> configOptions)	Used only by Multiplexing handler.

Method Summary

Modifier and Type	Method	Description
protected TableName	caseInsensitiveTableSearch(Connection connection, String databaseName, String tableName)	While being a no-op by default, this function will be overriden by subclasses that support this search.
protected org.apache.arrow.vector.types.pojo.ArrowType	convertDatasourceTypeToArrow(int columnIndex, int precision, Map<String,String> configOptions, ResultSetMetaData metadata)	A method that takes in a JDBC type; and converts it to Arrow Type This can be overriden by other Metadata Handlers extending JDBC
GetTableResponse	doGetQueryPassthroughSchema(BlockAllocator blockAllocator, GetTableRequest getTableRequest)	Used to get definition (field names, types, descriptions, etc...) of a Query PassThrough.
abstract GetSplitsResponse	doGetSplits(BlockAllocator blockAllocator, GetSplitsRequest getSplitsRequest)	Used to split-up the reads required to scan the requested batch of partition(s).
GetTableResponse	doGetTable(BlockAllocator blockAllocator, GetTableRequest getTableRequest)	Used to get definition (field names, types, descriptions, etc...) of a Table.
ListSchemasResponse	doListSchemaNames(BlockAllocator blockAllocator, ListSchemasRequest listSchemasRequest)	Used to get the list of schemas (aka databases) that this source contains.
ListTablesResponse	doListTables(BlockAllocator blockAllocator, ListTablesRequest listTablesRequest)	Used to get the list of tables that this source contains.
protected String	escapeNamePattern(String name, String escape)
protected org.apache.arrow.vector.types.pojo.ArrowType	getArrayArrowTypeFromTypeName(String typeName, int precision, int scale)	Converts an ARRAY column's TYPE_NAME (provided by the jdbc metadata) to an ArrowType.
protected JdbcCredentialProvider	getCredentialProvider()
protected JdbcConnectionFactory	getJdbcConnectionFactory()
abstract void	getPartitions(BlockWriter blockWriter, GetTableLayoutRequest request, QueryStatusChecker queryStatusChecker)	Used to get the partitions that must be read from the request table in order to satisfy the requested predicate.
abstract org.apache.arrow.vector.types.pojo.Schema	getPartitionSchema(String catalogName)	Delegates creation of partition schema to database type implementation.
protected List<String>	getSplitClauses(TableName tableName)
protected ListTablesResponse	listPaginatedTables(Connection connection, ListTablesRequest listTablesRequest)
protected List<TableName>	listTables(Connection jdbcConnection, String databaseName)
protected GetSplitsResponse	setupQueryPassthroughSplit(GetSplitsRequest request)	Helper function that provides a single partition for Query Pass-Through

Methods inherited from class com.amazonaws.athena.connector.lambda.handlers.MetadataHandler

doGetDataSourceCapabilities, doGetTableLayout, doHandleRequest, doPing, enhancePartitionSchema, getSecret, handleRequest, makeEncryptionKey, makeSpillLocation, onPing, resolveSecrets

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

TABLES_AND_VIEWS

public static final String TABLES_AND_VIEWS

See Also:

Constant Field Values

jdbcQueryPassthrough

protected JdbcQueryPassthrough jdbcQueryPassthrough

Constructor Detail

JdbcMetadataHandler

protected JdbcMetadataHandler(String sourceType,
                              Map<String,String> configOptions)

Used only by Multiplexing handler. All calls will be delegated to respective database handler.

JdbcMetadataHandler

protected JdbcMetadataHandler(DatabaseConnectionConfig databaseConnectionConfig,
                              JdbcConnectionFactory jdbcConnectionFactory,
                              Map<String,String> configOptions)

JdbcMetadataHandler

protected JdbcMetadataHandler(DatabaseConnectionConfig databaseConnectionConfig,
                              com.amazonaws.services.secretsmanager.AWSSecretsManager secretsManager,
                              com.amazonaws.services.athena.AmazonAthena athena,
                              JdbcConnectionFactory jdbcConnectionFactory,
                              Map<String,String> configOptions)

Method Detail

getJdbcConnectionFactory

protected JdbcConnectionFactory getJdbcConnectionFactory()

getCredentialProvider

protected JdbcCredentialProvider getCredentialProvider()

doListSchemaNames

public ListSchemasResponse doListSchemaNames(BlockAllocator blockAllocator,
                                             ListSchemasRequest listSchemasRequest)
                                      throws Exception

Description copied from class: MetadataHandler

Used to get the list of schemas (aka databases) that this source contains.

Specified by:

doListSchemaNames in class MetadataHandler

Parameters:

blockAllocator - Tool for creating and managing Apache Arrow Blocks.

listSchemasRequest - Provides details on who made the request and which Athena catalog they are querying.

Returns:

A ListSchemasResponse which primarily contains a Set of schema names and a catalog name corresponding the Athena catalog that was queried.

Throws:

Exception

doListTables

public ListTablesResponse doListTables(BlockAllocator blockAllocator,
                                       ListTablesRequest listTablesRequest)
                                throws Exception

Description copied from class: MetadataHandler

Used to get the list of tables that this source contains.

Specified by:

doListTables in class MetadataHandler

Parameters:

blockAllocator - Tool for creating and managing Apache Arrow Blocks.

listTablesRequest - Provides details on who made the request and which Athena catalog and database they are querying.

Returns:

A ListTablesResponse which primarily contains a List enumerating the tables in this catalog, database tuple. It also contains the catalog name corresponding the Athena catalog that was queried.

Throws:

Exception

listPaginatedTables

protected ListTablesResponse listPaginatedTables(Connection connection,
                                                 ListTablesRequest listTablesRequest)
                                          throws SQLException

Throws:

SQLException

listTables

protected List<TableName> listTables(Connection jdbcConnection,
                                     String databaseName)
                              throws SQLException

Throws:

SQLException

escapeNamePattern

protected String escapeNamePattern(String name,
                                   String escape)

doGetTable

public GetTableResponse doGetTable(BlockAllocator blockAllocator,
                                   GetTableRequest getTableRequest)
                            throws Exception

Description copied from class: MetadataHandler

Used to get definition (field names, types, descriptions, etc...) of a Table.

Specified by:

doGetTable in class MetadataHandler

Parameters:

blockAllocator - Tool for creating and managing Apache Arrow Blocks.

getTableRequest - Provides details on who made the request and which Athena catalog, database, and table they are querying.

Returns:

A GetTableResponse which primarily contains: 1. An Apache Arrow Schema object describing the table's columns, types, and descriptions. 2. A Set of partition column names (or empty if the table isn't partitioned).

Throws:

Exception

doGetQueryPassthroughSchema

public GetTableResponse doGetQueryPassthroughSchema(BlockAllocator blockAllocator,
                                                    GetTableRequest getTableRequest)
                                             throws Exception

Description copied from class: MetadataHandler

Used to get definition (field names, types, descriptions, etc...) of a Query PassThrough.

Overrides:

doGetQueryPassthroughSchema in class MetadataHandler

Parameters:

blockAllocator - Tool for creating and managing Apache Arrow Blocks.

getTableRequest - Provides details on who made the request and which Athena catalog, database, and table they are querying.

Returns:

A GetTableResponse which primarily contains: 1. An Apache Arrow Schema object describing the table's columns, types, and descriptions. 2. A Set of partition column names (or empty if the table isn't partitioned).

Throws:

Exception

convertDatasourceTypeToArrow

protected org.apache.arrow.vector.types.pojo.ArrowType convertDatasourceTypeToArrow(int columnIndex,
                                                                                    int precision,
                                                                                    Map<String,String> configOptions,
                                                                                    ResultSetMetaData metadata)
                                                                             throws SQLException

A method that takes in a JDBC type; and converts it to Arrow Type This can be overriden by other Metadata Handlers extending JDBC

Parameters:

columnIndex -

precision -

configOptions -

metadata -

Returns:

Arrow Type

Throws:

SQLException

caseInsensitiveTableSearch

protected TableName caseInsensitiveTableSearch(Connection connection,
                                               String databaseName,
                                               String tableName)
                                        throws Exception

While being a no-op by default, this function will be overriden by subclasses that support this search.

Parameters:

connection -

databaseName -

tableName -

Returns:

TableName containing the resolved case sensitive table name.

Throws:

Exception

getPartitionSchema

public abstract org.apache.arrow.vector.types.pojo.Schema getPartitionSchema(String catalogName)

Delegates creation of partition schema to database type implementation.

Parameters:

catalogName - Athena provided catalog name.

Returns:

schema. See Schema

getPartitions

public abstract void getPartitions(BlockWriter blockWriter,
                                   GetTableLayoutRequest request,
                                   QueryStatusChecker queryStatusChecker)
                            throws Exception

Description copied from class: MetadataHandler

Used to get the partitions that must be read from the request table in order to satisfy the requested predicate.

Specified by:

getPartitions in class MetadataHandler

Parameters:

blockWriter - Used to write rows (partitions) into the Apache Arrow response.

request - Provides details of the catalog, database, and table being queried as well as any filter predicate.

queryStatusChecker - A QueryStatusChecker that you can use to stop doing work for a query that has already terminated

Throws:

Exception

doGetSplits

public abstract GetSplitsResponse doGetSplits(BlockAllocator blockAllocator,
                                              GetSplitsRequest getSplitsRequest)

Description copied from class: MetadataHandler

Used to split-up the reads required to scan the requested batch of partition(s).

Specified by:

doGetSplits in class MetadataHandler

Parameters:

blockAllocator - Tool for creating and managing Apache Arrow Blocks.

getSplitsRequest - Provides details of the catalog, database, table, andpartition(s) being queried as well as any filter predicate.

Returns:

A GetSplitsResponse which primarily contains: 1. A Set which represent read operations Amazon Athena must perform by calling your read function. 2. (Optional) A continuation token which allows you to paginate the generation of splits for large queries.

getSplitClauses

protected List<String> getSplitClauses(TableName tableName)

getArrayArrowTypeFromTypeName

protected org.apache.arrow.vector.types.pojo.ArrowType getArrayArrowTypeFromTypeName(String typeName,
                                                                                     int precision,
                                                                                     int scale)

Converts an ARRAY column's TYPE_NAME (provided by the jdbc metadata) to an ArrowType.

Parameters:

typeName - The column's TYPE_NAME (e.g. _int4, _text, _float8, etc...)

precision - Used for BigDecimal ArrowType

scale - Used for BigDecimal ArrowType

Returns:

Utf8 ArrowType (VARCHAR)

setupQueryPassthroughSplit

protected GetSplitsResponse setupQueryPassthroughSplit(GetSplitsRequest request)

Helper function that provides a single partition for Query Pass-Through