Package com.amazonaws.athena.connectors.google.bigquery

Class BigQueryMetadataHandler

java.lang.Object

com.amazonaws.athena.connector.lambda.handlers.MetadataHandler

com.amazonaws.athena.connectors.google.bigquery.BigQueryMetadataHandler

All Implemented Interfaces:

FederationRequestHandler, com.amazonaws.services.lambda.runtime.RequestStreamHandler

public class BigQueryMetadataHandler
extends MetadataHandler

Field Summary

Fields inherited from class com.amazonaws.athena.connector.lambda.handlers.MetadataHandler

configOptions, DISABLE_SPILL_ENCRYPTION, KMS_KEY_ID_ENV, SPILL_BUCKET_ENV, SPILL_PREFIX_ENV

Method Summary

Modifier and Type	Method	Description
GetDataSourceCapabilitiesResponse	doGetDataSourceCapabilities(BlockAllocator allocator, GetDataSourceCapabilitiesRequest request)	Used to describe the types of capabilities supported by a data source.
GetTableResponse	doGetQueryPassthroughSchema(BlockAllocator allocator, GetTableRequest request)	Used to get definition (field names, types, descriptions, etc...) of a Query PassThrough.
GetSplitsResponse	doGetSplits(BlockAllocator allocator, GetSplitsRequest request)	Making minimum(10) splits based on constraints.
GetTableResponse	doGetTable(BlockAllocator blockAllocator, GetTableRequest getTableRequest)	Used to get definition (field names, types, descriptions, etc...) of a Table.
ListSchemasResponse	doListSchemaNames(BlockAllocator blockAllocator, ListSchemasRequest listSchemasRequest)	Used to get the list of schemas (aka databases) that this source contains.
ListTablesResponse	doListTables(BlockAllocator blockAllocator, ListTablesRequest listTablesRequest)	Used to get the list of tables that this source contains.
void	getPartitions(BlockWriter blockWriter, GetTableLayoutRequest request, QueryStatusChecker queryStatusChecker)	Currently not supporting Partitions since Bigquery having quota limits with triggering concurrent queries and having bit complexity to extract and use the partitions in the query instead we are using limit and offset for non constraints query with basic concurrency limit

Methods inherited from class com.amazonaws.athena.connector.lambda.handlers.MetadataHandler

doGetTableLayout, doHandleRequest, doPing, enhancePartitionSchema, getCachableSecretsManager, getRequestOverrideConfig, getSecret, handleRequest, makeEncryptionKey, makeSpillLocation, onPing, resolveSecrets, resolveWithDefaultCredentials

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface com.amazonaws.athena.connector.lambda.handlers.FederationRequestHandler

getAthenaClient, getRequestOverrideConfig, getS3Client, getSessionCredentials

Method Detail

doGetDataSourceCapabilities

public GetDataSourceCapabilitiesResponse doGetDataSourceCapabilities(BlockAllocator allocator,
                                                                     GetDataSourceCapabilitiesRequest request)

Description copied from class: MetadataHandler

Used to describe the types of capabilities supported by a data source. An engine can use this to determine what portions of the query to push down. A connector that returns any optimization will guarantee that the associated predicate will be pushed down.

Overrides:

doGetDataSourceCapabilities in class MetadataHandler

Parameters:

allocator - Tool for creating and managing Apache Arrow Blocks.

request - Provides details about the catalog being used.

Returns:

A GetDataSourceCapabilitiesResponse object which returns a map of supported optimizations that the connector is advertising to the consumer. The connector assumes all responsibility for whatever is passed here.

doListSchemaNames

public ListSchemasResponse doListSchemaNames(BlockAllocator blockAllocator,
                                             ListSchemasRequest listSchemasRequest)
                                      throws IOException

Description copied from class: MetadataHandler

Used to get the list of schemas (aka databases) that this source contains.

Specified by:

doListSchemaNames in class MetadataHandler

Parameters:

blockAllocator - Tool for creating and managing Apache Arrow Blocks.

listSchemasRequest - Provides details on who made the request and which Athena catalog they are querying.

Returns:

A ListSchemasResponse which primarily contains a Set of schema names and a catalog name corresponding the Athena catalog that was queried.

Throws:

IOException

doListTables

public ListTablesResponse doListTables(BlockAllocator blockAllocator,
                                       ListTablesRequest listTablesRequest)
                                throws IOException

Description copied from class: MetadataHandler

Used to get the list of tables that this source contains.

Specified by:

doListTables in class MetadataHandler

Parameters:

blockAllocator - Tool for creating and managing Apache Arrow Blocks.

listTablesRequest - Provides details on who made the request and which Athena catalog and database they are querying.

Returns:

A ListTablesResponse which primarily contains a List enumerating the tables in this catalog, database tuple. It also contains the catalog name corresponding the Athena catalog that was queried.

Throws:

IOException

doGetTable

public GetTableResponse doGetTable(BlockAllocator blockAllocator,
                                   GetTableRequest getTableRequest)
                            throws IOException

Description copied from class: MetadataHandler

Used to get definition (field names, types, descriptions, etc...) of a Table.

Specified by:

doGetTable in class MetadataHandler

Parameters:

blockAllocator - Tool for creating and managing Apache Arrow Blocks.

getTableRequest - Provides details on who made the request and which Athena catalog, database, and table they are querying.

Returns:

A GetTableResponse which primarily contains: 1. An Apache Arrow Schema object describing the table's columns, types, and descriptions. 2. A Set of partition column names (or empty if the table isn't partitioned).

Throws:

IOException

doGetQueryPassthroughSchema

public GetTableResponse doGetQueryPassthroughSchema(BlockAllocator allocator,
                                                    GetTableRequest request)
                                             throws Exception

Description copied from class: MetadataHandler

Used to get definition (field names, types, descriptions, etc...) of a Query PassThrough.

Overrides:

doGetQueryPassthroughSchema in class MetadataHandler

Parameters:

allocator - Tool for creating and managing Apache Arrow Blocks.

request - Provides details on who made the request and which Athena catalog, database, and table they are querying.

Returns:

A GetTableResponse which primarily contains: 1. An Apache Arrow Schema object describing the table's columns, types, and descriptions. 2. A Set of partition column names (or empty if the table isn't partitioned).

Throws:

Exception

getPartitions

public void getPartitions(BlockWriter blockWriter,
                          GetTableLayoutRequest request,
                          QueryStatusChecker queryStatusChecker)

Currently not supporting Partitions since Bigquery having quota limits with triggering concurrent queries and having bit complexity to extract and use the partitions in the query instead we are using limit and offset for non constraints query with basic concurrency limit

Specified by:

getPartitions in class MetadataHandler

Parameters:

blockWriter - Used to write rows (partitions) into the Apache Arrow response.

request - Provides details of the catalog, database, and table being queried as well as any filter predicate.

queryStatusChecker - A QueryStatusChecker that you can use to stop doing work for a query that has already terminated

doGetSplits

public GetSplitsResponse doGetSplits(BlockAllocator allocator,
                                     GetSplitsRequest request)

Making minimum(10) splits based on constraints. Since without constraints query may give lambda timeout if table has large data, concurrencyLimit is configurable and it can be changed based on Google BigQuery Quota Limits.

Specified by:

doGetSplits in class MetadataHandler

Parameters:

allocator - Tool for creating and managing Apache Arrow Blocks.

request - Provides details of the catalog, database, table, and partition(s) being queried as well as any filter predicate.

Returns:

GetSplitsResponse