Package com.amazonaws.athena.connectors.gcs

Class GcsMetadataHandler

  • All Implemented Interfaces:
    com.amazonaws.services.lambda.runtime.RequestStreamHandler
    public class GcsMetadataHandler
extends GlueMetadataHandler

    • Constructor Detail

      • GcsMetadataHandler

        protected GcsMetadataHandler​(EncryptionKeyFactory keyFactory,
                             software.amazon.awssdk.services.secretsmanager.SecretsManagerClient awsSecretsManager,
                             software.amazon.awssdk.services.athena.AthenaClient athena,
                             String spillBucket,
                             String spillPrefix,
                             software.amazon.awssdk.services.glue.GlueClient glueClient,
                             org.apache.arrow.memory.BufferAllocator allocator,
                             Map<String,​String> configOptions)
                      throws IOException
        Throws:
        IOException

    • Method Detail

      • doListSchemaNames

        public ListSchemasResponse doListSchemaNames​(BlockAllocator allocator,
                                             ListSchemasRequest request)
                                      throws Exception
        Used to get the list of schemas (aka databases) that this source contains.
        Overrides:
        doListSchemaNames in class GlueMetadataHandler
        Parameters:
        allocator - Tool for creating and managing Apache Arrow Blocks.
        request - Provides details on who made the request and which Athena catalog they are querying.
        Returns:
        A ListSchemasResponse which primarily contains a Set of schema names and a catalog name corresponding the Athena catalog that was queried.
        Throws:
        Exception

      • doListTables

        public ListTablesResponse doListTables​(BlockAllocator allocator,
                                       ListTablesRequest request)
                                throws Exception
        Used to get the list of tables that this source contains.
        Overrides:
        doListTables in class GlueMetadataHandler
        Parameters:
        allocator - Tool for creating and managing Apache Arrow Blocks.
        request - Provides details on who made the request and which Athena catalog and database they are querying.
        Returns:
        A ListTablesResponse which primarily contains a List enumerating the tables in this catalog, database tuple. It also contains the catalog name corresponding the Athena catalog that was queried.
        Throws:
        Exception

      • doGetTable

        public GetTableResponse doGetTable​(BlockAllocator blockAllocator,
                                   GetTableRequest request)
                            throws Exception
        Used to get definition (field names, types, descriptions, etc...) of a Table.
        Overrides:
        doGetTable in class GlueMetadataHandler
        Parameters:
        blockAllocator - Tool for creating and managing Apache Arrow Blocks.
        request - Provides details on who made the request and which Athena catalog, database, and table they are querying.
        Returns:
        A GetTableResponse which primarily contains: 1. An Apache Arrow Schema object describing the table's columns, types, and descriptions. 2. A Set of partition column names (or empty if the table isn't partitioned). 3. A TableName object confirming the schema and table name the response is for. 4. A catalog name corresponding the Athena catalog that was queried.
        Throws:
        Exception

      • getPartitions

        public void getPartitions​(BlockWriter blockWriter,
                          GetTableLayoutRequest request,
                          QueryStatusChecker queryStatusChecker)
                   throws URISyntaxException
        Used to get the partitions that must be read from the request table in order to satisfy the requested predicate.
        Specified by:
        getPartitions in class MetadataHandler
        Parameters:
        blockWriter - Used to write rows (partitions) into the Apache Arrow response.
        request - Provides details of the catalog, database, and table being queried as well as any filter predicate.
        queryStatusChecker - A QueryStatusChecker that you can use to stop doing work for a query that has already terminated
        Throws:
        URISyntaxException

      • doGetSplits

        public GetSplitsResponse doGetSplits​(BlockAllocator allocator,
                                     GetSplitsRequest request)
                              throws Exception
        Used to split up the reads required to scan the requested batch of partition(s).

        Here we execute the read operations files form particular GCS bucket

        Specified by:
        doGetSplits in class MetadataHandler
        Parameters:
        allocator - Tool for creating and managing Apache Arrow Blocks.
        request - Provides details of the catalog, database, table, and partition(s) being queried as well as any filter predicate.
        Returns:
        A GetSplitsResponse which primarily contains: 1. A Set which represent read operations Amazon Athena must perform by calling your read function. 2. (Optional) A continuation token which allows you to paginate the generation of splits for large queries.
        Throws:
        Exception