Package com.amazonaws.athena.connectors.tpcds

Class TPCDSMetadataHandler

  • All Implemented Interfaces:
    com.amazonaws.services.lambda.runtime.RequestStreamHandler
    public class TPCDSMetadataHandler
extends MetadataHandler
    Handles metadata requests for the Athena TPC-DS Connector.

    For more detail, please see the module's README.md, some notable characteristics of this class include:

    1. Provides 5 Schems, each representing a different scale factor (1,10,100,250,1000) 2. Each schema has 25 TPC-DS tables 3. Each table is divided into NUM_SPLITS splits * scale_factor/10

    • Constructor Detail

      • TPCDSMetadataHandler

        public TPCDSMetadataHandler​(Map<String,​String> configOptions)

      • TPCDSMetadataHandler

        protected TPCDSMetadataHandler​(EncryptionKeyFactory keyFactory,
                               software.amazon.awssdk.services.secretsmanager.SecretsManagerClient secretsManager,
                               software.amazon.awssdk.services.athena.AthenaClient athena,
                               String spillBucket,
                               String spillPrefix,
                               Map<String,​String> configOptions)

    • Method Detail

      • doGetDataSourceCapabilities

        public GetDataSourceCapabilitiesResponse doGetDataSourceCapabilities​(BlockAllocator allocator,
                                                                     GetDataSourceCapabilitiesRequest request)
        Description copied from class: MetadataHandler
        Used to describe the types of capabilities supported by a data source. An engine can use this to determine what portions of the query to push down. A connector that returns any optimization will guarantee that the associated predicate will be pushed down.
        Overrides:
        doGetDataSourceCapabilities in class MetadataHandler
        Parameters:
        allocator - Tool for creating and managing Apache Arrow Blocks.
        request - Provides details about the catalog being used.
        Returns:
        A GetDataSourceCapabilitiesResponse object which returns a map of supported optimizations that the connector is advertising to the consumer. The connector assumes all responsibility for whatever is passed here.

      • doListSchemaNames

        public ListSchemasResponse doListSchemaNames​(BlockAllocator allocator,
                                             ListSchemasRequest request)
        Returns our static list of schemas which correspond to the scale factor of the dataset we will generate.
        Specified by:
        doListSchemaNames in class MetadataHandler
        Parameters:
        allocator - Tool for creating and managing Apache Arrow Blocks.
        request - Provides details on who made the request and which Athena catalog they are querying.
        Returns:
        A ListSchemasResponse which primarily contains a Set of schema names and a catalog name corresponding the Athena catalog that was queried.
        See Also:
        MetadataHandler

      • doListTables

        public ListTablesResponse doListTables​(BlockAllocator allocator,
                                       ListTablesRequest request)
        Used to get the list of static tables from TerraData's TPCDS generator.
        Specified by:
        doListTables in class MetadataHandler
        Parameters:
        allocator - Tool for creating and managing Apache Arrow Blocks.
        request - Provides details on who made the request and which Athena catalog and database they are querying.
        Returns:
        A ListTablesResponse which primarily contains a List enumerating the tables in this catalog, database tuple. It also contains the catalog name corresponding the Athena catalog that was queried.
        See Also:
        MetadataHandler

      • doGetTable

        public GetTableResponse doGetTable​(BlockAllocator allocator,
                                   GetTableRequest request)
        Used to get definition (field names, types, descriptions, etc...) of a Table using the static metadata provided by TerraData's TPCDS generator.
        Specified by:
        doGetTable in class MetadataHandler
        Parameters:
        allocator - Tool for creating and managing Apache Arrow Blocks.
        request - Provides details on who made the request and which Athena catalog, database, and table they are querying.
        Returns:
        A GetTableResponse which primarily contains: 1. An Apache Arrow Schema object describing the table's columns, types, and descriptions. 2. A Set of partition column names (or empty if the table isn't partitioned).
        See Also:
        MetadataHandler

      • doGetQueryPassthroughSchema

        public GetTableResponse doGetQueryPassthroughSchema​(BlockAllocator allocator,
                                                    GetTableRequest request)
                                             throws Exception
        Description copied from class: MetadataHandler
        Used to get definition (field names, types, descriptions, etc...) of a Query PassThrough.
        Overrides:
        doGetQueryPassthroughSchema in class MetadataHandler
        Parameters:
        allocator - Tool for creating and managing Apache Arrow Blocks.
        request - Provides details on who made the request and which Athena catalog, database, and table they are querying.
        Returns:
        A GetTableResponse which primarily contains: 1. An Apache Arrow Schema object describing the table's columns, types, and descriptions. 2. A Set of partition column names (or empty if the table isn't partitioned).
        Throws:
        Exception

      • getPartitions

        public void getPartitions​(BlockWriter blockWriter,
                          GetTableLayoutRequest request,
                          QueryStatusChecker queryStatusChecker)
                   throws Exception
        We do not support partitioning at this time since Partition Pruning Performance is not part of the dimensions we test using TPCDS. By making this a NoOp the Athena Federation SDK will automatically generate a single placeholder partition to signal to Athena that there is indeed data that needs to be read and that it should call get splits.
        Specified by:
        getPartitions in class MetadataHandler
        Parameters:
        blockWriter - Used to write rows (partitions) into the Apache Arrow response.
        request - Provides details of the catalog, database, and table being queried as well as any filter predicate.
        queryStatusChecker - A QueryStatusChecker that you can use to stop doing work for a query that has already terminated
        Throws:
        Exception
        See Also:
        MetadataHandler

      • doGetSplits

        public GetSplitsResponse doGetSplits​(BlockAllocator allocator,
                                     GetSplitsRequest request)
        Used to split-up the reads required to scan the requested batch of partition(s). We are generating a fixed number of splits based on the scale factor.
        Specified by:
        doGetSplits in class MetadataHandler
        Parameters:
        allocator - Tool for creating and managing Apache Arrow Blocks.
        request - Provides details of the catalog, database, table, andpartition(s) being queried as well as any filter predicate.
        Returns:
        A GetSplitsResponse which primarily contains: 1. A Set which represent read operations Amazon Athena must perform by calling your read function. 2. (Optional) A continuation token which allows you to paginate the generation of splits for large queries.
        See Also:
        MetadataHandler

      • setupQueryPassthroughSplit

        protected GetSplitsResponse setupQueryPassthroughSplit​(GetSplitsRequest request)
        Helper function that provides a single partition for Query Pass-Through