Package com.amazonaws.athena.connectors.dynamodb

Class DynamoDBMetadataHandler

  • All Implemented Interfaces:
    com.amazonaws.services.lambda.runtime.RequestStreamHandler
    public class DynamoDBMetadataHandler
extends GlueMetadataHandler
    Handles metadata requests for the Athena DynamoDB Connector.

    For more detail, please see the module's README.md, some notable characteristics of this class include:

    1. Glue DataCatalog is used for schema information by default unless disabled. If disabled or the table
    is not found, it falls back to doing a small table scan and derives a schema from that.
    2. Determines if the data splits will need to perform DDB Queries or Scans.
    3. Splits up the hash key into distinct Query splits if possible, otherwise falls back to creating Scan splits.
    4. Also determines the best index to use (if available) if the available predicates align with Key Attributes.
    5. Creates scan splits that support Parallel Scan and tries to choose the optimal number of splits.
    6. Pushes down all other predicates into ready-to-use filter expressions to pass to DDB.

    • Constructor Detail

      • DynamoDBMetadataHandler

        public DynamoDBMetadataHandler​(Map<String,​String> configOptions)

    • Method Detail

      • doGetDataSourceCapabilities

        public GetDataSourceCapabilitiesResponse doGetDataSourceCapabilities​(BlockAllocator allocator,
                                                                     GetDataSourceCapabilitiesRequest request)
        Description copied from class: MetadataHandler
        Used to describe the types of capabilities supported by a data source. An engine can use this to determine what portions of the query to push down. A connector that returns any optimization will guarantee that the associated predicate will be pushed down.
        Overrides:
        doGetDataSourceCapabilities in class MetadataHandler
        Parameters:
        allocator - Tool for creating and managing Apache Arrow Blocks.
        request - Provides details about the catalog being used.
        Returns:
        A GetDataSourceCapabilitiesResponse object which returns a map of supported optimizations that the connector is advertising to the consumer. The connector assumes all responsibility for whatever is passed here.

      • doListSchemaNames

        public ListSchemasResponse doListSchemaNames​(BlockAllocator allocator,
                                             ListSchemasRequest request)
                                      throws Exception
        Since DynamoDB does not have "schemas" or "databases", this lists all the Glue databases (if not disabled) that contain "dynamo-db-flag" in their URIs . Otherwise returns just a "default" schema.
        Overrides:
        doListSchemaNames in class GlueMetadataHandler
        Parameters:
        allocator - Tool for creating and managing Apache Arrow Blocks.
        request - Provides details on who made the request and which Athena catalog they are querying.
        Returns:
        The ListSchemasResponse which mostly contains the list of schemas (aka databases).
        Throws:
        Exception
        See Also:
        GlueMetadataHandler

      • doListTables

        public ListTablesResponse doListTables​(BlockAllocator allocator,
                                       ListTablesRequest request)
                                throws Exception
        Lists all Glue tables (if not disabled) in the schema specified that indicate use for DynamoDB metadata. Indications for DynamoDB use in Glue are:
        1. The top level table properties/parameters contains a key called "classification" with value "dynamodb".
        2. Or the storage descriptor's location field contains "dynamodb".
        3. Or the storage descriptor has a parameter called "classification" with value "dynamodb".

        If the specified schema is "default", this also returns an intersection with actual tables in DynamoDB. Pagination only implemented for DynamoDBTableResolver.listTables()

        Overrides:
        doListTables in class GlueMetadataHandler
        Parameters:
        allocator - Tool for creating and managing Apache Arrow Blocks.
        request - Provides details on who made the request and which Athena catalog they are querying.
        Returns:
        The ListTablesResponse which mostly contains the list of table names.
        Throws:
        Exception
        See Also:
        GlueMetadataHandler

      • doGetQueryPassthroughSchema

        public GetTableResponse doGetQueryPassthroughSchema​(BlockAllocator allocator,
                                                    GetTableRequest request)
                                             throws Exception
        Description copied from class: MetadataHandler
        Used to get definition (field names, types, descriptions, etc...) of a Query PassThrough.
        Overrides:
        doGetQueryPassthroughSchema in class MetadataHandler
        Parameters:
        allocator - Tool for creating and managing Apache Arrow Blocks.
        request - Provides details on who made the request and which Athena catalog, database, and table they are querying.
        Returns:
        A GetTableResponse which primarily contains: 1. An Apache Arrow Schema object describing the table's columns, types, and descriptions. 2. A Set of partition column names (or empty if the table isn't partitioned).
        Throws:
        Exception

      • doGetTable

        public GetTableResponse doGetTable​(BlockAllocator allocator,
                                   GetTableRequest request)
                            throws Exception
        Fetches a table's schema from Glue DataCatalog if present and not disabled, otherwise falls back to doing a small table scan derives a schema from that.
        Overrides:
        doGetTable in class GlueMetadataHandler
        Parameters:
        allocator - Tool for creating and managing Apache Arrow Blocks.
        request - Provides details on who made the request and which Athena catalog, database, and table they are querying.
        Returns:
        A GetTableResponse mostly containing the columns, their types, and any table properties for the requested table.
        Throws:
        Exception
        See Also:
        GlueMetadataHandler

      • enhancePartitionSchema

        public void enhancePartitionSchema​(SchemaBuilder partitionSchemaBuilder,
                                   GetTableLayoutRequest request)
        Generates a partition schema with metadata derived from available predicates. This metadata will be copied to splits in the #doGetSplits call. At this point it is determined whether we can partition by hash key or fall back to a full table scan.
        Overrides:
        enhancePartitionSchema in class MetadataHandler
        Parameters:
        partitionSchemaBuilder - The SchemaBuilder you can use to add additional columns and metadata to the partitions response.
        request - The GetTableLayoutResquest that triggered this call.
        See Also:
        GlueMetadataHandler

      • getPartitions

        public void getPartitions​(BlockWriter blockWriter,
                          GetTableLayoutRequest request,
                          QueryStatusChecker queryStatusChecker)
                   throws Exception
        Generates hash key partitions if possible or generates a single partition with the heuristically determined optimal scan segment count specified inside of it
        Specified by:
        getPartitions in class MetadataHandler
        Parameters:
        blockWriter - Used to write rows (partitions) into the Apache Arrow response.
        request - Provides details of the catalog, database, and table being queried as well as any filter predicate.
        queryStatusChecker - A QueryStatusChecker that you can use to stop doing work for a query that has already terminated
        Throws:
        Exception
        See Also:
        GlueMetadataHandler

      • doGetSplits

        public GetSplitsResponse doGetSplits​(BlockAllocator allocator,
                                     GetSplitsRequest request)
        Copies data from partitions and creates splits, serializing as necessary for later calls to RecordHandler#readWithContraint. This API supports pagination.
        Specified by:
        doGetSplits in class MetadataHandler
        Parameters:
        allocator - Tool for creating and managing Apache Arrow Blocks.
        request - Provides details of the catalog, database, table, andpartition(s) being queried as well as any filter predicate.
        Returns:
        A GetSplitsResponse which primarily contains: 1. A Set which represent read operations Amazon Athena must perform by calling your read function. 2. (Optional) A continuation token which allows you to paginate the generation of splits for large queries.
        See Also:
        GlueMetadataHandler

      • convertField

        protected org.apache.arrow.vector.types.pojo.Field convertField​(String name,
                                                                String glueType)
        Description copied from class: GlueMetadataHandler
        Maps a Glue field to an Apache Arrow Field.
        Overrides:
        convertField in class GlueMetadataHandler
        Parameters:
        name - The name of the field in Glue.
        glueType - The type of the field in Glue.
        Returns:
        The corresponding Apache Arrow Field.
        See Also:
        GlueMetadataHandler