Package com.amazonaws.athena.connectors.docdb

Class DocDBMetadataHandler

  • All Implemented Interfaces:
    com.amazonaws.services.lambda.runtime.RequestStreamHandler
    public class DocDBMetadataHandler
extends GlueMetadataHandler
    Handles metadata requests for the Athena DocumentDB Connector.

    For more detail, please see the module's README.md, some notable characteristics of this class include:

    1. Uses a Glue table property (docfb-metadata-flag) to indicate that the table (whose name matched the DocDB collection name) can indeed be used to supplement metadata from DocDB itself. 2. Attempts to resolve sensitive fields such as DocDB connection strings via SecretsManager so that you can substitute variables with values from by doing something like: mongodb://${docdb_instance_1_creds}@myhostname.com:123/?ssl=true&ssl_ca_certs=rds-combined-ca-bundle.pem&replicaSet=rs0

    • Constructor Detail

      • DocDBMetadataHandler

        public DocDBMetadataHandler​(Map<String,​String> configOptions)

      • DocDBMetadataHandler

        protected DocDBMetadataHandler​(com.amazonaws.services.glue.AWSGlue glue,
                               DocDBConnectionFactory connectionFactory,
                               EncryptionKeyFactory keyFactory,
                               com.amazonaws.services.secretsmanager.AWSSecretsManager secretsManager,
                               com.amazonaws.services.athena.AmazonAthena athena,
                               String spillBucket,
                               String spillPrefix,
                               Map<String,​String> configOptions)

    • Method Detail

      • doGetDataSourceCapabilities

        public GetDataSourceCapabilitiesResponse doGetDataSourceCapabilities​(BlockAllocator allocator,
                                                                     GetDataSourceCapabilitiesRequest request)
        Description copied from class: MetadataHandler
        Used to describe the types of capabilities supported by a data source. An engine can use this to determine what portions of the query to push down. A connector that returns any optimization will guarantee that the associated predicate will be pushed down.
        Overrides:
        doGetDataSourceCapabilities in class MetadataHandler
        Parameters:
        allocator - Tool for creating and managing Apache Arrow Blocks.
        request - Provides details about the catalog being used.
        Returns:
        A GetDataSourceCapabilitiesResponse object which returns a map of supported optimizations that the connector is advertising to the consumer. The connector assumes all responsibility for whatever is passed here.

      • doListTables

        public ListTablesResponse doListTables​(BlockAllocator blockAllocator,
                                       ListTablesRequest request)
                                throws Exception
        List collections in the requested schema in your DocumentDB instance treating the requested schema as an DocumentDB database.
        Overrides:
        doListTables in class GlueMetadataHandler
        Parameters:
        blockAllocator - Tool for creating and managing Apache Arrow Blocks.
        request - Provides details on who made the request and which Athena catalog they are querying.
        Returns:
        The ListTablesResponse which mostly contains the list of table names.
        Throws:
        Exception
        See Also:
        GlueMetadataHandler

      • doGetTable

        public GetTableResponse doGetTable​(BlockAllocator blockAllocator,
                                   GetTableRequest request)
                            throws Exception
        If Glue is enabled as a source of supplemental metadata we look up the requested Schema/Table in Glue and filters out any results that don't have the DOCDB_METADATA_FLAG set. If no matching results were found in Glue, then we resort to inferring the schema of the DocumentDB collection using SchemaUtils.inferSchema(...). If there is no such table in DocumentDB the operation will fail.
        Overrides:
        doGetTable in class GlueMetadataHandler
        Parameters:
        blockAllocator - Tool for creating and managing Apache Arrow Blocks.
        request - Provides details on who made the request and which Athena catalog, database, and table they are querying.
        Returns:
        A GetTableResponse mostly containing the columns, their types, and any table properties for the requested table.
        Throws:
        Exception
        See Also:
        GlueMetadataHandler

      • doGetQueryPassthroughSchema

        public GetTableResponse doGetQueryPassthroughSchema​(BlockAllocator allocator,
                                                    GetTableRequest request)
                                             throws Exception
        Description copied from class: MetadataHandler
        Used to get definition (field names, types, descriptions, etc...) of a Query PassThrough.
        Overrides:
        doGetQueryPassthroughSchema in class MetadataHandler
        Parameters:
        allocator - Tool for creating and managing Apache Arrow Blocks.
        request - Provides details on who made the request and which Athena catalog, database, and table they are querying.
        Returns:
        A GetTableResponse which primarily contains: 1. An Apache Arrow Schema object describing the table's columns, types, and descriptions. 2. A Set of partition column names (or empty if the table isn't partitioned).
        Throws:
        Exception

      • getPartitions

        public void getPartitions​(BlockWriter blockWriter,
                          GetTableLayoutRequest request,
                          QueryStatusChecker queryStatusChecker)
                   throws Exception
        Our table doesn't support complex layouts or partitioning so we simply make this method a NoOp.
        Specified by:
        getPartitions in class MetadataHandler
        Parameters:
        blockWriter - Used to write rows (partitions) into the Apache Arrow response.
        request - Provides details of the catalog, database, and table being queried as well as any filter predicate.
        queryStatusChecker - A QueryStatusChecker that you can use to stop doing work for a query that has already terminated
        Throws:
        Exception
        See Also:
        GlueMetadataHandler

      • doGetSplits

        public GetSplitsResponse doGetSplits​(BlockAllocator blockAllocator,
                                     GetSplitsRequest request)
        Since our connector does not support parallel scans we generate a single Split and include the connection details as a property on the split so that the RecordHandler has easy access to it.
        Specified by:
        doGetSplits in class MetadataHandler
        Parameters:
        blockAllocator - Tool for creating and managing Apache Arrow Blocks.
        request - Provides details of the catalog, database, table, andpartition(s) being queried as well as any filter predicate.
        Returns:
        A GetSplitsResponse which primarily contains: 1. A Set which represent read operations Amazon Athena must perform by calling your read function. 2. (Optional) A continuation token which allows you to paginate the generation of splits for large queries.
        See Also:
        GlueMetadataHandler

      • convertField

        protected org.apache.arrow.vector.types.pojo.Field convertField​(String name,
                                                                String glueType)
        Description copied from class: GlueMetadataHandler
        Maps a Glue field to an Apache Arrow Field.
        Overrides:
        convertField in class GlueMetadataHandler
        Parameters:
        name - The name of the field in Glue.
        glueType - The type of the field in Glue.
        Returns:
        The corresponding Apache Arrow Field.
        See Also:
        GlueMetadataHandler