aws_ddk_core.pipelines.DataPipeline¶

class aws_ddk_core.pipelines.DataPipeline(*args: Any, **kwargs)¶

Class that represents a data pipeline. Used to wire stages via Event Rules. For example:

DataPipeline(self, id, description="My pipeline")
    .add_stage(my_lambda_stage)
    .add_stage(my_glue_stage)

__init__(scope: constructs.Construct, id: str, name: Optional[str] = None, description: Optional[str] = None) → None¶

Create a data pipeline.

Parameters

scope (Construct) – Scope within which this construct is defined
id (str) – Identifier of the pipeline
name (Optional[str]) – Name of the pipeline
description (Optional[str]) – Description of the pipeline

Methods

`__init__`(scope, id[, name, description])	Create a data pipeline.
`add_notifications`([notifications_topic])	Create a rule that matches specificed event pattern with the specified target.
`add_rule`([id, event_pattern, event_targets, ...])	Create a rule that matches specificed event pattern with the specified target.
`add_stage`(stage[, skip_rule, override_rule])	Add a stage to the data pipeline.
`is_construct`(x)	Checks if `x` is a construct.
`to_string`()	Returns a string representation of this construct.

Attributes

node

The tree node.

add_notifications(notifications_topic: Optional[aws_cdk.aws_sns.ITopic] = None) → aws_ddk_core.pipelines.pipeline.DataPipeline¶

Create a rule that matches specificed event pattern with the specified target.

Parameters: notifications_topic (Optional[ITopic]) – Existing SNS Topic to consume notifications with.
Returns: pipeline – DataPipeline
Return type: DataPipeline

add_rule(id: Optional[str] = None, event_pattern: Optional[aws_cdk.aws_events.EventPattern] = None, event_targets: Optional[List[aws_cdk.aws_events.IRuleTarget]] = None, override_rule: Optional[aws_cdk.aws_events.Rule] = None) → aws_ddk_core.pipelines.pipeline.DataPipeline¶

Create a rule that matches specificed event pattern with the specified target.

Parameters

id (Optional[str]) – Identifier of the rule
event_pattern (Optional[EventPattern]) – Event pattern of the rule
event_targets (Optional[List[IRuleTarget]]) – Target of the rule - usually previous_stage.get_targets()
override_rule (Optional[Rule]) – Custom rule

Returns

pipeline – DataPipeline

Return type

DataPipeline

add_stage(stage: aws_ddk_core.pipelines.stage.DataStage, skip_rule: bool = False, override_rule: Optional[aws_cdk.aws_events.Rule] = None) → aws_ddk_core.pipelines.pipeline.DataPipeline¶

Add a stage to the data pipeline.

By default, creates an Event Rule that matches output event pattern of the previous stage with targets of the next one.

Parameters

stage (DataStage) – Stage instance
skip_rule (bool) – Skip creation of the default rule
override_rule (Optional[Rule]) – Override the default rule by the rule specified in this parameter

Returns

pipeline – DataPipeline

Return type

DataPipeline