aws_ddk_core.pipelines.DataPipeline

class aws_ddk_core.pipelines.DataPipeline(*args: Any, **kwargs)

Class that represents a data pipeline. Used to wire stages via Event Rules. For example:

DataPipeline(self, id, description="My pipeline")
    .add_stage(my_lambda_stage)
    .add_stage(my_glue_stage)
__init__(scope: constructs.Construct, id: str, name: Optional[str] = None, description: Optional[str] = None) None

Create a data pipeline.

Parameters
  • scope (Construct) – Scope within which this construct is defined

  • id (str) – Identifier of the pipeline

  • name (Optional[str]) – Name of the pipeline

  • description (Optional[str]) – Description of the pipeline

Methods

__init__(scope, id[, name, description])

Create a data pipeline.

add_notifications([notifications_topic])

Create a rule that matches specificed event pattern with the specified target.

add_rule([id, event_pattern, event_targets, ...])

Create a rule that matches specificed event pattern with the specified target.

add_stage(stage[, skip_rule, override_rule, ...])

Add a stage to the data pipeline.

is_construct(x)

Checks if x is a construct.

to_string()

Returns a string representation of this construct.

Attributes

node

The tree node.

add_notifications(notifications_topic: Optional[aws_cdk.aws_sns.ITopic] = None) aws_ddk_core.pipelines.pipeline.DataPipeline

Create a rule that matches specificed event pattern with the specified target.

Parameters

notifications_topic (Optional[ITopic]) – Existing SNS Topic to consume notifications with.

Returns

pipeline – DataPipeline

Return type

DataPipeline

add_rule(id: Optional[str] = None, event_pattern: Optional[aws_cdk.aws_events.EventPattern] = None, event_targets: Optional[List[aws_cdk.aws_events.IRuleTarget]] = None, override_rule: Optional[aws_cdk.aws_events.Rule] = None, rule_name: Optional[str] = None) aws_ddk_core.pipelines.pipeline.DataPipeline

Create a rule that matches specificed event pattern with the specified target.

Parameters
  • id (Optional[str]) – Identifier of the rule

  • event_pattern (Optional[EventPattern]) – Event pattern of the rule

  • event_targets (Optional[List[IRuleTarget]]) – Target of the rule - usually previous_stage.get_targets()

  • override_rule (Optional[Rule]) – Custom rule

  • rule_name (Optional[str]) – Name of Event Rule. Default - Auto-generated by CloudFormation

Returns

pipeline – DataPipeline

Return type

DataPipeline

add_stage(stage: aws_ddk_core.pipelines.stage.DataStage, skip_rule: bool = False, override_rule: Optional[aws_cdk.aws_events.Rule] = None, rule_name: Optional[str] = None) aws_ddk_core.pipelines.pipeline.DataPipeline

Add a stage to the data pipeline.

By default, creates an Event Rule that matches output event pattern of the previous stage with targets of the next one.

Parameters
  • stage (DataStage) – Stage instance

  • skip_rule (bool) – Skip creation of the default rule

  • override_rule (Optional[Rule]) – Override the default rule by the rule specified in this parameter

  • rule_name (Optional[str]) – Name of Event Rule. Default - Auto-generated by CloudFormation

Returns

pipeline – DataPipeline

Return type

DataPipeline