aws_ddk_core.pipelines.DataPipeline¶
- class aws_ddk_core.pipelines.DataPipeline(*args: Any, **kwargs)¶
Class that represents a data pipeline. Used to wire stages via Event Rules. For example:
DataPipeline(self, id, description="My pipeline") .add_stage(my_lambda_stage) .add_stage(my_glue_stage)
- __init__(scope: constructs.Construct, id: str, name: Optional[str] = None, description: Optional[str] = None) None ¶
Create a data pipeline.
- Parameters
scope (Construct) – Scope within which this construct is defined
id (str) – Identifier of the pipeline
name (Optional[str]) – Name of the pipeline
description (Optional[str]) – Description of the pipeline
Methods
__init__
(scope, id[, name, description])Create a data pipeline.
add_notifications
([notifications_topic])Create a rule that matches specificed event pattern with the specified target.
add_rule
([id, event_pattern, event_targets, ...])Create a rule that matches specificed event pattern with the specified target.
add_stage
(stage[, skip_rule, override_rule, ...])Add a stage to the data pipeline.
is_construct
(x)Checks if
x
is a construct.to_string
()Returns a string representation of this construct.
Attributes
node
The tree node.
- add_notifications(notifications_topic: Optional[aws_cdk.aws_sns.ITopic] = None) aws_ddk_core.pipelines.pipeline.DataPipeline ¶
Create a rule that matches specificed event pattern with the specified target.
- Parameters
notifications_topic (Optional[ITopic]) – Existing SNS Topic to consume notifications with.
- Returns
pipeline – DataPipeline
- Return type
- add_rule(id: Optional[str] = None, event_pattern: Optional[aws_cdk.aws_events.EventPattern] = None, event_targets: Optional[List[aws_cdk.aws_events.IRuleTarget]] = None, override_rule: Optional[aws_cdk.aws_events.Rule] = None, rule_name: Optional[str] = None) aws_ddk_core.pipelines.pipeline.DataPipeline ¶
Create a rule that matches specificed event pattern with the specified target.
- Parameters
id (Optional[str]) – Identifier of the rule
event_pattern (Optional[EventPattern]) – Event pattern of the rule
event_targets (Optional[List[IRuleTarget]]) – Target of the rule - usually previous_stage.get_targets()
override_rule (Optional[Rule]) – Custom rule
rule_name (Optional[str]) – Name of Event Rule. Default - Auto-generated by CloudFormation
- Returns
pipeline – DataPipeline
- Return type
- add_stage(stage: aws_ddk_core.pipelines.stage.DataStage, skip_rule: bool = False, override_rule: Optional[aws_cdk.aws_events.Rule] = None, rule_name: Optional[str] = None) aws_ddk_core.pipelines.pipeline.DataPipeline ¶
Add a stage to the data pipeline.
By default, creates an Event Rule that matches output event pattern of the previous stage with targets of the next one.
- Parameters
stage (DataStage) – Stage instance
skip_rule (bool) – Skip creation of the default rule
override_rule (Optional[Rule]) – Override the default rule by the rule specified in this parameter
rule_name (Optional[str]) – Name of Event Rule. Default - Auto-generated by CloudFormation
- Returns
pipeline – DataPipeline
- Return type