Orbit plugins are individual python modules published PyPi. We are continually adding plugin capabilites to the platform. This is the current list of published plugins:
Each Plugin has a configuration that you need to define. The structure is as follows (in yaml format):
- PluginId: <the name of the plugin>
Module: <the module name>
Path: <relative path of plugin src code>
Parameters:
- parameter 1: <value>
- parameter 2: <value>
...
- parameter N: <value>
PluginId - a unique idetnifier for the plugin
Module - the name of the module (get this from the examples below for your plugin)
Path - the relative path of the src code for the plugin (relative to the cli/ directory)
Parameters - configurable plugin-specific parameters
The plugin configurations are located in a yaml configuration file that is referenced in the manifest. For example, the lake-creator plugin declares the plugin requirement for a team defined in lake-creator team. In this example, the individual plugins are defined in lake-creator plugin. Adding or remvoing plugin definitions effectively adds or removes the plugin deployment from your environment when deployed. The lake-creator plugin references these plugin definitions in the teams section.
This plugin configures redshift cluster creation, accessiblity and termination capabilities using Orbit SDK helper methods. Before deleting teamspaces, recommended approach is to do gracefull terminate of all teamspace specific redshift clusters. While destroying teamspace, left over redshift clusters are terminated forcibly by design.
- PluginId:
Module:
Path: ../plugins/redshift/
Parameters:
enable_user_activity_logging: "true"
require_ssl: "true"
use_fips_ssl: "true"
node_type: "DC2.large"
number_of_nodes: "2"
References:
This plugin enables Orbit users to have access to an AWS Code Commit repository.
- PluginId: code_commit
Module: code_commit
Path: ../plugins/code_commit/
None
References
This plugin enables the execution of a custom AWS Cloudformation template.
- PluginId:
Module:
Path: ../plugins/custom_cfn/
Parameters:
- CfnTemplatePath: "./bundle/plugins/demo-lake-user-cfn-template.yaml"
References:
This plugin enables the orbit workspace to interact and submit spark jobs to EKS. Please see here for samples.
- PluginId: enable_emr_on_eks
Module: emr_on_eks
Path: ../plugins/emr_on_eks/
None
References:
This plugin enables Amazon FSx for Lustre file system availabilty to the Orbit platform.
- PluginId: fast_fs_lustre
Module: lustre
Path: ../plugins/lustre/
Parameters:
- storage: 1200Gi
-folder: /fsx/data1
-storage define the site of the file system
-folder define a mount path for the file system
References:
- PluginId: ray
Module: ray
Path: ../plugins/ray/
Parameters:
workers: 3
release_tag: 1.3.0
-workers the number of workers in the ray cluster
-release_tag - the version of Ray used
This plugin enables the kubernetes cluster to execute jobs on Sagemaker from the Orbit platform. Please see here for samples.
- PluginId: sm-operator
Module: sm-operator
Path: ../plugins/sm-operator/
None
References:
- PluginId:
Module:
Path: ../plugins/team_script_launcher/
Parameters:
script: |
i=0
while [ $i -lt 1 ];
do
find /efs/shared/regression -mtime +5 -type f -exec rm {} \;
sleep 60
done
echo "DONE LOOP"
restartPolicy: Never
-script - Required. The bash script to run after the team has been created. A Pod will be created to run this script. The pod can run a single time script , or even a script that run forever as a Daemon as shown in the example.
-restartPolicy- Optional. Default is ‘Never’. The restart policy of the pod.
-image - Optional. The image address for the pod. Default is the base notebook image.
-uid - Optional. The user id for the pod security context.
-gid - Optional. The group id for the pod security context.
This plugin preserve idle capacity on the EKS cluster to allow quick startup time of containers even when the cluster has reached its capacity and need to scale up. Administrator can define how much capacity in terms of cpu and memory to preserve for the next containers. When cluster has utilized all space other than this reserve space, it will initiate scale up, but the containers that fits this preserve space can still start immediately.
- PluginId:
Module:
Path: ../plugins/overprovisioning/
Parameters:
replicas: 3
cpu: 2
memory: 4Gi