Orbit Plugin Definiton and Configuration

Orbit plugins are individual python modules published PyPi. We are continually adding plugin capabilites to the platform. This is the current list of published plugins:

Each Plugin has a configuration that you need to define. The structure is as follows (in yaml format):

- PluginId: <the name of the plugin>
  Module: <the module name>
  Path: <relative path of plugin src code>
  Parameters:
    - parameter 1: <value>
    - parameter 2: <value>
    ...
    - parameter N: <value>

PluginId - a unique idetnifier for the plugin

Module - the name of the module (get this from the examples below for your plugin)

Path - the relative path of the src code for the plugin (relative to the cli/ directory)

Parameters - configurable plugin-specific parameters

The plugin configurations are located in a yaml configuration file that is referenced in the manifest. For example, the lake-creator plugin declares the plugin requirement for a team defined in lake-creator team. In this example, the individual plugins are defined in lake-creator plugin. Adding or remvoing plugin definitions effectively adds or removes the plugin deployment from your environment when deployed. The lake-creator plugin references these plugin definitions in the teams section.


Amazon Redshift

This plugin configures redshift cluster creation, accessiblity and termination capabilities using Orbit SDK helper methods. Before deleting teamspaces, recommended approach is to do gracefull terminate of all teamspace specific redshift clusters. While destroying teamspace, left over redshift clusters are terminated forcibly by design.

- PluginId: 
  Module: 
  Path: ../plugins/redshift/
  Parameters:
    enable_user_activity_logging: "true"
    require_ssl: "true"
    use_fips_ssl: "true"
    node_type: "DC2.large"
    number_of_nodes: "2"

Parameters

References:


AWS Code Commit

This plugin enables Orbit users to have access to an AWS Code Commit repository.

- PluginId: code_commit
  Module: code_commit
  Path: ../plugins/code_commit/ 

Parameters

None

References


AWS Cloudformation

This plugin enables the execution of a custom AWS Cloudformation template.

- PluginId: 
  Module: 
  Path: ../plugins/custom_cfn/
  Parameters:
    - CfnTemplatePath: "./bundle/plugins/demo-lake-user-cfn-template.yaml"   

Parameters

  • CfnTemplatePath - this defines where the template to be executed is located. Orbit creates a ‘bundle’ of all code to be deployed. In the example above, the path specifies a template that is in the ‘bundle’ at the same level as the manifests - demo-lake-user-cfn-template.yaml

References:


EMR on EKS

This plugin enables the orbit workspace to interact and submit spark jobs to EKS. Please see here for samples.

- PluginId: enable_emr_on_eks
  Module: emr_on_eks
  Path: ../plugins/emr_on_eks/

Parameters

None

References:


Lustre

This plugin enables Amazon FSx for Lustre file system availabilty to the Orbit platform.

- PluginId: fast_fs_lustre
  Module: lustre
  Path: ../plugins/lustre/
  Parameters:
    - storage: 1200Gi
    -folder: /fsx/data1

Parameters

-storage define the site of the file system

-folder define a mount path for the file system

References:


Ray

- PluginId: ray
  Module: ray
  Path: ../plugins/ray/
    Parameters:
        workers: 3
        release_tag: 1.3.0
  

Parameters

-workers the number of workers in the ray cluster

-release_tag - the version of Ray used


Sagemaker Operator

This plugin enables the kubernetes cluster to execute jobs on Sagemaker from the Orbit platform. Please see here for samples.

- PluginId: sm-operator
  Module: sm-operator
  Path: ../plugins/sm-operator/

Parameters

None

References:


Team Script Launcher Plugin

- PluginId:  
  Module: 
  Path: ../plugins/team_script_launcher/ 
    Parameters:
        script: |
            i=0
            while [ $i -lt 1 ];
            do
              find /efs/shared/regression -mtime +5 -type f -exec rm {} \;
              sleep 60
            done
            echo "DONE LOOP"
        restartPolicy: Never
  

Parameters

-script - Required. The bash script to run after the team has been created. A Pod will be created to run this script. The pod can run a single time script , or even a script that run forever as a Daemon as shown in the example.

-restartPolicy- Optional. Default is ‘Never’. The restart policy of the pod.

-image - Optional. The image address for the pod. Default is the base notebook image.

-uid - Optional. The user id for the pod security context.

-gid - Optional. The group id for the pod security context.


Overprovisioning Plugin

This plugin preserve idle capacity on the EKS cluster to allow quick startup time of containers even when the cluster has reached its capacity and need to scale up. Administrator can define how much capacity in terms of cpu and memory to preserve for the next containers. When cluster has utilized all space other than this reserve space, it will initiate scale up, but the containers that fits this preserve space can still start immediately.

- PluginId: 
  Module:
  Path: ../plugins/overprovisioning/ 
  Parameters:
    replicas: 3
    cpu: 2
    memory: 4Gi
  

Parameters

  • replicas - number of containers to allocate for the reserve space
  • cpu - number of cpu for each allocated containers. e.g., ‘2’
  • memory - Memory capacity for each container. e.g., ‘4Gi’