Create your own queue

Things to know before you start

By default, Scale-Out Computing on AWS creates 4 queues: high, normal (default), low and alwayson.

Queue with automatic instance provisioning¶

Create the queue¶

Via the web UI¶

As an admin, click "Queue Management" section on the left sidebar. Select a queue name then choose "Automatic Provisioning"

Via command-line¶

On your scheduler host, run qmgr as root and enter the following commands:

# Create queue name. Note: can't start with numerical character and it's recommended to use lowercase only
Qmgr:create queue <queue_name>

# Set the queue to execution
Qmgr:set queue <queue_name> queue_type = Execution

# Set default compute node - See below for more information
Qmgr:set queue <queue_name> default_chunk.compute_node = tbd

# Enable / Start the queue
Qmgr:set queue <queue_name> enabled = True
Qmgr:set queue <queue_name> started = True

# Exit
Qmgr:exit

What is compute_node=tbd

On Scale-Out Computing on AWS, unless you configure queue with AlwaysOn instances, nodes will be provisioned based on queue status. When you submit a job, Scale-Out Computing on AWS will automatically provision capacity for this job and compute_node is the scheduler making sure only one job can run on this instance. compute_node=tbd is the default value, making sure any new jobs won't run on existing (if any) nodes

Configure automatic host provisioning¶

If you want to enable automatic host provisioning, edit this file: /apps/soca/$SOCA_CONFIGURATION/cluster_manager/settings/queue_mapping.yml

Option1: I want to use the same settings as an existing queue¶

In this case, simply update the array with your new queue

queue_type:
  compute:
    queues: ["queue1", "queue2", "queue3"] # <- Add your queue to the array
    instance_ami: "ami-1234567"
    instance_type: "c5.large"
    ...

Option2: I want to configure specific settings¶

In this case, you will first need to create a new section on the YAML file (see example with memory)

queue_type:
  compute:
    queues: ["queue1"]
    instance_ami: "ami-1234567"
    instance_type: "c5.large"
    scratch_size: "100"
  memory: # <- Add new section
    queues: ["queue2"]
    instance_ami: "ami-9876543"
    instance_type: "r5.24xlarge"
    scratch_size: "600"

Finally, add a new crontab on the scheduler machine (as root). Use -c to path to the YAML file and -t to the YAML section you just created

*/3 * * * * source /etc/environment;  /apps/soca/$SOCA_CONFIGURATION/python/latest/bin/python3 /apps/soca/$SOCA_CONFIGURATION/cluster_manager/dispatcher.py -c /apps/soca/$SOCA_CONFIGURATION/cluster_manager/settings/queue_mapping.yml -t memory

Automatic Host provisioning logs¶

All logs queues are stored under /apps/soca/$SOCA_CONFIGURATION/cluster_manager/logs/<queue_name>

Queue with AlwaysOn instances¶

Important

Scale-Out Computing on AWS automatically created one AlwaysOn queue for you called "alwayson" during the first installation
In this mode, instances will never be stopped programmatically. You are responsible to terminate the capacity manually by deleting the associated CloudFormation stack

Create the queue¶

Via the web UI¶

As an admin, click "Queue Management" section on the left sidebar. Select a queue name then choose "Always On"

Via command-line¶

On your scheduler host, run qmgr as root and enter the following commands:

 # Create queue name. Note: can't start with numerical character and it's recommended to use lowercase only
Qmgr:create queue <queue_name>

# Set the queue to execution
Qmgr:set queue <queue_name> queue_type = Execution

# Enable / Start the queue
Qmgr:set queue <queue_name> enabled = True
Qmgr:set queue <queue_name> started = True

# Exit
Qmgr:exit

Start provisioning some capacity¶

Run python3 apps/soca/cluster_manager/add_nodes.py and enable --keep_forever True flag

# Launch 1 c5.large always on
python3 /apps/soca/$SOCA_CONFIGURATION/cluster_manager/add_nodes.py --instance_type c5.large \
    --desired_capacity 1 \
    --queue <queue_name> \
    --job_name instancealwayson \
    --job_owner mcrozes \
    --keep_forever True

 IMPORTANT:
 You specified --keep-forever flag. This instance will be running 24/7 until you MANUALLY terminate the Cloudformation Stack

If you need help with this script, run python3 add_nodes.py -h

Delete AlwaysOn capacity¶

Simply go to your CloudFormation console, locate the stack following the naming convention: soca-cluster-name-keepforever-queue_name-uniqueid and terminate it.

Delete a queue¶

Via the web ui, go to "Queue Management" then navigate to the "Delete Queue" tab