Track jobs by a custom attribute (OpenPBS / PBS Pro)

Engineering Development Hub automatically ingests HPC job data into OpenSearch (see Monitor your cluster and job activity). You can extend this with your own custom job attribute, for example the application a job runs, a cost center, or a project campaign, and it will flow into the metrics database automatically. Once indexed, the attribute becomes a first-class field you can filter, search, and aggregate, including for cost attribution. No changes to EDH code are required.

This guide uses edh_application (the application a job runs, for example lsdyna or ansys-fluent) as the worked example.

Scheduler scope

This guide covers the OpenPBS / PBS Pro scheduler, which EDH uses by default. The resourcedef, qmgr, and qsub -l steps are PBS-specific. Slurm and LSF support custom job attributes through different mechanisms; the analytics concept is the same but those steps are not covered here.

Prerequisites

Administrator (root) access to the HPC controller node, to edit resourcedef and restart the scheduler.
Analytics enabled on the cluster (the OpenSearch ingestion described in Monitor your cluster and job activity).
The ability to submit HPC jobs, or to edit a submission profile.

How it works¶

Every completed HPC job produces a scheduler accounting record. The job_tracking analytics script reads these records and indexes each job as a document in the edh_jobs_$EDH_CLUSTER_ID_$SCHEDULER_ID OpenSearch index. Any scheduler resource you define and set on a job is captured on that job's document automatically, so adding a new analytics dimension is a two-part change on the HPC controller: define the resource, then restart the scheduler so it is registered.

Step 1 - Define the resource¶

Custom PBS resources are declared in the scheduler's resourcedef file on the HPC controller. Locate it from your PBS configuration:

grep PBS_HOME /etc/pbs.conf
# resourcedef lives at: <PBS_HOME>/server_priv/resourcedef

Add one line for your attribute. Use type=string for free-form text values:

edh_application type=string

Make it permanent across cluster rebuilds

Editing resourcedef applies the change to the running controller. To ensure the resource is recreated on every deployment, add the same line to your scheduler bootstrap template configure_server.sh.j2 (in the block that writes resourcedef). This is the authoritative, rebuild-safe location for custom resource definitions.

Step 2 - Restart the HPC scheduler¶

A full scheduler restart is required

PBS resource definitions are read by the scheduler at startup. After editing resourcedef, always perform a full stop/start (restart) of the scheduler service so the new resource is registered cleanly.

# The default scheduler identifier is "openpbs-default"
# If your cluster uses a custom scheduler identifier
# substitute it into the service name first.
sudo systemctl restart edh_scheduler_openpbs-default.service

Confirm the service is active and the resource is registered. qmgr is not on the default PATH, so source /etc/pbs.conf to resolve PBS_EXEC (this works regardless of where the scheduler is installed) and call the binary by full path:

source /etc/pbs.conf
"$PBS_EXEC/bin/qmgr" -c "list resource edh_application"

Expected output:

Resource edh_application
    type = string

Running jobs are not interrupted

Restarting the scheduler does not terminate running jobs. Compute nodes reconnect to the scheduler automatically once it is back up.

Step 3 - Set the attribute on jobs¶

Submit as a regular cluster user (a permitted, non-root HPC user), typically from a login node where the PBS client is on PATH. Set the attribute at submission time with -l:

qsub -l edh_application=lsdyna -l instance_types=c8i.large -- /path/to/job.sh

Or set it automatically in a submission profile so your users do not have to specify it by hand.

Submit as a cluster user, not root

Run qsub as a regular user that has queue access, not as root. OpenPBS rejects root job submissions by default, and EDH runs and accounts each job under the submitting user, so the user and cost-attribution fields in analytics only populate correctly for a real cluster user.

Value naming

Use a single-token slug made of letters, numbers, dot, dash, or underscore (for example lsdyna, ansys-fluent, openfoam-2312). Avoid spaces and special characters. If you want a friendly display name, keep the stored value a slug and map it to a label in your dashboards.

Step 4 - See it in the metrics database¶

After a job completes, its metadata is indexed into edh_jobs_$EDH_CLUSTER_ID_$SCHEDULER_ID. The job_tracking script runs hourly and can also be run on demand:

/opt/edh/$EDH_CLUSTER_ID/cluster_manager/analytics/wrapper.sh job_tracking

Keep the value a consistent type

The edh_jobs_* index uses schemaless dynamic mapping: the first value indexed for a new field locks that field's type for the life of the index. Always send edh_application as a text slug (per the value naming guidance above) so later jobs are not rejected for a type mismatch.

Once indexed, your attribute is a normal field on the job document. Below is a real indexed job, showing edh_application captured automatically next to the built-in fields and the derived cost estimates:

{
  "job_id": "17",
  "jobname": "lsdyna_frontal_crash_v3",
  "user": "jsmith",
  "queue": "normal",
  "base_os": "amazonlinux2023",
  "instance_types": "c8i.large",
  "edh_application": "lsdyna",
  "Exit_status": 0,
  "start_iso": "2026-08-24T14:46:43",
  "end_iso": "2026-08-24T14:47:44",
  "simulation_time_hours": 0.0169,
  "estimated_price_ondemand": 0.008969,
  "soca_cluster_id": "my-cluster"
}

Analyze it in OpenSearch Dashboards¶

Open OpenSearch Dashboards and select the edh_jobs_* index pattern in Discover. Your edh_application field is now available to filter, search, and visualize.

From there you can:

Filter and search jobs by application.
Build a visualization grouped by edh_application.
Attribute spend by summing estimated_price_ondemand grouped by edh_application to see cost per application.

Summary¶

Step	Action
1	Add `edh_application type=string` to `resourcedef` (and to `configure_server.sh.j2` for rebuild safety)
2	`sudo systemctl restart edh_scheduler_openpbs-default.service`
3	Submit jobs with `-l edh_application=<slug>` (or via a submission profile)
4	Filter, search, and cost-attribute by `edh_application` in OpenSearch Dashboards

One line in resourcedef plus a scheduler restart gives you a new analytics dimension across all jobs, with automatic cost attribution and zero code changes.