Skip to content

edhctl logs

edhctl logs helps you fetch and inspect logs from across your EDH environment without having to connect to each host or open the AWS console.

Run edhctl logs --help to display the help message. Logs are grouped into three areas:

./edhctl logs
Usage: app.py logs [OPTIONS] COMMAND [ARGS]...

Commands:
  hpc            # Bootstrap logs from HPC compute nodes
  lambda         # Logs from the EDH Lambda functions
  web-interface  # Logs from the EDH web interface

logs hpc

Inspect the bootstrap logs produced by your HPC compute nodes. This is the fastest way to understand what happened while a node for a given job was being provisioned and configured.

logs hpc list

List the log files available for a job. Pass the job ID with --job-id.

./edhctl logs hpc list --job-id 0
Job: 0
Path: /apps/edh/edh-demo/shared/logs/bootstrap/compute_node/0/5c04ddbf-a5e9-463d-ab0f-453eaa7e7178

  ip-11-0-121-62/
    01_user_data.sh.log
    02_setup.log
    bootstrap_s3_location.log
    cloud-init-output-final-1787308713.log
    filesystems_mount.log
    install_required_packages.sh.log
    messages

logs hpc fetch

Fetch the bootstrap log lines for a job. Use --log-level to keep only the entries at or above a given severity (for example WARNING), which makes it easy to spot what went wrong.

./edhctl logs hpc fetch --job-id 0 --log-level WARNING
[ip-11-0-121-62/02_setup.log] configure: WARNING: no system libmpdec found; falling back to bundled libmpdec
[ip-11-0-121-62/02_setup.log] WARNING: The scripts pip3 and pip3.13 are installed in '/opt/edh/edh-demo/python/3.13.13/x86_64/amazonlinux2023/bin' which is not on PATH.
[ip-11-0-121-62/install_required_packages.sh.log] WARNING: ...

Each line is prefixed with the host and the log file it came from, so you can tell where a message originated when a job runs across several nodes.

logs lambda

Inspect the logs of the AWS Lambda functions that EDH uses to operate your cluster.

logs lambda list

List the EDH Lambda functions for your cluster.

./edhctl logs lambda list
edh-demo-ADComputerCleaner
edh-demo-AsgCapacityBumper
edh-demo-CapacityExecutor
edh-demo-Metrics
edh-demo-SpotInterruptionCapture
edh-demo-VdiPoolReconciler
...

logs lambda failed

Report the functions that have recent failures. When everything is healthy, the failure count is 0.

./edhctl logs lambda failed
{
    "failed_lambda": 0,
    "failures": {}
}

logs lambda fetch and watch

Use fetch to retrieve log lines for a function, and watch to stream them as they arrive. Run ./edhctl logs lambda fetch --help or ./edhctl logs lambda watch --help to see the available options.

logs web-interface

Fetch the logs of the EDH web interface. This is useful when the UI is returning errors and you want to see what the server reported.

logs web-interface fetch

./edhctl logs web-interface fetch --help
Usage: app.py logs web-interface fetch [OPTIONS]

Options:
  --log-file [uwsgi|web_interface]
                                  Log file to read  [required]
  --log-level [DEBUG|INFO|WARNING|ERROR|CRITICAL]
                                  Filter logs by level (repeatable)
  --from TEXT                     Start time: ISO 8601 or epoch seconds
  --to TEXT                       End time: ISO 8601 or epoch seconds
  --limit INTEGER                 Maximum number of log lines (default: 100)
  --output [text|json]            Output format (default: text)
  --ai-assistant TEXT             Analyze errors with Amazon Bedrock

The main options are:

  • --log-file: which log file to read, uwsgi or web_interface (required).
  • --log-level: keep only entries at the given level. Repeat the flag to combine levels, for example --log-level ERROR --log-level WARNING.
  • --from / --to: restrict the output to a time window, given as an ISO 8601 timestamp (for example 2024-01-15T10:30:00) or epoch seconds.
  • --limit: the maximum number of lines to return (default 100).
  • --output: text (default) or json.

For example, show only the errors from the uwsgi log:

./edhctl logs web-interface fetch --log-file uwsgi --log-level ERROR
Error ID: AWS_API_ERROR | Error Message: Error while invoking secretsmanager API. ResourceNotFoundException - SecretId arn:aws:secretsmanager:us-east-1:<REDACTED>:secret:/edh/edh-demo/SessionSignerKey-JAQDuP does not exist | Status Code: 500 | Error RequestId ab6b1df7-57b0-4ecb-b907-f78d965e5c68 | Error Trace:
>> File: /opt/edh/edh-demo/cluster_manager/utils/errors/aws_api.py, Line: 15, Function: AWS_API_ERROR, Frame ID 140035109019968
Error ID: AWS_API_ERROR | Error Message: Error while invoking secretsmanager API. ResourceNotFoundException - SecretId arn:aws:secretsmanager:us-east-1:<REDACTED>:secret:/edh/edh-demo/SessionEncryptionKey-O6fzvK does not exist | Status Code: 500 | Error RequestId 5aeea973-319c-4f61-bcff-121e9a801288 | Error Trace:
>> File: /opt/edh/edh-demo/cluster_manager/utils/errors/aws_api.py, Line: 15, Function: AWS_API_ERROR, Frame ID 140035109020736

Analyze errors with the AI Assistant

Add --ai-assistant to have Amazon Bedrock read the matching log entries and explain what is going wrong, what likely caused it, and how to fix it. This is only active with --log-level WARNING, ERROR, or CRITICAL, so you point the analysis at real problems rather than routine output.

You can pass a model ID to --ai-assistant, or omit the value to use the default model. This is the same Amazon Bedrock integration offered in the browser through the AI Assistant in the EDH web interface.

./edhctl logs web-interface fetch --log-file uwsgi \
   --log-level ERROR \ 
   --ai-assistant

Error ID: AWS_API_ERROR | Error Message: Error while invoking secretsmanager API. ResourceNotFoundException - SecretId arn:aws:secretsmanager:us-east-1:<REDACTED>:secret:/edh/edh-demo/SessionSignerKey-JAQDuP does not exist | Status Code: 500 | Error RequestId ab6b1df7-57b0-4ecb-b907-f78d965e5c68 | Error Trace:
>> File: /opt/edh/edh-demo/cluster_manager/utils/errors/aws_api.py, Line: 15, Function: AWS_API_ERROR, Frame ID 140035109019968
Error ID: AWS_API_ERROR | Error Message: Error while invoking secretsmanager API. ResourceNotFoundException - SecretId arn:aws:secretsmanager:us-east-1:<REDACTED>:secret:/edh/edh-demo/SessionEncryptionKey-O6fzvK does not exist | Status Code: 500 | Error RequestId 5aeea973-319c-4f61-bcff-121e9a801288 | Error Trace:
>> File: /opt/edh/edh-demo/cluster_manager/utils/errors/aws_api.py, Line: 15, Function: AWS_API_ERROR, Frame ID 140035109020736


--- EDH AI Analysis ---
## Analysis

The EDH web interface is failing to retrieve two critical secrets from AWS Secrets Manager:

1. **`/edh/edh-demo/SessionSignerKey-JAQDuP`** - Used by Flask to sign session cookies (HMAC).
2. **`/edh/edh-demo/SessionEncryptionKey-O6fzvK`** - Used to encrypt session data.

Both return `ResourceNotFoundException`, meaning the secrets referenced by the application do not exist (or are no longer accessible) at those exact ARNs. Since these secrets are essential to bootstrap the Flask session layer, the web UI will return HTTP 500 on virtually every request.

## Potential Root Causes

1. **Secrets were deleted** (manually or via a stack tear-down / cleanup script). AWS Secrets Manager appends a random 6-character suffix (e.g., `-JAQDuP`, `-O6fzvK`), so once deleted, an identical name cannot be recreated with the same ARN.
2. **Cross-account / cross-region mismatch** - the EDH host is looking in `us-east-1` / account `<REDACTED>`; verify the instance is in the correct account and region.
3. **Stack drift or partial CloudFormation rollback** - the CFN stack that provisioned the SOCA/EDH environment may have been partially deleted while EC2 hosts (with cached secret ARNs in `/apps/edh/edh-demo/cluster_manager/settings/*` or env vars) remain running.
4. **IAM permission issue masked as NotFound** - less likely (usually AccessDenied), but worth confirming the instance role has `secretsmanager:GetSecretValue` on these ARNs.
5. **Cluster name mismatch** - the app is configured for cluster `edh-demo` but secrets were provisioned under a different cluster identifier.

## Recommended Next Steps

1. **Verify the secrets exist:**
   aws secretsmanager list-secrets --region us-east-1 \
     --filters Key=name,Values=/edh/edh-demo \
     --query 'SecretList[].[Name,ARN]' --output table

2. **Check the CloudFormation stack** for `edh-demo` - confirm it is in `CREATE_COMPLETE` / `UPDATE_COMPLETE` and has not been (partially) deleted. The secrets are typically created by the main SOCA/EDH stack.

3. **If the stack is intact but secrets are missing**, restore them:
   - Check for Secrets Manager recovery window: `aws secretsmanager list-secrets --include-planned-deletion`
   - `aws secretsmanager restore-secret --secret-id <arn>` if within the 7-30 day window.
4. **If unrecoverable**, you will need to:
   - Create new secrets with fresh random Fernet/HMAC keys,
   - Update the ARN references in the app config (and any Parameter Store / tags),
   - Restart uWSGI: `systemctl restart uwsgi` (or the SOCA equivalent).
   - Note: existing user session cookies will be invalidated, forcing re-login.
5. **Verify IAM** on the controller instance role includes `secretsmanager:GetSecretValue` for the new ARNs.
6. **Longer term**: if this is a demo environment being torn down/rebuilt frequently, consider avoiding stack deletions that leave orphaned EC2 hosts referencing dead secret ARNs.

Once the secrets are accessible again, the 500 errors on the web interface should clear immediately (no app code change required if ARNs are unchanged).
--- End ---

Powered by Amazon Bedrock

The --ai-assistant analysis uses Amazon Bedrock and is billed by usage. The models available depend on your cluster configuration.

Build advanced AI capabilities

The --ai-assistant analysis is powered by the SocaAiAssistant helper in EDH. If you want to build your own AI-driven tooling on top of EDH (for example custom log analysis, summaries, or automated triage), you can reuse the same helper. See its implementation in assistant.py on GitHub.

The same Amazon Bedrock integration is also available directly in the browser through the AI Assistant in the EDH web interface, where you can chat with a model, talk to an agent, or analyze your files without using the CLI.