Common Issues
This page covers frequently encountered issues during deployment, web application usage, API interaction, and pipeline execution for Visual Asset Management System (VAMS).
CDK Deployment Errors
Amazon ECS VPC Endpoint Conflicts
When redeploying with pipeline configuration changes, you may encounter AWS CloudFormation errors related to Amazon ECS VPC interface endpoints. This occurs when Amazon ECS endpoint changes conflict with AWS CloudFormation stack change restrictions.
Symptoms:
- CloudFormation stack rollback with VPC endpoint creation failures
- Errors referencing duplicate interface endpoints for Amazon ECS
Resolution:
- Temporarily disable the affected pipelines (Isaac Lab Training, Gaussian Splat Toolbox) in
infra/config/config.json. - Deploy with pipelines disabled:
npx cdk deploy --all --require-approval never(frominfra/). - Re-enable the pipelines in the configuration file.
- Deploy again:
npx cdk deploy --all --require-approval never.
This issue typically occurs only when toggling multiple pipelines simultaneously. Deploying pipeline changes incrementally can help avoid it.
Subnet or VPC Resource Deletion Failures
When you disable a feature that required private or public subnets (for example, turning off a pipeline, the ALB public subnet, or reducing the Amazon OpenSearch Service availability zone count), AWS CloudFormation may fail to remove the now-unused subnets and their associated networking resources.
Symptoms:
DELETE_FAILEDon subnet, route table, NAT gateway, or Elastic IP resources- Errors stating a subnet has a dependency or that a network interface is still in use
UPDATE_ROLLBACK_FAILEDon the VPC nested stack after a configuration downgrade
Cause:
A subnet cannot be deleted while any elastic network interface (ENI) remains attached to it. VAMS attaches ENIs to subnets from several sources: interface VPC endpoints, NAT gateways, and—when useGlobalVpc.useForAllLambdas is enabled—every Lambda function placed in the VPC. When a feature is disabled, AWS CloudFormation attempts to delete the orphaned subnets before all dependent ENIs have detached, and the deletion fails. Changing the availability zone count or the set of subnet types also re-partitions the VPC CIDR, which compounds the issue because a subnet's CIDR cannot be changed in place.
Resolution:
Perform a staged "drain, redeploy, re-enable" procedure so the dependent ENIs are removed before the subnets are deleted:
-
In
infra/config/config.json, disable everything that requires VPC networking so the dependent ENIs are torn down. This includes:- Amazon OpenSearch Service provisioned (
openSearch.useProvisioned.enabled = false) - All pipelines that use AWS Batch, Amazon ECS, or Amazon EKS (Splat Toolbox, Isaac Lab Training, NVIDIA Cosmos, NVIDIA Gr00t, ModelOps, RapidPipeline, point cloud / 3D thumbnail / GenAI labeling previews)
useGlobalVpc.useForAllLambdas = falseso Lambda functions are removed from the VPC. Their network interfaces stay in the VPC afterwards; step 2 removes them- The ALB public subnet if it is being removed (
useAlb.usePublicSubnet = false), or the ALB itself if the web tier is changing
- Amazon OpenSearch Service provisioned (
-
Remove any Lambda functions and their VPC ENIs. Deploy with networking disabled so AWS CloudFormation detaches the Lambda functions from the VPC and deletes their ENIs:
cd infra && npx cdk deploy --all --require-approval neverThe update that takes a function out of the VPC removes the function's permission to delete its network interfaces before AWS Lambda deletes them, and Lambda does not try again. The interfaces stay in the VPC in the
availablestate, and the subnets and the old Lambda security groups cannot be deleted while they exist. List them:aws ec2 describe-network-interfaces \--filters Name=vpc-id,Values=<VAMS_VPC_ID> Name=status,Values=available "Name=description,Values=AWS Lambda VPC ENI-*" \--query 'NetworkInterfaces[].[NetworkInterfaceId,Description,RequesterId,Attachment.AttachmentId]' --output textLambda changes such an interface's type from
lambdatointerfacewhen it stops managing it, so do not filter on the interface type. Delete an interface only when its status isavailableand it has no attachment, itsRequesterIdis the AWS Lambda service rather than your own account ID, and the function named afterAWS Lambda VPC ENI-in its description no longer exists or has no subnets in its VPC configuration (aws lambda get-function-configuration --function-name <FUNCTION_NAME> --query 'VpcConfig.SubnetIds'returns an empty list, or the function is not found):aws ec2 delete-network-interface --network-interface-id <ENI_ID>An old Lambda security group that no stack owns any more stays behind too. Before deleting one by hand, check that no stack owns it (
aws cloudformation describe-stack-resources --physical-resource-id <SECURITY_GROUP_ID>finds no stack), that no network interface uses it and that no rule in another group references it. Never delete a security group that a stack still owns, or one that an AWS Batch compute environment of a GPU pipeline uses. -
Re-enable the features you need in
infra/config/config.json, including any new subnet or availability-zone configuration (for example, setopenSearch.useProvisioned.availabilityZoneCountto2for a two-AZ Region such aseusc-de-east-1). -
Deploy again so the VPC is provisioned with the new subnet layout:
cd infra && npx cdk deploy --all --require-approval never -
If Amazon OpenSearch Service was disabled and re-enabled, re-index the search data. The domain is created empty, so run the re-index tool (or set
openSearch.reindexOnCdkDeploy = truefor one deployment). See Reindexing OpenSearch.
Disabling Amazon OpenSearch Service and re-enabling it does not lose asset data. Search data is rebuilt from the authoritative Amazon DynamoDB tables and Amazon S3 buckets by the re-index process; only the search index itself is recreated.
To minimize the chance of these failures, keep the availability zone count and subnet types stable across deployments. VAMS provisions a fixed AZ count per subnet type so that toggling individual pipelines does not add or remove subnets. The largest source of subnet churn is changing openSearch.useProvisioned.availabilityZoneCount or switching useAlb.usePublicSubnet, both of which change the subnet set.
Docker Buildx Container Image Errors
The 3D Basic Conversion and CAD/Mesh Metadata Extraction pipelines deploy as AWS Lambda container images, which AWS CDK builds with Docker and pushes to Amazon ECR during npx cdk deploy. Pushing one of these images or creating its function can fail with failed commit on ref "manifest-sha256:..." or Lambda function XXX reached terminal FAILED state due to InvalidImage.
Symptoms:
unexpected status from PUT request to https://....dkr.ecr.REGION.amazonaws.com/v2/foo/manifests/bar: 400 Bad RequestInvalidImage(ImageLayerFailure: UnsupportedImageLayerDetected)orThe image manifest, config or layer media type for the source image ... is not supportedwhile AWS CloudFormation creates the pipeline's Lambda function- The conversion pipeline nested stack fails and the core stack rolls back
Cause:
AWS Lambda accepts a container image only as a single-platform image manifest. When Docker stores images in the containerd image store, the default in Docker Desktop 4.34 and later and in new installations of Docker Engine 29 and later, a build adds a provenance attestation by default and the image is pushed as an OCI image index, which Lambda rejects. The AWS CDK CLI version pinned in infra/package.json turns that default off for every image it builds: it runs the build with BUILDX_NO_DEFAULT_ATTESTATIONS=1 regardless of the value in your shell. A deployment run with npx cdk from the infra/ directory therefore pushes a single manifest.
When the error appears anyway, the image index is already cached. AWS CDK tags each image with a hash of its source, and it skips the build and the push when an image with that tag exists, first in the bootstrap container assets repository in Amazon ECR and then as the local Docker image cdkasset-<hash>. An image index pushed by an earlier attempt, for example one built by another AWS CDK CLI version or outside the AWS CDK CLI, is reused by every later deployment until its tag is deleted.
Resolution:
-
Run the deployment from the
infra/directory withnpx cdkso that the pinned AWS CDK CLI builds the images.npx cdk --versionshows the version in use. -
List the asset hashes of the two conversion images. In
cdk.out/*.assets.json, written by the failed deployment or bynpx cdk synth, each image underdockerImagesis keyed by its hash, which this command (run from theinfra/directory) prints on the line before the image'sdisplayName:grep -B1 -E 'vamsExecute(3dBasic|MeshCadMetadataExtraction)Conversion/AssetImage' cdk.out/*.assets.json -
Delete those image tags from the container assets repository. With the default bootstrap qualifier, the repository is named
cdk-hnb659fds-container-assets-<account>-<region>:aws ecr batch-delete-image \--repository-name cdk-hnb659fds-container-assets-<account>-<region> \--image-ids imageTag=<hash> -
Remove the local copies, so AWS CDK builds the images again instead of pushing the cached ones:
docker image rm cdkasset-<hash> -
Deploy again with
npx cdk deploy --all --require-approval never.
Removing these two images is sufficient; clearing the whole Docker cache is not required.
Every VAMS container image is pinned to linux/amd64. Docker Desktop on macOS and Windows includes the emulation that builds these images on an ARM64 host such as a Mac with Apple silicon, so no emulation setup is required. On an ARM64 Linux host running Docker Engine, install amd64 emulation as described in the Docker multi-platform builds guide.
If you are using Finch or Podman instead of Docker, buildx-specific issues may not apply. However, you may encounter different container build errors specific to your engine. Verify the issue reproduces with Docker before reporting. See the Prerequisites page for alternative container engine setup.
Web Build or Infrastructure CDK Errors
After upgrading VAMS or switching branches, stale dependencies can cause build failures.
Symptoms:
- TypeScript compilation errors in
web/orinfra/ - Module resolution failures during
cdk synth
Resolution:
# Clear and reinstall web dependencies
cd web && rm -rf node_modules && npm install && npm run build
# Clear and reinstall infrastructure dependencies
cd infra && rm -rf node_modules && npm install
Always run npm install in both the web/ and infra/ directories after pulling new code or switching branches.
External VPC Import Failures
When importing an external Amazon VPC with subnets, the first deployment attempt may fail because AWS CDK cannot resolve VPC context before stack synthesis.
Symptoms:
- VPC or subnet lookup errors during
cdk synth - Stack deployment fails referencing missing VPC context
Resolution:
Perform a two-phase deployment:
# Phase 1: Import VPC context and deploy non-VPC stacks
npx cdk deploy --all --require-approval never --context loadContextIgnoreVPCStacks=true
# Phase 2: Deploy all stacks including VPC-dependent ones
npx cdk deploy --all --require-approval never
Alternatively, set loadContextIgnoreVPCStacks: true in infra/config/config.json for the first deployment, then set it back to false for subsequent deployments.
AWS KMS Key Permission Errors
AWS KMS key policy errors can occur when AWS CloudFormation custom resources attempt to modify Amazon S3 or Amazon DynamoDB tables encrypted with a customer-managed KMS key.
Symptoms:
- Custom resource Lambda functions fail with
AccessDeniedExceptionfor AWS KMS operations - Stack deployment rolls back during default data population steps
Resolution:
Verify that the KMS key policy includes the required principals. If using an external CMK via app.useKmsCmkEncryption.optionalExternalCmkArn, ensure the key policy grants the following actions to the AWS CloudFormation service principal and the deployment role:
kms:Decryptkms:Encryptkms:GenerateDataKeykms:ReEncrypt*
Amazon Cognito User Pool Schema Errors
The user pool fails to create or update on its custom:groups attribute. app.authProvider.useCognito.retainLegacyGroupsAttribute decides whether the user pool declares that attribute, and the value has to match the user pool the deployment has.
Symptoms:
- A new deployment fails creating the user pool with
Attribute names MUST conform to the following rules(InvalidRequest), and the core stack rolls back. - An update fails on the user pool with
Existing schema attributes cannot be modified or deleted(InvalidRequest), and the core stack rolls back toUPDATE_ROLLBACK_COMPLETE.
Resolution:
- For the create error, set
retainLegacyGroupsAttributetofalseininfra/config/config.json. A core stack whose first creation failed is left inROLLBACK_COMPLETEand cannot be updated, so delete it in the AWS CloudFormation console, then deploy again. Amazon Cognito rejects the attribute name when it creates a user pool. - For the update error, set
retainLegacyGroupsAttributetotrue(or remove the key, which resolves totrue) and deploy again. The existing user pool declares the attribute, and Amazon Cognito keeps a custom attribute for the life of the pool.
See Update the solution.
Amazon OpenSearch Provisioned Domain Update Failures
Updating a provisioned Amazon OpenSearch Service domain can fail when the requested domain configuration is incompatible with the existing indexes.
Symptoms:
- Stack enters
UPDATE_ROLLBACK_FAILEDreferencing the OpenSearch domain and the indexer / search / reindexer resources - Domain update rejected with
Cluster.Index.ReplicaCount: The following indices do not have multiples of 3 copies ... Index should have 3 or multiples of 3 copies including primary and replica for 3-AZ with Standby
Cause:
Amazon OpenSearch Service Multi-AZ with Standby requires every index to have copies (primary plus replicas) in a multiple of three. VAMS enables Multi-AZ with Standby only when availabilityZoneCount is 3, and in that case creates the asset and file indexes with two replicas (three copies). The error occurs when an existing domain that already holds single-copy indexes (a prior 2-AZ deployment) is switched to availabilityZoneCount: 3 in place: AWS CloudFormation validates the new Standby domain configuration against the existing indexes before their replica count can be raised, and rejects it.
Resolution:
- Recover the stuck stack: in the AWS CloudFormation console, choose Continue update rollback on the
UPDATE_ROLLBACK_FAILEDstack, skipping the OpenSearch domain and dependent indexer/search resources if prompted, so the stack returns toUPDATE_ROLLBACK_COMPLETE. - To stay on the existing domain unchanged, set
app.openSearch.useProvisioned.availabilityZoneCountback to2(the default) and redeploy:cd infra && npx cdk deploy --all --require-approval never. - To move to a 3-AZ Multi-AZ-with-Standby domain, create it fresh: deploy once with OpenSearch disabled (
useProvisioned.enabled = false) to remove the existing domain, then re-enable it withavailabilityZoneCount: 3and deploy again. Run the reindex tool (or setreindexOnCdkDeploy = truefor one deployment) to repopulate the new domain from source data — no asset data is lost.
A fresh 3-AZ domain is created as Multi-AZ with Standby with three data nodes and three copies of each index. A 2-AZ domain runs without Standby with two data nodes and a single copy. Leave availabilityZoneCount at its default of 2 to keep an existing domain unchanged.
Web Application Issues
Content Security Policy Errors in Local Development
During local development, Content Security Policy (CSP) errors may block certain viewer functionality or API calls.
Symptoms:
- Browser console errors mentioning
Content-Security-Policy - Viewers fail to load external resources
- WebAssembly modules blocked
Resolution:
VAMS includes a service worker that sets the required cross-origin isolation headers. Ensure the service worker is registered by verifying:
- The development server is running on
https://orlocalhost. - The browser has not blocked the service worker registration.
- For WASM-based viewers, verify
allowUnsafeEvalFeaturesis enabled in the deployment configuration if testing against a deployed backend.
The Vite development server proxy handles most CSP issues automatically. If problems persist, clear your browser cache and service worker registrations.
WASM-Based Viewers Not Loading
Viewers that use WebAssembly (Needle USD Viewer, Three.js CAD Viewer, Cesium 3D Tileset Viewer, ThatOpen IFC BIM Viewer, SuperSplat Editor) require specific HTTP headers to function.
Symptoms:
- Viewer shows a loading spinner indefinitely
- Browser console errors mentioning
SharedArrayBufferorCross-Origin-Opener-Policy
Resolution:
WASM-based viewers require Cross-Origin Isolation headers (Cross-Origin-Opener-Policy: same-origin and Cross-Origin-Embedder-Policy: require-corp). These are provided in two ways:
- Amazon CloudFront deployment: Headers are set automatically by the CloudFront distribution.
- Application Load Balancer (ALB) deployment: A front-end service worker attempts to set the headers. If your organization's security policy blocks service workers, WASM viewers will not function.
For the Needle USD Viewer, the SuperSplat Editor, the ThatOpen IFC BIM Viewer, and Three.js CAD formats, you must also enable allowUnsafeEvalFeatures in infra/config/config.json because their WASM loaders require runtime code generation.
Safari Limitations for WASM Viewers
Safari does not fully support the cross-origin isolation requirements needed by certain WASM-based viewers.
Symptoms:
- Needle USD Viewer, Three.js CAD formats (.stp, .step, .iges, .brep), and Cesium Viewer fail to load in Safari
- Standard mesh formats (.gltf, .glb, .obj, .stl) in the Three.js Viewer work correctly
Resolution:
Use a Chromium-based browser (Google Chrome, Microsoft Edge) or Mozilla Firefox for WASM-dependent viewers. Non-WASM viewers and standard mesh formats work in all supported browsers.
SuperSplat Editor Reports That WebGPU Is Required
The SuperSplat Editor renders through WebGPU and has no WebGL fallback.
Symptoms:
- Opening a
.ply,.sog,.splat, or.lccfile in the SuperSplat Editor shows the message "SuperSplat requires WebGPU, which this browser does not support" instead of the editor - The BabylonJS and PlayCanvas Gaussian Splat viewers render the same
.plyand.sogfiles
Resolution:
Use a browser with WebGPU enabled: current Google Chrome or Microsoft Edge, Safari 26 or later, or Mozilla Firefox where WebGPU is enabled. Where a browser policy disables WebGPU or the machine has no supported GPU driver, open .ply files with the BabylonJS or PlayCanvas Gaussian Splat viewer and .sog files with the PlayCanvas Gaussian Splat viewer instead; .lcc and .splat files have no alternative viewer.
Login Loop or Configuration Fetch Failures
Users may experience a login loop where the application repeatedly redirects to the sign-in page.
Symptoms:
- Page refreshes continuously after successful authentication
- Browser console shows errors fetching
/api/amplify-configor/api/secure-config
Resolution:
- Clear your browser cache and local storage for the VAMS domain.
- Verify the API Gateway endpoint is accessible from your network.
- If using IP range restrictions (
authorizerOptions.allowedIpRanges), confirm your IP address is within an allowed range. - For external OAuth identity provider configurations, verify all endpoint URLs in the configuration are correct and reachable.
API Issues
429 Rate Limiting
VAMS applies API rate limiting through Amazon API Gateway throttling.
Symptoms:
- API responses return HTTP 429 (Too Many Requests)
- Bulk operations fail intermittently
Resolution:
Increase the rate limits in infra/config/config.json:
{
"app": {
"api": {
"globalRateLimit": 100,
"globalBurstLimit": 200
}
}
}
The default values are globalRateLimit: 50 requests per second and globalBurstLimit: 100. Redeploy after changing these values.
Increasing rate limits raises the potential cost of Amazon API Gateway usage and may affect downstream service limits. Monitor your Amazon CloudWatch metrics after adjustments.
Timeout on Large Operations
Amazon API Gateway returns a timeout once the configured integration timeout elapses (app.api.apiGatewayRest.apiGatewayTimeoutTime, 29 seconds by default), while the underlying AWS Lambda function continues processing for up to 15 minutes.
Symptoms:
- API returns a 504 Gateway Timeout
- The operation actually completes successfully in the background
Affected operations:
- Listing or exporting assets with thousands of files
- Amazon OpenSearch re-indexing for large datasets
- Bulk metadata operations
Resolution:
For operations that may exceed the integration timeout, check your AWS Lambda function logs in Amazon CloudWatch to confirm whether the operation completed. The VamsCLI provides automatic pagination and retry logic that handles timeout scenarios for bulk operations.
Deployments that regularly run these operations can raise app.api.apiGatewayRest.apiGatewayTimeoutTime (up to 300 seconds) so the request completes synchronously instead of returning a 504. Values above 29 seconds require an approved account-level Integration timeout quota increase (L-E5AE38E3) in the deployment Region before deploying — see the configuration reference.
Asset ID Refused as Not Unique Although No Asset Exists
Symptoms:
- Creating an asset returns HTTP 400 "Asset identifier is not unique for the given S3 bucket location"
- No asset with that ID exists:
vamscli assets get -d <DATABASE_ID> <ASSET_ID> --show-archivedreports that the asset is not found
Cause:
Asset creation first writes an empty folder object named after the asset ID into the database's default Amazon S3 bucket, then records the asset. When a later step fails, for example creating the asset's notification topic, the folder object stays behind. Creation refuses an asset ID whose folder holds any object version or delete marker, so that ID cannot be created again. #1385 tracks removing the leftover object.
Resolution:
Remove every version of the empty folder object, then create the asset again.
-
Find the bucket and its base assets prefix:
vamscli database get -d <DATABASE_ID>shows the database's default bucket ID, andvamscli database list-bucketsshows that bucket's name and base assets prefix. -
Build the folder object's key: the base assets prefix with a trailing slash and no leading slash (a prefix of
/is the bucket root and adds nothing), then the asset ID, then/. In a bucket whose base assets prefix is/, the key for assetmy.assetismy.asset/. -
List every version and delete marker under that key:
aws s3api list-object-versions --bucket <BUCKET_NAME> --prefix "<KEY>" \--query "{Versions: Versions[].[Key, VersionId, Size], DeleteMarkers: DeleteMarkers[].[Key, VersionId]}"Continue only if every entry has exactly that key and every version has size 0. Anything else under the key is asset data. If another database uses the same bucket and base assets prefix, also confirm with
vamscli assets get -d <OTHER_DATABASE_ID> <ASSET_ID> --show-archivedthat it holds no asset with this ID. -
Delete each listed version and delete marker by its version ID:
aws s3api delete-object --bucket <BUCKET_NAME> --key "<KEY>" --version-id <VERSION_ID>A delete without
--version-idadds a delete marker, which still blocks the asset ID. -
Create the asset again.
Amazon OpenSearch Indexing Delays After Bulk Operations
After uploading many files or performing bulk metadata changes, search results may not immediately reflect the updates.
Symptoms:
- Newly uploaded assets do not appear in search results
- Metadata changes are not reflected in search filters
Resolution:
Amazon OpenSearch indexing is asynchronous. After bulk operations, allow 30-60 seconds for indexing to complete. If indexing appears stuck:
- Check the Amazon CloudWatch logs for the indexing Lambda functions.
- Verify the Amazon OpenSearch cluster health in the AWS Management Console.
- If necessary, trigger a re-index by setting
reindexOnCdkDeploy: truein the configuration and redeploying, or by using the manual re-index tool ininfra/deploymentDataMigration/. - If a file is in the asset's file list but never appears in search, or a deleted asset or file is still listed, check the indexing dead-letter queues (next section).
Indexing Dead-Letter Queue Holds Messages
Symptoms:
-
One of these Amazon CloudWatch alarms is in the
ALARMstate:Alarm Source queue <configuration name>-<app.baseStackName>-<region>-bucketSyncCreatedDLQ--<n><configuration name>-<app.baseStackName>-<region>-bucketSyncCreated--<n><configuration name>-<app.baseStackName>-<region>-bucketSyncDeletedDLQ--<n><configuration name>-<app.baseStackName>-<region>-bucketSyncDeleted--<n><configuration name>-<app.baseStackName>-<region>-fileIndexerDLQ<configuration name>-<app.baseStackName>-<region>-fileIndexer<configuration name>-<app.baseStackName>-<region>-assetIndexerDLQ<configuration name>-<app.baseStackName>-<region>-assetIndexer -
Files appear in the asset's file list but not in search results, have no version history entry, or did not fire a file-upload workflow trigger
-
Deleted assets or files still appear in search results
Cause:
A bucket-sync or indexer Lambda could not process those events within three receives, so Amazon SQS moved them to the dead-letter queue of its source queue. The objects in Amazon S3 are unaffected. For the bucket-sync created queue in the commercial partition and the AWS European Sovereign Cloud, AWS Lambda recursive loop detection is a cause when the bucket-sync Lambda's recursive loop setting is not Allow; its RecursiveInvocationsDropped metric is above zero when that is the cause. For the indexer queues, an interruption of the Amazon OpenSearch Service domain or collection is the usual cause.
Resolution:
Fix the cause shown in the logs of the Lambda that reads the source queue, if any, then move the messages back to the source queue listed in the table above. Each dead-letter queue keeps messages for 14 days.
-
Find the source queue and read its dead-letter queue ARN from the redrive policy:
aws sqs get-queue-url --queue-name <source queue name>aws sqs get-queue-attributes --queue-url <source queue URL> --attribute-names RedrivePolicy QueueArn -
Start the move from the dead-letter queue back to the source queue:
aws sqs start-message-move-task --source-arn <dead-letter queue ARN> --destination-arn <source queue ARN> -
Track it until the status is
COMPLETED:aws sqs list-message-move-tasks --source-arn <dead-letter queue ARN>
The credentials you use need sqs:StartMessageMoveTask, sqs:ListMessageMoveTasks, sqs:ReceiveMessage, sqs:DeleteMessage and sqs:GetQueueAttributes on the dead-letter queue, sqs:SendMessage on the source queue and, with app.useKmsCmkEncryption.enabled, kms:Decrypt and kms:GenerateDataKey on the VAMS key. Each replayed event is processed as the original event would have been. A bucket-sync created event indexes the file, writes its version history row, and fires matching file-upload workflow triggers; a bucket-sync deleted event removes the file's metadata and search entry; an indexer event updates the search index. The alarm returns to OK once the queue is empty. The Reindex utility restores the search entries only.
Pipeline Issues
Container Pull Failures
Pipeline containers running on AWS Batch with AWS Fargate may fail to pull container images from Amazon Elastic Container Registry (Amazon ECR).
Symptoms:
- AWS Batch job fails with
CannotPullContainerError - Timeout errors during image pull
Resolution:
Pipeline containers require network access to Amazon ECR endpoints. Verify:
- The VPC has the required VPC endpoints for Amazon ECR (
com.amazonaws.region.ecr.apiandcom.amazonaws.region.ecr.dkr) and Amazon S3. - If using pipelines that require internet access (such as RapidPipeline or ModelOps), ensure the VPC has NAT Gateway or public subnet access configured.
- Check that the security groups attached to AWS Batch compute environments allow outbound HTTPS traffic.
GPU Instance Unavailability for AWS Batch Jobs
Some pipelines (Isaac Lab Training, Gaussian Splat Toolbox) require GPU instances that may not be available in all AWS Regions or Availability Zones.
Symptoms:
- AWS Batch jobs remain in
RUNNABLEstate indefinitely - No compute environment instances are launched
Resolution:
- Verify GPU instance type availability in your AWS Region (e.g.,
g6e.2xlarge,g5.xlarge). - Request a service quota increase for the required instance types through the AWS Service Quotas console.
- For Isaac Lab Training, consider enabling the
keepWarmInstanceoption to reduce cold start times at the cost of continuous compute charges.
Deadline Cloud Jobs Never Start and Workers Are Replaced Repeatedly
A DeadlineCloud pipeline execution hangs, and the Deadline Cloud fleet keeps launching and discarding workers. The usual cause is the fleet role missing Amazon CloudWatch Logs write permission — the AWS managed AWSDeadlineCloud-FleetWorker policy does not include it.
Symptoms:
- The VAMS execution stays in a running state until it times out, with no error reported by VAMS
- The farm, fleet, and queue all report
ACTIVE, and the Deadline Cloud job was created successfully - The job's task never leaves its pending state; jobs that are cancelled show
CANCELEDhaving never run - The fleet's worker-not-responding alarm fires, and the fleet replaces its instance every few minutes
- The worker agent log records
AccessDeniedExceptiononUpdateWorkerwithdoes not have sufficient access to perform CreateLogStream
Resolution:
Add the two log actions to the fleet role, alongside the worker permissions it already holds:
{
"Effect": "Allow",
"Action": ["logs:CreateLogStream", "logs:PutLogEvents"],
"Resource": [
"arn:aws:logs:*:<account-id>:log-group:/aws/deadline/*",
"arn:aws:logs:*:<account-id>:log-group:/aws/deadline/*:log-stream:*"
]
}
Submit a job again and confirm its task reaches SUCCEEDED. A worker on a scale-to-zero fleet is created, runs, and is released quickly, so it may not be observable in ListWorkers between polls — the job's terminal status is the reliable signal, not catching the worker in a particular state.
Because the alarm names worker responsiveness rather than an IAM permission, this reads as a capacity or instance-availability problem. Check the fleet role's log permissions before requesting quota increases or changing instance types.
See Configuration reference for the full list of operator-supplied Deadline Cloud prerequisites.
Pipeline Timeout vs. Workflow Timeout
Pipeline step timeouts and overall workflow timeouts are configured separately. A pipeline step that exceeds its timeout will cause the workflow to fail.
Symptoms:
- Workflow execution fails but logs show the container was still processing
- AWS Step Functions execution history shows a timeout error on a specific state
Resolution:
Adjust the timeout values in the pipeline or workflow configuration. Large files (approaching the 100 GB limit for the 3D Thumbnail pipeline) may require extended processing time. Monitor the AWS Batch job logs and AWS Step Functions execution history to determine appropriate timeout values.
For long-running pipeline operations, check the AWS Batch job logs in Amazon CloudWatch rather than relying solely on the API response or web UI status.