Garnet Framework Integration

The Garnet Framework is an open-source solution for building digital twin and knowledge graph applications on AWS using the NGSI-LD standard. VAMS integrates with Garnet Framework to automatically synchronize all data changes into an external NGSI-LD knowledge graph, enabling advanced data federation, querying, and analytics capabilities across your visual asset data.

---

How VAMS Integrates with Garnet

VAMS provides a one-way data synchronization from VAMS to the Garnet Framework. When enabled, every change to a database, asset, asset link, or file in VAMS is automatically converted to an NGSI-LD entity and sent to the Garnet Framework ingestion queue.

The integration uses an event-driven architecture that mirrors the existing Amazon OpenSearch indexing pattern. VAMS DynamoDB Streams and Amazon S3 event notifications feed into Amazon SNS topics, which fan out to dedicated Amazon SQS queues for the Garnet indexer Lambda functions. Each indexer Lambda converts the change event into an NGSI-LD entity and sends it to the external Garnet Framework ingestion Amazon SQS queue.

info

Garnet Framework integration is entirely optional. Enabling it does not affect the core VAMS functionality or any other indexing systems such as Amazon OpenSearch. The Garnet indexers operate as additional consumers of the same Amazon SNS notification topics.

---

Configuration

To enable Garnet Framework integration, set app.addons.useGarnetFramework.enabled to true in infra/config/config.json and provide the ingestion queue URL of your Garnet Framework deployment.

{
    "app": {
        "addons": {
            "useGarnetFramework": {
                "enabled": true,
                "garnetApiEndpoint": "https://XXX.execute-api.us-east-1.amazonaws.com",
                "garnetApiToken": "not-used",
                "garnetIngestionQueueSqsUrl": "https://sqs.us-east-1.amazonaws.com/123456789012/garnet-ingestion-queue"
            }
        }
    }
}

Field	Required	Description
enabled	Yes	Set to true to deploy the Garnet Framework indexer Lambda functions and Amazon SQS queues.
garnetApiEndpoint	Yes	The Garnet Framework API endpoint URL. Must be a valid URL. Validated but not used: the indexers do not call the Garnet Framework API.
garnetApiToken	Yes	Validated but not used: no VAMS resource receives the value. A placeholder such as not-used satisfies validation; do not store a Garnet credential in config.json.
garnetIngestionQueueSqsUrl	Yes	Amazon SQS queue URL for the Garnet Framework ingestion queue. Format: https://sqs.REGION.amazonaws.com/ACCOUNT/QUEUE_NAME.

warning

When Garnet Framework is enabled, all three fields (garnetApiEndpoint, garnetApiToken, garnetIngestionQueueSqsUrl) are required. Deployment will fail with a configuration validation error if any field is missing or empty. Only garnetIngestionQueueSqsUrl is used: each indexer Lambda function sends its NGSI-LD entities to that queue and makes no call to the Garnet Framework API.

For the complete configuration reference, see the Configuration Reference.

---

Architecture

The following diagram illustrates the data flow from VAMS to the Garnet Framework.

Each indexer Lambda function:

1. Receives an Amazon SQS message containing an Amazon SNS notification wrapping a DynamoDB stream record (or Amazon S3 event notification).
2. Reads the full entity data from the relevant DynamoDB tables.
3. Converts the VAMS entity to an NGSI-LD formatted entity.
4. Sends the NGSI-LD entity to the external Garnet Framework ingestion Amazon SQS queue.

---

Data Mapping

VAMS creates four NGSI-LD entity types in the Garnet Framework knowledge graph. Each VAMS entity is converted to an NGSI-LD entity with properties, relationships, and geo-properties based on the VAMS metadata value types.

Entity Types and URN Patterns

NGSI-LD Entity Type	URN Pattern	VAMS Source
VAMSDatabase	urn:vams:database:{databaseId}	Database records
VAMSAsset	urn:vams:asset:{databaseId}:{assetId}	Asset records
VAMSAssetLink	urn:vams:assetlink:{assetLinkId}	Asset link records
VAMSFile	urn:vams:file:{databaseId}:{assetId}:{encodedFilePath}	File records

VAMSDatabase Entity

The database indexer converts VAMS database records to VAMSDatabase NGSI-LD entities.

Deleting a database moves its VAMS record to an archived partition. The database keeps its VAMSDatabase entity and entity ID, and the indexer sets isArchived to true on it.

Mapped properties:

NGSI-LD Property	Type	VAMS Source
databaseId	Property	Database ID
description	Property	Database description
defaultBucketId	Property	Default Amazon S3 bucket ID
bucketName	Property	Amazon S3 bucket name (from bucket lookup)
baseAssetsPrefix	Property	Amazon S3 base prefix for assets
restrictMetadataOutsideSchemas	Property	Schema restriction flag
restrictFileUploadsToExtensions	Property	File extension restriction list
assetCount	Property	Number of assets in the database
createdBy	Property	Creator user ID
dateCreated	Property	Creation timestamp (DateTime)
dateModified	Property	Last modification timestamp (DateTime)
isArchived	Property	Whether the database is archived
location	GeoProperty	Geographic location (if set)
metadata_*	Varies	Custom metadata (see metadata mapping)

Relationships:

NGSI-LD Relationship	Target Entity URN
usesBucket	urn:vams:bucket:{bucketId}

Example NGSI-LD entity:

{
    "id": "urn:vams:database:my-database",
    "type": "VAMSDatabase",
    "scope": ["/Database/my-database"],
    "databaseId": {
        "type": "Property",
        "value": "my-database"
    },
    "description": {
        "type": "Property",
        "value": "Production asset database"
    },
    "assetCount": {
        "type": "Property",
        "value": 42
    },
    "dateCreated": {
        "type": "Property",
        "value": {
            "@type": "DateTime",
            "@value": "2024-01-15T10:30:00Z"
        }
    },
    "isArchived": {
        "type": "Property",
        "value": false
    },
    "usesBucket": {
        "type": "Relationship",
        "object": "urn:vams:bucket:bucket-abc123"
    }
}

VAMSAsset Entity

The asset indexer converts VAMS asset records to VAMSAsset NGSI-LD entities. When an asset changes, the indexer also re-indexes all associated asset links to keep relationship data consistent.

Archiving an asset moves its VAMS record to an archived partition. The asset keeps its VAMSAsset entity and entity ID: scope, databaseId, and belongsToDatabase name the database the asset belongs to, and the entity carries the asset's metadata, current version, and relationship properties as before. isArchived is true while the asset is archived and false after it is unarchived, and the belongsToAsset relationship of each of the asset's VAMSFile entities keeps pointing at the same entity.

Mapped properties:

NGSI-LD Property	Type	VAMS Source
assetName	Property	Asset display name
assetType	Property	Asset type classification
description	Property	Asset description
databaseId	Property	Parent database ID
bucketId	Property	Amazon S3 bucket ID
bucketName	Property	Amazon S3 bucket name (from bucket lookup)
baseAssetsPrefix	Property	Amazon S3 base prefix for the bucket
isDistributable	Property	Distribution flag
tags	Property	Tag list
createdBy	Property	Creator user ID
dateCreated	Property	Creation timestamp (DateTime)
dateModified	Property	Last modification timestamp (DateTime)
isArchived	Property	Whether the asset is archived
hasChildren	Property	Has child assets (parentChild links)
hasParents	Property	Has parent assets (parentChild links)
hasRelated	Property	Has related assets (related links)
currentVersionId	Property	Current version ID
versionCreatedAt	Property	Current version creation date (DateTime)
versionComment	Property	Current version comment
metadata_*	Varies	Custom metadata (see metadata mapping)

Relationships:

NGSI-LD Relationship	Target Entity URN
belongsToDatabase	urn:vams:database:{databaseId}
usesBucket	urn:vams:bucket:{bucketId}

VAMSAssetLink Entity

The asset indexer also handles asset link records, converting them to VAMSAssetLink NGSI-LD entities that represent relationships between assets.

Mapped properties:

NGSI-LD Property	Type	VAMS Source
assetLinkId	Property	Asset link identifier
relationshipType	Property	Link type (parentChild or related)
assetLinkAliasId	Property	Optional alias for the link
tags	Property	Tag list
fromDatabaseId	Property	Source asset database ID
fromAssetId	Property	Source asset ID
toDatabaseId	Property	Target asset database ID
toAssetId	Property	Target asset ID
dateCreated	Property	Creation timestamp (DateTime)
metadata_*	Varies	Custom metadata (see metadata mapping)

Relationships:

NGSI-LD Relationship	Target Entity URN
fromAsset	urn:vams:asset:{fromDatabaseId}:{fromAssetId}
toAsset	urn:vams:asset:{toDatabaseId}:{toAssetId}

VAMSFile Entity

The file indexer converts VAMS file records to VAMSFile NGSI-LD entities, enriched with Amazon S3 object information.

Mapped properties:

NGSI-LD Property	Type	VAMS Source
filePath	Property	Relative file path within the asset
fileExtension	Property	File extension (lowercase)
fileSize	Property	File size in bytes (from Amazon S3)
lastModified	Property	Last modified timestamp (DateTime)
contentType	Property	MIME content type (from Amazon S3)
etag	Property	Amazon S3 ETag
s3VersionId	Property	Amazon S3 version ID
s3Key	Property	Full Amazon S3 object key
bucketName	Property	Amazon S3 bucket name
assetName	Property	Parent asset name (for context)
isArchived	Property	Whether the file is archived
metadata_*	Varies	Custom metadata (see metadata mapping)
attribute_*	Property	File attributes (always string type)

Relationships:

NGSI-LD Relationship	Target Entity URN
belongsToAsset	urn:vams:asset:{databaseId}:{assetId}

The file indexer reads the state of the Amazon S3 key that an event names rather than trusting the event. A live key is described by its current version. A key whose current version is a delete marker (an archived file, including each file of an archived asset) is described by its newest remaining version, with isArchived set to true. A key with no remaining version (a permanently deleted file) leaves its entity unchanged; see Limitations. An Amazon S3 HeadObject error other than not-found fails the record so that it is redriven.

note

The file indexer automatically skips folder markers, preview files (.previewFile.*), and files in excluded directories such as pipeline/, preview/, temp-upload/, and workspace/.

Custom Metadata Mapping

All four entity types support custom metadata. VAMS metadata fields are mapped to NGSI-LD properties with a metadata_ prefix to avoid naming conflicts with core properties. The NGSI-LD property type is determined by the VAMS metadataValueType:

VAMS metadataValueType	NGSI-LD Property Type	Notes
geopoint, geojson	GeoProperty	Value is parsed as GeoJSON
json	JsonProperty	Value is parsed as JSON object
xyz, wxyz, matrix4x4, lla	JsonProperty	Spatial transform data stored as JSON
All other types (string, number, boolean, date)	Property	Standard NGSI-LD property

For example, a VAMS metadata field named location with type geopoint becomes:

{
    "metadata_location": {
        "type": "GeoProperty",
        "value": {
            "type": "Point",
            "coordinates": [-73.935242, 40.73061]
        }
    }
}

---

Reindexing Existing Data

When you enable Garnet Framework integration on an existing VAMS deployment, only new and updated data is automatically synchronized. Existing data that was created before Garnet was enabled is not retroactively indexed.

To index existing assets and files, use the Reindex utility in the deployment migration scripts. It re-publishes asset and file records through the same Amazon SNS topics that feed the Garnet indexers, without clearing any existing indexes, so garnetDataIndexAsset sends a VAMSAsset entity for each asset and garnetDataIndexFile a VAMSFile entity for each file.

The reindex utility does not publish databases or asset links. An existing database reaches Garnet as a VAMSDatabase entity the next time its database record or database metadata changes. An existing asset link reaches Garnet as a VAMSAssetLink entity the next time the link or its metadata changes, or the next time the asset record of either linked asset changes. An asset metadata or file change, including the one a reindex writes, does not re-send an asset's links.

tip

The reindex process shares the same notification infrastructure used by Amazon OpenSearch indexing. Running a reindex will update both OpenSearch and Garnet indexes simultaneously. You do not need to clear OpenSearch indexes to trigger the reindex.

---

DynamoDB Stream Sources

Each Garnet indexer Lambda function processes events from specific DynamoDB tables. The indexer identifies the source table by matching the table name against the eventSourceARN in the stream record.

Indexer Lambda	DynamoDB Tables Monitored
garnetDataIndexDatabase	Database storage table, database metadata storage table
garnetDataIndexAsset	Asset storage table, asset file metadata storage table (asset-level only), asset links table, asset links metadata table
garnetDataIndexFile	Asset file metadata storage table (file-level only), file attribute storage table; also Amazon S3 event notifications

---

Failure Handling

Each Garnet indexer Amazon SQS queue has its own dead-letter queue, and each event source reports partial batch failures. A record whose indexing fails is returned to the queue for redrive rather than deleted, so the entity is retried instead of being dropped. After three delivery attempts the record moves to that indexer's dead-letter queue, which retains it for 14 days for inspection.

A dead-letter queue per indexer keeps one indexer's poison records distinguishable from another's. This matches the failure handling of the core VAMS indexers.

---

Limitations

• One-way synchronization only. Data flows from VAMS to the Garnet Framework. Changes made directly in the Garnet Framework knowledge graph are not reflected back into VAMS.
• External Garnet deployment required. VAMS does not deploy the Garnet Framework itself. You must have an existing Garnet Framework deployment and provide its ingestion queue URL. The garnetApiEndpoint and garnetApiToken fields must also be set, although the add-on does not use them (see Configuration).
• Metadata key prefixing. Custom metadata fields are prefixed with metadata_ in the NGSI-LD entities. Custom file attributes are prefixed with attribute_. This prevents conflicts with core NGSI-LD properties but means queries in Garnet must use the prefixed names.
• Deletions are not propagated. The Garnet ingestion queue upserts entities with options=update, which neither removes an entity nor drops an attribute. A permanently deleted database, asset, asset link, or file, and a removed metadata key, remain in the knowledge graph. Remove them through the Garnet Framework API (DELETE /ngsi-ld/v1/entities/{entityId}, or POST /ngsi-ld/v1/entityOperations/delete for a batch).

---

Related Resources

• Garnet Framework documentation
• NGSI-LD specification
• Configuration Reference -- Garnet Framework
• Partner Integrations