VirtualiZarr Parsers¶

VirtualiZarr parser for generating virtual Zarr datasets from imagery files.

OversightMLParser implements the VirtualiZarr Parser protocol and produces ManifestStore objects that can be serialized to Kerchunk JSON indices. It works for any format supported by IO.open(): NITF, standalone JPEG 2000, TIFF, and GeoTIFF.

The parser conforms to the VirtualiZarr parser callable protocol (url, registry) -> ManifestStore. It reads bytes by opening url with fsspec and handing the seekable handle to IO.open(), which issues on-demand byte-range reads for the tiled formats (NITF, TIFF/GeoTIFF, JPEG 2000) — so a local path and an s3:// URL follow the same code path and neither downloads the whole file to build the index.

Single file — pass one URL. If the file contains overview assets (e.g. COG overview IFDs), the parser builds a hierarchical store automatically.
Multi-file pyramid — pass the base URL; sibling .r1/.r2/… R-set companions are discovered on the same filesystem and mapped to overview levels. The parser builds a hierarchical store with GeoZarr multiscales metadata describing the pyramid structure.

Note

virtualizarr is an optional dependency. Install with pip install osml-imagery-io[virtualizarr] to enable parser support.

OversightMLParser¶

class aws.osml.io.virtualizarr_parsers.OversightMLParser¶

Bases: object

VirtualiZarr parser for any imagery format supported by IO.open().

Supports NITF (2.0, 2.1, NSIF 1.0, SICD, SIDD), standalone JPEG 2000 (.j2k, .jp2), TIFF, and GeoTIFF. Format detection is derived from the URL extension — the parser itself is format-agnostic.

Conforms to the VirtualiZarr parser protocol: an instance is a callable (url: str, registry) -> ManifestStore. Bytes are read by opening url with fsspec and handing the seekable handle to IO.open(), which issues on-demand byte-range reads for the block-capable formats (TIFF, JPEG 2000, NITF, DTED) — so a local path and an s3:// URL follow the same code path and neither downloads the whole file to build the index.

Multi-file R-set pyramids are reconstructed automatically: given a base url, sibling <base>.r1 / .r2 / … files are discovered on the same filesystem and mapped to overview levels.

Examples

Index a local file (chunk refs point at the same local path):

parser = OversightMLParser()
manifest_store = parser("/data/image.ntf")

Index a remote file directly (range reads, no full download):

parser = OversightMLParser()
manifest_store = parser("s3://bucket/image.ntf")

Multi-file pyramid (image.ntf + image.ntf.r1 auto-discovered):

parser = OversightMLParser()
manifest_store = parser("s3://bucket/image.ntf")

See write_tile_index() for portable ({{base}}-template) and URL-rewritten output — relocating chunk references is a serialization-time concern, not a parse-time one.

Constructor¶

OversightMLParser() takes no parse-time configuration — the URL passed when the parser is called is the single source of truth for both reading and chunk references.

parser = OversightMLParser()

Calling the parser¶

parser(url, registry=None) reads and indexes the imagery at url. Local paths, file:// URIs, and s3:// URIs all work (opened via fsspec). Chunk references in the returned store point at url. R-set overview companions (<url>.r1, <url>.r2, …) are discovered automatically.

# Local file (chunk refs point at the local path)
store = parser("/data/image.ntf")

# Remote file — range reads, no full download
store = parser("s3://bucket/image.ntf")

# Multi-file pyramid — image.ntf.r1 etc. auto-discovered from the base URL
store = parser("s3://bucket/image.ntf")

To relocate chunk references (portable {{base}} indexes, or index a local copy but reference an s3:// location), use write_tile_index’s template_base / url_overrides arguments — see below.

Flat vs hierarchical output¶

When the parser detects overview assets (keys matching image:N:overview:M), it produces a hierarchical ManifestStore with one subgroup per resolution level. Otherwise it produces a flat store with arrays at the root — identical to the pre-multiscale behavior.

For hierarchical stores, each subgroup contains a single array named "data", and the root group’s attributes include GeoZarr multiscales metadata and a zarr_conventions array declaring convention identity:

ManifestGroup (root)
├── groups:
│   ├── "0" → ManifestGroup(arrays={"data": level_0_array})
│   ├── "1" → ManifestGroup(arrays={"data": level_1_array})
│   └── "2" → ManifestGroup(arrays={"data": level_2_array})
└── attributes:
    ├── "source": "s3://bucket/image.ntf"
    ├── "zarr_conventions": [{ ... }]
    └── "multiscales": { ... }

`multiscales` metadata structure¶

The root group’s multiscales attribute conforms to the GeoZarr multiscales convention (UUID d35379db-88df-4056-af3a-620245f8e347). It contains:

layout — one entry per resolution level with an asset path matching the subgroup name, an optional derived_from referencing the parent level, and a transform object with relative scale and translation arrays
resampling_method — optional; recorded when a downsampling_method keyword argument is provided to the parser

Scale transforms use relative factors between adjacent levels (not absolute from level 0). The scale and translation arrays have two elements: [Y, X].

A zarr_conventions array in the root attributes declares convention identity:

{
  "source": "s3://bucket/image.tif",
  "zarr_conventions": [
    {
      "uuid": "d35379db-88df-4056-af3a-620245f8e347",
      "schema_url": "https://raw.githubusercontent.com/zarr-conventions/multiscales/refs/tags/v1/schema.json",
      "spec_url": "https://github.com/zarr-conventions/multiscales/blob/v1/README.md",
      "name": "multiscales",
      "description": "Multiscale layout of zarr datasets"
    }
  ],
  "multiscales": {
    "layout": [
      {
        "asset": "0",
        "transform": {"scale": [1.0, 1.0], "translation": [0.0, 0.0]}
      },
      {
        "asset": "1",
        "derived_from": "0",
        "transform": {"scale": [2.0, 2.0], "translation": [0.0, 0.0]}
      }
    ],
    "resampling_method": "average"
  }
}

write_tile_index¶

aws.osml.io.virtualizarr_parsers.write_tile_index(store, output, segments=None, *, template_base=None, url_overrides=None, zarr_format=2)¶

Write a tile index to JSON or Parquet with multi-range support.

This is the recommended way to serialize a ManifestStore produced by OversightMLParser. It handles the multi-range reference entries that VirtualiZarr’s built-in serialization does not support.

zarr_format selects which Zarr version the index describes. Both are Kerchunk reference files served through fsspec — the difference is the store keys inside, and therefore which consumer path reads them:

2 (default) — a .zgroup / .zarray / .zattrs layout. Codecs resolve through the numcodecs registry by id and are called synchronously with a single buffer.
3 — a native zarr.json layout with c.<band>.<row>.<col> chunk keys. Codecs resolve by URI through the zarr.codecs entry points and are called by zarr’s asynchronous batched codec pipeline.

By default chunk references point at the URL the store was parsed from. Relocating those references is a serialization-time concern controlled here:

Portable / relocatable index — pass template_base="{{base}}" to rewrite every chunk-reference URL to {{base}}<filename> and emit a Kerchunk v1 "templates": {"base": ""} dict. At read time the base is supplied via template_overrides={"base": "s3://bucket/path/"} to MultiReferenceFileSystem / ReferenceFileSystem.
Explicit URL rewrite — pass url_overrides={old_url: new_url} to remap concrete URLs (e.g. index by reading a local copy, then point the references at the s3:// location the data will be served from).

template_base and url_overrides are mutually exclusive.

Parameters:

store (ManifestStore) – The manifest store returned by OversightMLParser().
output (str) – Output file path. Extension determines format: .json for Kerchunk JSON, .parquet for a Kerchunk Parquet directory. Parquet requires zarr_format=2, cannot carry multi-range chunk references, and needs pyarrow (the osml-imagery-io[zarr] extra) — see _emit_refs(). It must be read back with MultiReferenceFileSystem, which a stock fsspec ReferenceFileSystem cannot do. Multi-resolution pyramids are supported in both formats.
segments (list[str] | None) – Subgroup keys to include (e.g. ["0", "2"]). If None, all subgroups are included.
template_base (str | None) – When given (typically "{{base}}"), produces a portable index whose chunk-reference URLs are rewritten to template_base + basename.
url_overrides (dict[str, str] | None) – Explicit {concrete_url: replacement_url} rewrite applied to chunk references and the root source attribute.
zarr_format (int) – 2 for the Kerchunk / Zarr v2 layout read through numcodecs, 3 for the native Zarr v3 layout read through the zarr.codecs entry-point pipeline.

Raises:

ValueError – If the output extension is not .json or .parquet, if a requested segment is not found, if both template_base and url_overrides are given, or if zarr_format is not 2 or 3.
ImportError – If .parquet output is requested without pyarrow installed.

Examples

Return type:: None

Absolute URL index (references point at the parsed URL):

parser = OversightMLParser()
store = parser("s3://my-bucket/imagery/image.ntf")
write_tile_index(store, "image.tile_index.json")

Native Zarr v3 index (read with no Kerchunk-to-v2 translation):

parser = OversightMLParser()
store = parser("s3://my-bucket/imagery/image.ntf")
write_tile_index(store, "image.v3.json", zarr_format=3)

Portable index (resolve base URL at read time):

parser = OversightMLParser()
store = parser("local/image.ntf")
write_tile_index(store, "image.tile_index.json", template_base="{{base}}")

Index a local copy, reference the remote location:

parser = OversightMLParser()
store = parser("local/image.ntf")
write_tile_index(
    store, "image.tile_index.json",
    url_overrides={os.path.abspath("local/image.ntf"): "s3://bucket/image.ntf"},
)

write_tile_index() automatically detects whether the store is flat or hierarchical and serializes accordingly. For hierarchical stores, the output Kerchunk JSON uses path-prefixed keys (e.g. 0/data/0.0.0, 1/data/0.0.0) and includes the root multiscales metadata in .zattrs.

Choosing a Zarr format¶

zarr_format selects which Zarr version the index describes. Both outputs are Kerchunk reference files served through fsspec — no pixel data is copied either way. What differs is the store keys inside, and therefore which consumer path reads them:

	`zarr_format=2` (default)	`zarr_format=3`
Metadata keys	`.zgroup`, `.zarray`, `.zattrs`	`zarr.json` per node
Chunk keys	`0/data/0.0.0`	`0/data/c.0.0.0`
Codec resolution	numcodecs registry, by `id`	`zarr.codecs` entry points, by URI
Codec call convention	synchronous, single buffer	async, batched pipeline
Output formats	`.json`, `.parquet` — both carry all four reference forms, including multi-range	`.json` only

parser = OversightMLParser()
store = parser("s3://bucket/image.ntf")

# Kerchunk / Zarr v2 (read through numcodecs)
write_tile_index(store, "image.tile_index.json")

# Native Zarr v3 (read through the entry-point codec pipeline)
write_tile_index(store, "image.v3.json", zarr_format=3)

Parquet output requires zarr_format=2, and requesting .parquet with zarr_format=3 raises ValueError rather than writing an index nothing can read back.

This is a deferral, not a structural limit. The Kerchunk Parquet container indexes chunk references positionally, deriving each row from the array’s chunk-grid shape and parsing the chunk key’s last segment as grid coordinates. A v3 store carries everything that needs — zarr.json states shape and chunk_grid.configuration.chunk_shape, and chunk_key_encoding names the chunk key separator — just not under the v2 names. What is missing is read-side support: deriving chunk counts from zarr.json rather than {field}/.zarray, parsing c.<band>.<row>.<col> chunk keys whose leading c is not a coordinate, and recognizing zarr.json as metadata. That is a bounded piece of work — a few more overrides alongside the ones MultiReferenceFileSystem already carries — and is simply not implemented.

Reading a Parquet index¶

Parquet indexes must be opened with MultiReferenceFileSystem. A stock fsspec ReferenceFileSystem cannot open one, and multi-resolution Parquet is fully supported:

from aws.osml.io.multi_reference_fs import MultiReferenceFileSystem
import zarr

fs = MultiReferenceFileSystem(fo="image.tile_index.parquet")
root = zarr.open_group(fs.get_mapper(""), mode="r", zarr_format=2)

full_res = root["0/data"]     # level 0
half_res = root["1/data"]     # level 1 — the pyramid survives the round-trip

Four properties of the Parquet container make this necessary, all handled by MultiReferenceFileSystem:

Multi-range chunks need extra columns. Upstream’s record schema has only scalar offset and size columns, so a chunk assembled from several non-contiguous fragments — every chunk, in RPCL JPEG 2000 imagery — has nowhere to go. This library appends range_path (the URL), plus offsets and sizes as list<int64> columns, and leaves path null on those rows. A reader that does not know the columns therefore fails on such a chunk rather than handing back one fragment of six as if it were the whole thing. MultiReferenceFileSystem decodes them into the multi-range reference form and fetches the fragments — concurrently on the async path — joining them in the stored order.
The record loader routes through pandas, and nulls decode inconsistently. fsspec loads each record with pandas.read_parquet(...) and then immediately calls to_numpy() on every column — pandas decodes via pyarrow and is thrown away, which would make pandas a runtime requirement of merely reading an index. Worse, the writer’s engine is not recorded in the store while LazyReferenceMapper defaults to fastparquet on read, and the two engines decode nulls differently — enough to misread every chunk reference. MultiReferenceFileSystem loads with pyarrow directly and normalizes nulls, so this library never imports pandas and an index written by another tool with fastparquet reads correctly too.
Nested group keys confuse the stock field listing. A hierarchical store has a .zgroup per pyramid level, which fsspec’s listdir() mistakes for an array field and then fails to find a .zarray for.
Templates cannot be stored. A Parquet store has nowhere to keep the Kerchunk "templates" dict, so a portable index (template_base="{{base}}") resolves its placeholders from template_overrides alone. The substitution is applied to range_path as well as path, so multi-range chunks of a portable index resolve too:
```
fs = MultiReferenceFileSystem(
    fo="image.tile_index.parquet",
    template_overrides={"base": "s3://bucket/path/"},
)
```

Reading and writing a Parquet index requires pyarrow, supplied by the zarr extra:

pip install "osml-imagery-io[zarr]"

Without it, both paths raise an ImportError naming the package and this extra.

Note

One cosmetic consequence of leaving path null on multi-range rows: fsspec’s ls() builds its chunk listing from rows with a non-null path, so multi-range chunks do not appear in a listing of their array field. Reads are unaffected — Zarr fetches chunks by key, and both consumer paths decode them correctly — but a fs.ls("0/data") on an index of interleaved imagery will show the metadata keys and no chunks. The JSON container does list them.

Relocating chunk references¶

Relocating chunk references is a serialization-time concern controlled by two mutually exclusive keyword arguments, and applies to both formats:

template_base — pass "{{base}}" to produce a portable index whose chunk-reference URLs are rewritten to {{base}}<filename> and emit a Kerchunk v1 "templates": {"base": ""} dict. At read time the base is supplied via template_overrides={"base": "s3://bucket/path/"}.
url_overrides — an explicit {old_url: new_url} mapping, e.g. index a local copy and point the references at the s3:// location the data will be served from.

# Portable index (resolve base URL at read time)
parser = OversightMLParser()
store = parser("local/image.ntf")
write_tile_index(store, "image.json", template_base="{{base}}")

# Index a local copy, reference the remote location
write_tile_index(
    store, "image.json",
    url_overrides={os.path.abspath("local/image.ntf"): "s3://bucket/image.ntf"},
)

VirtualiZarr Parsers¶

OversightMLParser¶

Constructor¶

Calling the parser¶

Flat vs hierarchical output¶

multiscales metadata structure¶

write_tile_index¶

Choosing a Zarr format¶

Reading a Parquet index¶

Relocating chunk references¶

`multiscales` metadata structure¶