MultiReferenceFileSystem

An fsspec filesystem that extends Kerchunk’s ReferenceFileSystem with multi-range byte fetching. This is used internally by the Zarr codec pipeline to read JPEG 2000 codestreams with interleaved tile-parts that span non-contiguous byte ranges in a single file.

Note

fsspec is an optional dependency. This class is available when pip install osml-imagery-io[zarr] is installed.

class aws.osml.io.multi_reference_fs.MultiReferenceFileSystem(*args, **kwargs)

Bases: ReferenceFileSystem

ReferenceFileSystem with multi-range chunk support.

Extends the Kerchunk reference spec to support a fourth reference form:

["url", [[offset, length], [offset, length], ...]]

for chunks whose data spans multiple non-contiguous byte ranges.

All existing reference forms (inline, whole-file, single-range) are handled by the parent class unchanged.

Kerchunk Parquet reference directories are opened through ArrayOnlyReferenceMapper rather than the parent’s default mapper, so hierarchical (multiscales) Parquet indexes are readable — including portable ones, whose {{base}} placeholders are resolved from template_overrides. JSON and dict inputs take the parent path unchanged.

ls(path, detail=True, **kwargs)

List path, falling back to the directory cache for group prefixes.

The parent short-circuits to LazyReferenceMapper.ls for any lazy mapper and never consults dircache. Because ArrayOnlyReferenceMapper reports only array fields, that lookup raises for a group prefix (and for the root) — so fall back to the dircache built by _dircache_from_items(), which does carry the group levels.

isdir(path)

Is path a directory (group prefix or array field)?

The parent asks LazyReferenceMapper.listdir(), which for ArrayOnlyReferenceMapper deliberately omits group prefixes, so consult the directory cache as well.