KNFSD Metrics Agent

This agent collects custom metrics about the operation of the NFS proxy. The agent can also support collecting NFS metrics from client instances, including enriching the metrics with the name of the NFS proxy instance the client is connected to.

While the KNFSD Metrics Agent's primary use is on the proxy, the agent is also designed to support running on clients to collect useful metrics such as the total execution and round trip time of NFS requests.

See Client Metrics for a guide on installing the KNFSD Metrics Agent on a client.

Plugins

The agent uses the OpenTelemetry Collector and can support exporting metrics in several formats including Amazon CloudWatch, Prometheus, and Elasticsearch.

Receivers

Connections

Reports on the number of incoming client connections to the NFS server. A connection is considered to be a client connection if the local TCP/UDP port is 2049.

See connections/metadata.yaml

• collection_interval (default = 1m): This receiver collects metrics on an interval. Valid time units are s, m, h.

receivers:
  connections:
    collection_interval: 1m

Mounts

Reports on NFS mount statistics such as round trip time (RTT) between the local NFS mounts and the remote NFS server.

See mounts/metadata.yaml.

• collection_interval (default = 1m): This receiver collects metrics on an interval. Valid time units are ms, s, m, h.

• query_proxy_instances:

• enabled (default = false): Enables querying each source NFS server to resolve which proxy instance a client is connected to. This assumes the NFS server is running the knfsd-agent.

• timeout (default = 10s): HTTP timeout per source server, this timeout is the full round trip time, so includes establishing the connection, and reading the response. Valid time units are ms, s, m, h.

• exclude:

  • servers: List of servers to be excluded from query_proxy_instances.

  NOTE: The name or IP listed in the exclude must match the name used in the mount. For example, if the mount is logs.example.com:/logs you must specify the exclude as logs.example.com.

  • local_paths: List of local paths to be excluded from query_proxy_instances.

  If a client mounts multiple paths from the same NFS server, if any of the paths match this exclude list then the NFS server will be excluded.

  It is advised if a client has multiple paths mounted from the same NFS server, as many paths should be included in the excludes as possible. This avoids issues if one or more of the paths are not mounted (due to autofs or errors) while scraping the metrics.

receivers:
  mounts:
    collection_interval: 1m
    query_proxy_instance:
      enabled: false
      timeout: 10s
      exclude:
        servers:
          - 10.0.0.2
          - logs.example.com
        local_paths:
          - /files/logs
          - /files/home

NFSD

Reports Kernel NFS server thread statistics and thread pool metrics from /proc/fs/nfsd/pool_stats.

See nfsd/metadata.yaml

• collection_interval (default = 1m): Collection interval. Valid time units are s, m, h.
• pool_stats_path (default = /proc/fs/nfsd/pool_stats): Path to pool_stats file.

receivers:
  nfsd:
    collection_interval: 30s

FS-Cache

Reports FS-Cache and Netfslib statistics from /proc/fs/fscache/stats.

The metrics are split into two categories:

• Netfslib metrics (netfs.*): Low-level network filesystem operations including reads, writes, downloads, uploads, cache operations, and retries.
• FS-Cache metrics (fscache.*): Cache-level statistics including cookie management, LRU operations, space management, and I/O counts.

See fscache/metadata.yaml for the complete list of metrics.

• collection_interval (default = 30s): Collection interval. Valid time units are s, m, h.
• stats_path (default = /proc/fs/fscache/stats): Path to the fscache stats file.

receivers:
  fscache:
    collection_interval: 30s

FS-Cache Fragmentation

Reports extent fragmentation of the FS-Cache backing files.

cachefiles writes to its backing files using O_DIRECT, which bypasses XFS delayed allocation, so each cache write allocates in isolation and a backing file accumulates roughly one extent per write. SEEK_HOLE in the cachefiles read path is a linear scan of the extent list, so a heavily fragmented cache becomes CPU bound. See known-issues for the full description, and the CACHEFILESD_EXTSIZE Terraform variable for the remedy.

Collecting these metrics requires walking the cache and reading each file's extent map with the FIEMAP ioctl, so it is scraped far less often than the /proc based receivers. Use the fscache.fragmentation.scrape_duration metric to tune the interval, keeping the duration well below the collection_interval.

The extent map also identifies unwritten extents, which is how fscache.extents.unwritten_bytes measures the capacity the XFS extent size hint has reserved ahead of use. XFS pads each allocation up to the hint and marks the surplus unwritten, so the figure is measured rather than estimated. It is consumed as backing files fill.

See fragmentation/metadata.yaml

• collection_interval (default = 10m): This receiver collects metrics on an interval. Valid time units are s, m, h.

• cache_path (default = /var/cache/fscache/cache): The path to the cachefilesd cache directory.

• min_file_size (default = 1048576): Files smaller than this are ignored. Small files cannot accumulate enough extents to matter, and they dominate the file count in most caches.

receivers:
  fragmentation:
    collection_interval: 10m
    cache_path: /var/cache/fscache/cache
    min_file_size: 1048576

Exports

Reports on NFS export statistics such as total number of operations, read and write bytes.

See exports/metadata.yaml.

• collection_interval (default = 1m): This receiver collects metrics on an interval. Valid time units are s, m, h.

receivers:
  exports:
    collection_interval: 1m

Oldest File

Reports on the age of the oldest file in FS-Cache.

This is included to aid with diagnosing issues. The only way to find the oldest file is by recursively scanning all files under /var/cache/fscache. For this reason this metric is NOT included in the pipeline by default.

If you do add this to the pipeline, you may need to increase the collection_interval to reduce excessive load on the cache file system. On larger caches it might require increasing the interval to 1h.

• collection_interval (default = 10m): This receiver collects metrics on an interval. Valid time units are s, m, h.

• cache_path (default = /var/cache/fscache/cache): The path to the cachefilesd cache directory.

receivers:
  oldestfile:
    collection_interval: 10m
    cache_path: /var/cache/fscache/cache

Slab Info

Reports NFS cache related slab metrics (i.e. NFS inode cache, dcache).

See slab/metadata.yaml

• collection_interval (default = 1m): This receiver collects metrics on an interval. Valid time units are s, m, h.

receivers:
  slabinfo:
    collection_interval: 1m

Processors

• Metrics Transform
• Resource Detection
• Resource
• Transform

Exporters

• AWS CloudWatch Metrics
• Debug
• OTLP
• OTLP HTTP
• Elastic Search
• File
• InfluxDB
• Prometheus
• Prometheus Remote Write

Extensions

• zPages

Configuring

The agent is configured using one or more YAML config files.

knfsd-metrics-agent --config common.yaml --config proxy.yaml

The config files are loaded in the order specified, with values from the later config files overwriting the earlier config files. Object keys are merged, while arrays are replaced.

To configure a receiver, processor, or exporter it must first be defined in the appropriate section, then added to a pipeline.

If you want to include the same receiver, processor, or exporter with different configurations then you can use the format type/instance, e.g. connections/debug

receivers:
  # Declare the receivers with default options.
  # Note the colon as these are objects.
  connections:
  exports:
  fscache:
  mounts:
  nfsd:
  oldestfile:
  slabinfo:
  fragmentation:

  # Declare a second instance of the connections receiver, with a different
  # interval.
  connections/debug:
    collection_interval: 10s

# https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/processor/resourcedetectionprocessor/internal/aws/ec2/documentation.md
processors:
  resourcedetection:
    detectors: [system, ec2]
    system:
      resource_attributes:
        host.name:
          enabled: true
    ec2:
      resource_attributes:
        host.name:
          enabled: false
        cloud.availability_zone:
          enabled: true

# https://pkg.go.dev/github.com/open-telemetry/opentelemetry-collector-contrib/exporter/awsemfexporter#readme-exporter-configuration
exporters:
  awsemf:
    log_group_name: "/knfsd/metrics"
    log_retention: 30
    namespace: "/knfsd/metrics"
    # minimize the number of metrics published to CloudWatch (save cost)
    dimension_rollup_option: "NoDimensionRollup"
    resource_to_telemetry_conversion:
      enabled: true

  # Useful when developing or debugging metrics. When running the agent from the
  # command line this will write the metrics to the terminal.
  debug:
    verbosity: basic

service:
  pipelines:
    metrics: # name of the pipeline
      receivers:
        - connections
        - exports
        - fscache
        - mounts
        - nfsd
        - slabinfo
        - fragmentation
      processors:
        - resourcedetection
      exporters:
        - awsemf

    debug: # second pipeline
      receivers:
        - connections/debug
      processors:
        # can use receivers/processors/exporters in multiple pipelines
        - resourcedetection
      exporters:
        - debug

Examples

Enabling/Disabling a metric

If you're not using a particular metric, you can disable the metric to reduce the volume of data being collected.

To enable or disable a metric, set enabled: true or enabled: false for the metric. Most of the metrics are enabled by default.

If you do not want to use any of the metrics collected by a receiver, you should disable the receiver completely instead of disabling the metrics within the receiver.

Because the metrics are object keys, these will be merged with the existing values, so you do not need to specify the entire config for the receiver, only the config for the metrics you're changing.

receivers:
  mounts:
    metrics:
      nfs.mount.rpc_backlog:
        enabled: false

See the common.yaml config for a list of the receivers and metrics.

Enabling/Disabling a receiver

To disable a receiver, remove the receiver from the pipeline. You do not need to disable the metrics in a receiver. Any unused receivers will be automatically disabled.

Because the pipeline uses an array of receivers to add or remove a receiver from the pipeline you have to specify the complete list of receivers.

To check the existing list, see the proxy.yaml or client.yaml files.

For example, to remove the slabinfo receiver from the proxy:

service:
  pipelines:
    metrics:
      receivers:
        - otlp
        - connections
        - exports
        - fscache
        - mounts
        - nfsd
        # - slabinfo removed
        - fragmentation

Likewise, to enable the oldestfile collector (which is disabled by default):

service:
  pipelines:
    metrics:
      receivers:
        - otlp
        - connections
        - exports
        - fscache
        - mounts
        - nfsd
        - slabinfo
        - fragmentation
        - oldestfile # added

Change collection interval

Decreasing the collection interval will increase the resolution of metrics but will also increase the volume of the data. On platforms such as AWS this increased data volume can incur charges.

Similarly, the collection interval can be increased, this will reduce the resolution of the metrics but also reduce the volume of data.

receivers:
  mounts:
    collection_interval: 5m

It is possible to change the collection interval for specific metrics, such as if you want to collect the read/write bytes every minute, but the rest of the mount metrics every ten minutes. See multiple-intervals.yaml.