Pre-commit System

NOTE: You do not need to follow these instructions to create the image & deploy the knfsd-file-cache solution on AWS. These instructions are for developers who want to contribute to the project.

The KNFSD File Cache project utilizes a comprehensive pre-commit framework to ensure code quality, consistency, and security before changes are committed to the repository. This system automates the execution of multiple code quality tools and formatters across different languages and file types.

The pre-commit system integrates seamlessly with the development workflow and can be invoked through convenient make targets for both automated and manual validation.

Overview

The pre-commit system is built around industry-standard tools that validate:

• Code formatting and style across multiple languages (Go, Python, Shell, HCL, YAML, JSON)
• Spelling and grammar in documentation and comments
• Security vulnerabilities and coding best practices
• Infrastructure as Code standards for Terraform and Packer
• Git commit message conventions

Quick Start

Installing Pre-commit

To enable pre-commit hooks for all git commits locally:

# Install pre-commit hooks (one-time setup)
pre-commit install

# Install commit message hooks for conventional commits
pre-commit install --hook-type commit-msg

Running Pre-commit Manually

You can run pre-commit checks manually using make targets:

# Run all pre-commit checks (aliased as 'pc' for convenience)
make pre-commit
make precommit
make pc

Updating Pre-commit Tools

To update all pre-commit tools to their latest versions:

# Update all repository versions in .pre-commit-config.yaml
# (aliased as 'precommit-update', 'pc-update' and 'autoupdate')
make pre-commit-update
make pc-update
make autoupdate

# Update the Go dependencies of every Go project
make goget

# Clean and update pre-commit cache
pre-commit clean
pre-commit install

Make Targets Reference

Main Makefile Commands

The root Makefile provides the following targets:

Command	Purpose	Tools Used	Files Created
make all	Run complete validation suite	All linting, Packer, Terraform, BATS, security scanners, Go tools	Various cache and build artifacts
make pc	Run all pre-commit hooks	pre-commit	-
make pc-update	Update pinned hook revisions	pre-commit	-
make lint	Run all linting tools	EditorConfig, Codespell, shfmt, ShellCheck, Black, MyPy, Pylint, IAM policy size check	-
make sec	Run every security scanner	Checkov, Gosec, KICS, Semgrep, Trivy	Scanner caches
make docs	Serve the documentation locally	MkDocs	-
make clean	Delete generated temporary files	find_temp_files.sh	-

make all runs lint, packer, terraform, bats, scan, golint and gotest.

Image Build Commands

These targets are not part of the pre-commit flow, but they live in the same root Makefile and are listed here for completeness:

Command	Purpose	Files Created
make image	Build the knfsd AMI with Packer	-
make image-debug	Build the AMI with packer build -debug and Packer logging enabled	packer.log
make image-log	Build the AMI with Packer logging enabled	packer.log
make iamlive	Start iamlive in proxy mode	iamlive.json
make image-iam	Build the AMI through the iamlive proxy to capture the IAM actions	iamlive.json

NOTE: make image-iam requires make iamlive to be running in another terminal.

Code Quality and Formatting

Command	Purpose	Tool	Configuration	Reference
make ec	Validate EditorConfig formatting	editorconfig-checker	.editorconfig	EditorConfig Documentation
make codespell	Check spelling in code and docs	Codespell	.codespellrc	Codespell Documentation
make shfmt	Report shell script formatting differences	shfmt	.editorconfig	shfmt Documentation
make shellcheck-sh	Lint tracked *.sh scripts	ShellCheck	.shellcheckrc	ShellCheck Documentation
make shellcheck-bash	Lint tracked *.bash scripts	ShellCheck	.shellcheckrc	ShellCheck Documentation
make shellcheck-bats	Lint tracked *.bats test files	ShellCheck	.shellcheckrc	ShellCheck Documentation
make black	Format Python code	Black	py314 target version	Black Documentation
make mypy	Type check Python code	MyPy	Built-in configuration	MyPy Documentation
make pylint	Lint Python code	Pylint	Built-in configuration	Pylint Documentation
make iam-size	Verify IAM policy JSON size limits	Embedded Python script	6,144 character AWS limit	IAM Quotas

The three shellcheck-* targets split the scan by file extension so that each file type is checked with the correct dialect. All of them run with --severity=style.

make iam-size validates that each docs/iam/*.json and examples/*/iam.json file is valid JSON and, ignoring whitespace, fits inside the AWS customer managed policy character limit.

Infrastructure as Code

Command	Aliases	Purpose	Tool	Configuration	Reference
make packer-fmt	packer	Format Packer HCL files	Packer	Built-in formatter	Packer fmt Command
make packer-val	packer	Validate Packer templates	Packer	Built-in validator	Packer validate Command
make tf-fmt	terraform	Format Terraform files	Terraform	Built-in formatter	Terraform fmt Command
make tf-lint	terraform	Lint Terraform code	TFLint	.tflint.hcl	TFLint Documentation
make tf-val	terraform	Validate Terraform syntax	Terraform	Built-in validator	Terraform validate Command

Testing

Command	Purpose	Tool	Reference
make bats	Run the proxy startup script tests in Docker	BATS	BATS Documentation
make gotest	Run the Go unit tests of every Go project	Go testing framework	Go Testing

make bats delegates to image/resources/startup/run-tests.sh, which builds a Docker image from image/resources/startup/tests/Dockerfile and runs the *.bats suites inside it, so a working Docker daemon is required.

make gotest is an aggregate of the per-project targets below, which can also be run individually:

Command	Project
make filter-exports-gotest	image/resources/filter-exports
make knfsd-agent-gotest	image/resources/knfsd-agent
make knfsd-fsidd-gotest	image/resources/knfsd-fsidd
make knfsd-metrics-agent-gotest	image/resources/knfsd-metrics-agent
make netapp-exports-gotest	image/resources/netapp-exports

NOTE: image/smoke-tests is deliberately excluded from make gotest, because its tests deploy real AWS infrastructure. Run those from the project directory, see Image Smoke Tests.

Security Scanning

Command	Aliases	Purpose	Tool	Configuration	Reference
make scan-checkov	checkov, scan	Infrastructure security scan	Checkov	.checkov.yaml	Checkov Documentation
make scan-gosec	gosec	Go source code security scan	Gosec	Built-in rules	Gosec Documentation
make scan-kics	kics, scan	Infrastructure security analysis	KICS	.kics.yaml	KICS Documentation
make scan-semgrep	semgrep, scan	Static code analysis	Semgrep	.semgrepignore	Semgrep Documentation
make scan-trivy	trivy, scan	Comprehensive vulnerability scan	Trivy	.trivyignore.yaml, .trivy.tfvars	Trivy Documentation

make scan runs the Checkov, KICS, Semgrep and Trivy scanners. Use make security (alias sec) to additionally include scan-gosec. make scan-kics runs through run-kics.sh and the Trivy targets write their cache to $TRIVY_CACHE_DIR ($HOME/.cache/trivy), which the dev container backs with the knfsd-dev-trivy-cache Docker volume so the vulnerability database and checks bundle survive make clean and container rebuilds.

License Scanning

Command	Purpose	Tool	Files Created	Reference
make lic-scan	Scan dependencies for licenses	Trivy	THIRD-PARTY-LICENSES	Trivy License Scanning
make lic-scan-ignore	License scan with ignore rules	Trivy	THIRD-PARTY-LICENSES	Trivy License Scanning

Go-specific Commands

The following commands operate across all Go projects in the repository:

Command	Aliases	Purpose	Reference
make golint	-	Lint all Go projects	golangci-lint Documentation
make gosec	scan-gosec	Go source code security scan	Gosec
make gotidy	-	Tidy Go modules	Go Modules
make goget	goupdate	Update Go dependencies	Go Modules
make gotest	-	Run the Go unit tests	Go Testing

golint, gosec, gotidy and goget cover all six Go projects, including image/smoke-tests. gotest covers the first five projects only.

Go Project Makefiles

Each Go project has its own Makefile with standardized targets:

Project Locations

• image/resources/filter-exports/Makefile
• image/resources/knfsd-agent/Makefile
• image/resources/knfsd-fsidd/Makefile
• image/resources/knfsd-metrics-agent/Makefile
• image/resources/netapp-exports/Makefile
• image/smoke-tests/Makefile (see Image Smoke Tests)

Standard Go Targets

Target	Purpose	Tools Used
golint	Lint Go code	golangci-lint with .golangci.yaml
gotidy	Clean up Go module dependencies	go mod tidy
gosec	Scan Go code for security issues	gosec ./...
goget / goupdate	Update Go dependencies	go get -t -u ./... followed by go mod tidy
gotest	Run Go tests with code coverage	go test -cover -vet=all -v ./...

Each project Makefile accepts a ROOT_DIR variable so that golangci-lint can locate the shared .golangci.yaml. It defaults to the repository root relative to the project, so the targets can also be run directly from the project directory.

Special Targets

KNFSD FSIDD Project

The knfsd-fsidd project runs its tests through a helper script instead of calling go test directly:

Target	Purpose
gotest	Runs test.sh

When developing locally it is faster to run ./test.sh up once, then ./test.sh run repeatedly.

NetApp Exports Project

The netapp-exports project includes an additional target, which every other target in that project depends on:

Target	Purpose	Files Created
gen_cert	Generate test certificates	internal/testcert/testcert.go

Image Smoke Tests

The smoke-tests project has no gotest target. Its tests deploy real AWS infrastructure, so they are run explicitly through the following targets:

Target	Purpose
build-remote	Cross-compile the remote.test binary for the NFS test client
test	Run all test stages (apply, check, destroy)
apply	Apply Terraform infrastructure only
check	Run tests without applying or destroying Terraform
destroy	Destroy Terraform infrastructure only
clean	Clean up test artifacts and Terraform files

build-remote cross-compiles for the Terraform ARCH variable, not the host architecture. Override it with TARGET_ARCH when ARCH is not the amd64 default:

make test TARGET_ARCH=arm64

make clean refuses to run while Terraform still holds state, so run make destroy first.

Pre-commit Configuration

The pre-commit system is configured through .pre-commit-config.yaml and includes:

Local Hooks

These hooks use the project's Makefile targets:

• ec - Validates file formatting consistency (make ec)
• codespell - Checks spelling in code and documentation (make codespell)
• shfmt - Reports shell script formatting differences (make shfmt)
• shellcheck-sh - Lints tracked *.sh scripts (make shellcheck-sh)
• shellcheck-bash - Lints tracked *.bash scripts (make shellcheck-bash)
• shellcheck-bats - Lints tracked *.bats test files (make shellcheck-bats)
• black - Formats Python code (make black)
• mypy - Performs static type checking on Python code (make mypy)
• pylint - Lints Python code for style and quality issues (make pylint)
• iam-size - Checks IAM policy JSON files against the AWS size limit (make iam-size)
• packer-format - Formats Packer HCL files (make packer-fmt)
• packer-validate - Validates Packer template syntax (make packer-val)
• terraform-format - Formats Terraform files (make tf-fmt)
• terraform-lint - Lints Terraform code with TFLint (make tf-lint)

NOTE: fail_fast is enabled, so pre-commit stops at the first failing hook. Files matching *.patch or *mountstats are excluded from all hooks.

External Repository Hooks

Pre-commit Hooks (pre-commit/pre-commit-hooks)

• check-added-large-files - Prevents committing large files
• check-case-conflict - Checks for case-insensitive filename conflicts
• check-json - Validates JSON syntax
• check-merge-conflict - Detects merge conflict markers
• check-shebang-scripts-are-executable - Ensures scripts with shebangs are executable
• check-vcs-permalinks - Validates VCS permalink formats
• check-yaml - Validates YAML syntax (excludes mkdocs.yml)
• check-yaml-unsafe - Validates mkdocs.yml syntax only, allowing the MkDocs !ENV custom tag
• detect-aws-credentials - Prevents committing AWS credentials
• detect-private-key - Prevents committing private keys
• mixed-line-ending - Enforces consistent line endings (LF)
• pretty-format-json - Formats JSON files consistently (excludes dashboard.json files)
• trailing-whitespace - Removes trailing whitespace

YAMLlint (adrienverge/yamllint)

• yamllint - Advanced YAML linting with .yamllint.yaml configuration

GitLeaks (zricethezav/gitleaks)

• gitleaks - Scans for secrets and credentials in git history

Conventional Commits (compilerla/conventional-pre-commit)

• conventional-pre-commit - Enforces conventional commit message format

Configuration Files

File	Purpose	Tool
.editorconfig	Code formatting consistency across editors	EditorConfig
.codespellrc	Spell checking configuration	Codespell
.shellcheckrc	Shell script linting rules	ShellCheck
.yamllint.yaml	YAML linting rules	YAMLlint
.tflint.hcl	Terraform linting configuration	TFLint
.golangci.yaml	Go linting configuration	golangci-lint
.checkov.yaml	Infrastructure security scanning	Checkov
.kics.yaml	Infrastructure security analysis	KICS
.semgrepignore	Static analysis exclusions	Semgrep
.trivyignore.yaml	Vulnerability scanning exclusions	Trivy
.trivy.tfvars	Terraform variables for misconfig scanning	Trivy
.gitleaksignore	Secret scanning exclusions	GitLeaks

VS Code Integration

When working in VS Code, you can easily run pre-commit checks using the integrated terminal:

# Open terminal in VS Code (Ctrl+` or Cmd+`)
make pc

# Or run specific checks
make lint
make tf-fmt
make golint

The terminal will show color-coded output from each tool, making it easy to identify and fix issues.

Benefits

• Consistency - Enforces uniform code style and formatting across the entire codebase
• Quality - Catches common programming errors and security vulnerabilities early
• Automation - Reduces manual code review overhead by automating style and basic quality checks
• Standards - Ensures compliance with industry best practices and coding standards
• Security - Prevents accidental commits of secrets, credentials, and vulnerable code patterns
• Documentation - Maintains high-quality documentation through spell checking and formatting

Troubleshooting

Tool-specific Issues

If individual tools fail:

# Update tool versions
make pc-update

# Clear pre-commit cache
pre-commit clean

# Run specific hook
pre-commit run <hook-name> --all-files

Go Tool Issues

If Go tools fail:

# Install/update Go tools
make goget

# Clean Go module cache
go clean -modcache