Pre-requisites

In this section we outline the main pre-requisites for using Project Lakechain and start deploying pipelines.

💻 Environment

Project Lakechain has been successfully tested on different Linux distributions, MacOS, and Cloud development environments such as GitHub Codespaces.

We recommend having 50GB of free storage on your development machine to be able to build and deploy all the middlewares and examples.

👇 We have a ready made Dev Container for GitHub Codespaces that you can use to get started quickly.

---

☁️ AWS Access

You will need access to an AWS account on your development machine with valid credentials. You can use the AWS CLI to verify that you have valid credentials.

ℹ️ The AWS documentation describes how to configure the AWS CLI.

$ aws sts get-caller-identity

{
  "UserId": "USEREXAMPLE",
  "Account": "123456789012",
  "Arn": "arn:aws:sts::123456789012:user/JohnDoe"
}

---

🐳 Docker

As some middlewares are packaged as Docker containers, you need to have Docker installed and running on your development machine. You can use the Docker CLI to verify that you have access to the Docker daemon.

$ docker version

Client: Docker Engine - Community
 Version:           20.10.7
 API version:       1.41
 Go version:        go1.13.15
 Git commit:        f0df350
 Built:             Wed Jun  2 11:56:39 2021
 OS/Arch:           darwin/amd64
 Context:           default

---

📦 Node.js + NPM

Node.js 20+ and NPM must be available to install the Lakechain project dependencies. You can use the Node.js CLI to verify that you have access to the Node.js runtime.

You can use nvm to easily manage multiple versions of Node.js on your development machine.

$ node --version

v20.3.1

---

🐍 Python and Pip

Python 3.9+ and Pip are used to package some Lakechain middlewares written in Python. You can use the Python binary to verify that you have a Python 3.9+ runtime up and running.

$ python3 --version

Python 3.11.5

---

Optional Dependencies

Although optional, as they will be installed by Project Lakechain and run using npx, we recommend installing the following dependencies on your development machine:

TypeScript 5.0+

$ npm install -g typescript

AWS CDK v2

$ npm install -g aws-cdk