What is Mage ?

Mage is Open-source data pipeline tool for transforming and integrating data. It’s alternative of Airflow gives more power to your Data Team.

• Integrate and synchronize data from 3rd party sources.

• Build real-time and batch pipelines to transform data using Python, SQL, and R.

• Run, monitor, and orchestrate thousands of pipelines without losing sleep

Mage has having powerful interactive notebook UI for development which  provides capabilities to write codes in Python, R and SQL.

Installation – Kubernetes

1. Start a Kubernetes cluster locally

   Enable Kubernetes in Docker Desktop to start a kubernetes cluster locally.

2. Download and update Kubernetes config file

   • Download Mage Kubernetes config file. https://github.com/mage-ai/mage-ai/blob/master/kube/app.yaml

1. • Replace the /path/to/mage_project in the config yaml with the path that you want to use to store your Mage projects.

2. Run Mage app in Kubernetes cluster

   • Install command line tool kubectl if you haven’t.
   • Run command kubectl create -f kube/app.yaml to run Mage in a Kubernetes pod.
   • You can check the pod status with command kubectl get pods -o wide.
   • Set up port forwarding with command kubectl port-forward mage-server 6789:6789.

3. Open http://localhost:6789 in your browser and build a pipeline.