What is Mage ?

Mage is Open-source data pipeline tool for transforming and integrating data. It’s alternative of Airflow gives more power to your Data Team.

  • Integrate and synchronize data from 3rd party sources.

  • Build real-time and batch pipelines to transform data using Python, SQL, and R.

  • Run, monitor, and orchestrate thousands of pipelines without losing sleep

mage pipeline in 2 minutes

Mage has having powerful interactive notebook UI for development which  provides capabilities to write codes in Python, R and SQL.

Installation – Kubernetes

  1. Start a Kubernetes cluster locally

    Enable Kubernetes in Docker Desktop to start a kubernetes cluster locally.

  2. Download and update Kubernetes config file

    • Replace the /path/to/mage_project in the config yaml with the path that you want to use to store your Mage projects.
  1. Run Mage app in Kubernetes cluster

    • Install command line tool kubectl if you haven’t.
    • Run command kubectl create -f kube/app.yaml to run Mage in a Kubernetes pod.
    • You can check the pod status with command kubectl get pods -o wide.
    • Set up port forwarding with command kubectl port-forward mage-server 6789:6789.
  2. Open http://localhost:6789 in your browser and build a pipeline.