Apache Airflow

[{"selector":"#anim-5aaba0b4-e279-47f8-972b-6f9e5abf443c","keyframes":{"opacity":[0,1]},"delay":250,"duration":1400,"easing":"cubic-bezier(0.2, 0.6, 0.0, 1)","fill":"both"}] [{"selector":"#anim-e0a084d3-e781-4071-878e-e680b094ec59","keyframes":{"transform":["translate3d(0px, 776.90607%, 0)","translate3d(0px, 0px, 0)"]},"delay":250,"duration":1400,"easing":"cubic-bezier(0.2, 0.6, 0.0, 1)","fill":"both"}]

Apache Airflow is an open-source workflow management platform.

[{"selector":"#anim-b7e405cf-18bd-49c4-b8a8-7cbdf77c511e","keyframes":{"opacity":[0,1]},"delay":250,"duration":1400,"easing":"cubic-bezier(0.2, 0.6, 0.0, 1)","fill":"both"}] [{"selector":"#anim-f60fbf2d-1607-40f0-b290-285bb9b74b31","keyframes":{"transform":["translate3d(0px, 235.28417%, 0)","translate3d(0px, 0px, 0)"]},"delay":250,"duration":1400,"easing":"cubic-bezier(0.2, 0.6, 0.0, 1)","fill":"both"}]

It started at Airbnb in October 2014, and was later open source to the apache community.

[{"selector":"#anim-21136278-93b9-479c-b02c-f63c3e92190e","keyframes":{"opacity":[0,1]},"delay":250,"duration":1400,"easing":"cubic-bezier(0.2, 0.6, 0.0, 1)","fill":"both"}] [{"selector":"#anim-65c81e99-4f2a-496c-a87b-fd93b9d05d81","keyframes":{"transform":["translate3d(0px, 175.12630%, 0)","translate3d(0px, 0px, 0)"]},"delay":250,"duration":1400,"easing":"cubic-bezier(0.2, 0.6, 0.0, 1)","fill":"both"}]

Apache airflow is used to define and schedule the data pipelines.

[{"selector":"#anim-8a9f644e-0e97-4a07-9796-df0eba79fd8a","keyframes":{"opacity":[0,1]},"delay":250,"duration":1400,"easing":"cubic-bezier(0.2, 0.6, 0.0, 1)","fill":"both"}] [{"selector":"#anim-8ce847a9-369a-470f-804b-3726a2424822","keyframes":{"transform":["translate3d(0px, 235.28417%, 0)","translate3d(0px, 0px, 0)"]},"delay":250,"duration":1400,"easing":"cubic-bezier(0.2, 0.6, 0.0, 1)","fill":"both"}]

In production, most people use apache airflow to schedule the spark job.

[{"selector":"#anim-effc52cd-8734-46ce-b362-2bcf5bc1f981","keyframes":{"opacity":[0,1]},"delay":250,"duration":1400,"easing":"cubic-bezier(0.2, 0.6, 0.0, 1)","fill":"both"}] [{"selector":"#anim-1ff38023-193b-4ebe-b902-8a18bcd03c98","keyframes":{"transform":["translate3d(0px, 175.12630%, 0)","translate3d(0px, 0px, 0)"]},"delay":250,"duration":1400,"easing":"cubic-bezier(0.2, 0.6, 0.0, 1)","fill":"both"}]

Apache airflow python to define the workflows.

[{"selector":"#anim-c5e97804-b03d-40da-8675-14475860f64a","keyframes":{"opacity":[0,1]},"delay":250,"duration":1400,"easing":"cubic-bezier(0.2, 0.6, 0.0, 1)","fill":"both"}] [{"selector":"#anim-a8b967c5-f54c-474a-adf7-fe8f16a29e05","keyframes":{"transform":["translate3d(0px, 362.61428%, 0)","translate3d(0px, 0px, 0)"]},"delay":250,"duration":1400,"easing":"cubic-bezier(0.2, 0.6, 0.0, 1)","fill":"both"}]

Airflow used PostgreSQL to store the metadata information.

[{"selector":"#anim-52698860-38fd-4f16-823c-6e0b8f9f5c96","keyframes":{"opacity":[0,1]},"delay":250,"duration":1400,"easing":"cubic-bezier(0.2, 0.6, 0.0, 1)","fill":"both"}] [{"selector":"#anim-c3b344d1-c536-4fd5-999a-c57d9551c1ae","keyframes":{"transform":["translate3d(0px, 235.28417%, 0)","translate3d(0px, 0px, 0)"]},"delay":250,"duration":1400,"easing":"cubic-bezier(0.2, 0.6, 0.0, 1)","fill":"both"}]

Apache airflow also provides an excellent UI to view and trigger the data pipelines.

[{"selector":"#anim-ff20d683-e87e-4b09-98d6-e000635fd9ce","keyframes":{"opacity":[0,1]},"delay":250,"duration":1400,"easing":"cubic-bezier(0.2, 0.6, 0.0, 1)","fill":"both"}] [{"selector":"#anim-3b3d99b2-639b-4003-a254-5b0374aa3f22","keyframes":{"transform":["translate3d(0px, 175.12630%, 0)","translate3d(0px, 0px, 0)"]},"delay":250,"duration":1400,"easing":"cubic-bezier(0.2, 0.6, 0.0, 1)","fill":"both"}]

Apache airflow can run on standalone VM, docker, and k8s.

[{"selector":"#anim-01ed6a77-4271-47f5-871a-39bdfa4e52e8","keyframes":{"opacity":[0,1]},"delay":250,"duration":1400,"easing":"cubic-bezier(0.2, 0.6, 0.0, 1)","fill":"both"}] [{"selector":"#anim-535cf3ea-1cd9-43a2-b348-cd9df0255de5","keyframes":{"transform":["translate3d(0px, 235.28417%, 0)","translate3d(0px, 0px, 0)"]},"delay":250,"duration":1400,"easing":"cubic-bezier(0.2, 0.6, 0.0, 1)","fill":"both"}]

Thank You

[{"selector":"#anim-b3545851-4ed8-4a02-95ad-7a490f48efba","keyframes":{"opacity":[0,1]},"delay":250,"duration":1400,"easing":"cubic-bezier(0.2, 0.6, 0.0, 1)","fill":"both"}] [{"selector":"#anim-0f466846-acae-44ac-94b4-c86a642d752e","keyframes":{"transform":["translate3d(0px, 577.96957%, 0)","translate3d(0px, 0px, 0)"]},"delay":250,"duration":1400,"easing":"cubic-bezier(0.2, 0.6, 0.0, 1)","fill":"both"}] [{"selector":"#anim-66ee4eee-0b09-49ce-b297-84f42ec394c3","keyframes":{"opacity":[0,1]},"delay":250,"duration":1400,"easing":"cubic-bezier(0.2, 0.6, 0.0, 1)","fill":"both"}] [{"selector":"#anim-2113b08d-0ce5-47a8-b222-037268f2b22f","keyframes":{"transform":["translate3d(0px, 548.05569%, 0)","translate3d(0px, 0px, 0)"]},"delay":250,"duration":1400,"easing":"cubic-bezier(0.2, 0.6, 0.0, 1)","fill":"both"}]