{"id":1126,"date":"2022-08-16T11:08:17","date_gmt":"2022-08-16T11:08:17","guid":{"rendered":"https:\/\/www.workfall.com\/learning\/blog\/?p=1126"},"modified":"2025-09-30T04:54:13","modified_gmt":"2025-09-30T04:54:13","slug":"how-to-easily-build-etl-pipeline-using-python-and-airflow","status":"publish","type":"post","link":"https:\/\/learning.workfall.com\/learning\/blog\/how-to-easily-build-etl-pipeline-using-python-and-airflow\/","title":{"rendered":"Easily build ETL Pipeline using Python and Airflow"},"content":{"rendered":"<span class=\"rt-reading-time\" style=\"display: block;\"><span class=\"rt-label rt-prefix\">Reading Time: <\/span> <span class=\"rt-time\">5<\/span> <span class=\"rt-label rt-postfix\">minutes<\/span><\/span>\n<p><img src=\"https:\/\/lh3.googleusercontent.com\/JMf-sG1xv6OyzgJSGNhSuet8yqXr0-lHKrcsLk_bpbMjYebXacl69pJsgA1pokLdiv1IE4pUVpv1y1ObOq8kovaTCrD6XTssjFok67g2jBRRowrjguzWZW9Sse4Ie-8n42rPm3J0e0-mjhmHq1XRI3E\" style=\"width: 1600px;\"><\/p>\n\n\n\n<p class=\"has-text-align-justify\">Apache Airflow is an open-source workflow management platform for authoring, scheduling, and monitoring workflows or data pipelines programmatically. Python is used to write Airflow, and Python scripts are used to create workflows. It was created by Airbnb. In this blog, we will show how to configure airflow on our machine as well as write a Python script for extracting, transforming, and loading (ETL) data and running the data pipeline that we have built.<\/p>\n\n\n\n<p>In this blog, we will cover:<\/p>\n\n\n\n<ul><li>What is a Workflow?<\/li><li>Hands-on<\/li><li>Conclusion<\/li><\/ul>\n\n\n\n<h2>What is a Workflow?<\/h2>\n\n\n\n<ul><li>A sequence of tasks that are started or scheduled or triggered by an event.<\/li><li>Frequently used to handle big data processing pipelines<\/li><\/ul>\n\n\n\n<p class=\"has-text-align-justify\">DAG for airflow Airflow employs a workflow as a Directed Acyclic Graph (DAG) in which multiple tasks can be executed independently.<\/p>\n\n\n\n<p>An example of DAG:<\/p>\n\n\n\n<figure class=\"wp-block-image\"><img src=\"https:\/\/lh5.googleusercontent.com\/rAgvuJ50Uy9v_-4pP5IIwYwxazTx8fLKRDOM836t40hZj38XR2g4ZJVK64iinlyH4TAoLhoTNeiwsZmYcWTc5MMqU2N4wtJITMq0Up0KIZAG9uQfVT9ELGmmLsP26trWyQ1eR0YUSmv5YgHuHO7yFXk\" alt=\"Easily build ETL Pipeline using Python and Airflow\"\/><\/figure>\n\n\n\n<h2>Hands-on<\/h2>\n\n\n\n<p>Things we will do to create our first Airflow ETL pipeline in this blog:<\/p>\n\n\n\n<ul><li>Setting up the Airflow and VS code editor.&nbsp;<\/li><li>Download the dummy cat details data from an API.&nbsp;<\/li><li>Transform the data&nbsp;<\/li><li>Load the data to a CSV file<\/li><\/ul>\n\n\n\n<p><strong>Setting up the Airflow and VS code editor<\/strong><\/p>\n\n\n\n<ul><li>Install the<a href=\"https:\/\/www.virtualbox.org\/\"> Virtual Box VM<\/a> to run the Ubuntu Server. Also, download the<a href=\"https:\/\/ubuntu.com\/download\/server\"> Ubuntu Server ISO file<\/a> before moving forward.<\/li><li>Open the Oracle VM Virtual box manager and click on the \u2018new\u2019 button.<\/li><li>Write the name of your Ubuntu server and click on next in the next few steps to complete the setup. Go with the recommended choices.<\/li><\/ul>\n\n\n\n<figure class=\"wp-block-image\"><img src=\"https:\/\/lh5.googleusercontent.com\/3szHGUM28aKAF9x4eKND9kgIeyFUbY53jLquYT_4Sd0Dj49noe316bGEBfSE0pTMJkhythc-LBkNCyjuWrz1TywBi0XI2Zz2BSTW1LnVgi9l41AeKjLVhmGLhGV1QON4Tgisqw6zf1D3_7j1V4Z8AC8\" alt=\"Easily build ETL Pipeline using Python and Airflow\"\/><\/figure>\n\n\n\n<ul><li>Once you have completed the setup, you will see the virtual machine in this manner:<\/li><\/ul>\n\n\n\n<figure class=\"wp-block-image\"><img src=\"https:\/\/lh4.googleusercontent.com\/lkOLtD3tyy2d1X8N5CKNFW8mtPT9ORv6LwUUfs-uaWN5ELz9LjmcvPQz05YFiHkG4cjGoMNWvOyuDMCTDVe5WtT_8Jxp8DFPccdhQuJ45Jwpl0hHirXwkagGvxmrhktcEg0p5RdAoOHSS2Md94P8Flc\" alt=\"Easily build ETL Pipeline using Python and Airflow\"\/><\/figure>\n\n\n\n<ul><li>Now let us configure the network, so go to Settings for the created machine and go to Network. Under Advanced, click on the Port Forwarding button:<\/li><\/ul>\n\n\n\n<p>In this, add the two ports as shown below:<\/p>\n\n\n\n<figure class=\"wp-block-image\"><img src=\"https:\/\/lh5.googleusercontent.com\/WBdwtFHAU8zam8ivpEYI3kD8dOJufukI0ghpuWCGVznJsKaplqS8PeIriq94w7Zqbd0Ij-gcZ_qhaj4s1CtpLt3YL_NjQhSt2N5gy0NLmy_5bvYopMWS2I7x60OxKE6Qmqibgh_WxiDYL_TZgpsk5ko\" alt=\"Easily build ETL Pipeline using Python and Airflow\"\/><\/figure>\n\n\n\n<p>Guest 8080 to Host 8250 will be used for Airflow UI<\/p>\n\n\n\n<p>Guest 22 to Host 2222 for SSH connection<\/p>\n\n\n\n<ul><li>Run the virtual machine and install the OS<\/li><\/ul>\n\n\n\n<figure class=\"wp-block-image\"><img src=\"https:\/\/lh3.googleusercontent.com\/2yS_gbrzo1-ckYjypguQbTtEoIMqHmzMNRXCHRRiW72_aYbrf_fbo9xt3d9KFHPLg7_aey2CFRVdvG2K6iAD3cBo7_3x2xQPF325DYmC_evhuSE7Jyb1oytZe06Y2Ga-HGmr9wmIjnqebrzb-LprQEQ\" alt=\"\"\/><\/figure>\n\n\n\n<ul><li>The installation window will look something similar to this:<\/li><\/ul>\n\n\n\n<figure class=\"wp-block-image\"><img src=\"https:\/\/lh5.googleusercontent.com\/xkeVRrKRZnwKo8iRVBO4TDR9g8oGyxZjWHMV_gyVOTUq9hKW0gOlfXle6tePfCi3W93nPora7IQ1zuZuFstTqGbPW2UjlNenzILj4GwMt8HpgfE1cDGu7miKEetyCAtWZYLWEA_KEn0hRjr9Gi-9SHI\" alt=\"Easily build ETL Pipeline using Python and Airflow\"\/><\/figure>\n\n\n\n<ul><li>Now, create a user, here it is \u201cairflow\u201d. \u2018sudo adduser airflow\u2019 and type in the password<br><\/li><li>Establish SSH connection<\/li><\/ul>\n\n\n\n<p>In the windows machine, open the terminal and make a connection by typing<\/p>\n\n\n\n<p><strong>ssh -p 2222 airflow@localhost<\/strong><\/p>\n\n\n\n<figure class=\"wp-block-image\"><img src=\"https:\/\/lh5.googleusercontent.com\/e1bykJMK0BU2BKFYzIoKUkOyxFySR70O7xatxwWhRHvQuGnzuTuxIKgAo02Ux5VERB6oaJn6gSjxqeBUjCLLoruSqoctKj5rsA8kxMGoCrcNSrca0ooVaCyqcXfoRvwu3SPheg8yZBFjWxTdBPdsVyQ\" alt=\"\"\/><\/figure>\n\n\n\n<ul><li>Install Python and Airflow on a Virtual Linux machine<\/li><\/ul>\n\n\n\n<p>To Install python, run the command:<\/p>\n\n\n\n<p><strong><code>sudo apt install python3-pip<\/code><\/strong><\/p>\n\n\n\n<p>Following that, install the Airflow by the following command:<\/p>\n\n\n\n<p><strong><code>sudo pip3 install apache-airflow<\/code><\/strong><\/p>\n\n\n\n<ul><li>Initialize the database for Airflow:<\/li><\/ul>\n\n\n\n<p>airflow db init<\/p>\n\n\n\n<ul><li>Create a user in airflow<\/li><\/ul>\n\n\n\n<p><em>airflow users create -u admin\u200a\u2014\u200af first_name\u200a\u2014\u200al last_name&gt;\u200a\u2014\u200arole Admin\u200a\u2014\u200ae your_email<\/em><\/p>\n\n\n\n<figure class=\"wp-block-image\"><img src=\"https:\/\/lh5.googleusercontent.com\/m3CbsJm6dP82jpLZ5dmywc97hPz9JgytDJfMRLjgRSf6MurMLce_rPJiRqd2GSdb9oJ8Of3SE2xE0ZWpPuH1fy4VAVj4-FDufzhOX9VBB5owD_2UjQjBQqkc-GfMqwhnl6PufZAgZSr-CzYTxp18UF4\" alt=\"\"\/><\/figure>\n\n\n\n<ul><li>Start the airflow web server<\/li><\/ul>\n\n\n\n<p>airflow webserver -D<\/p>\n\n\n\n<p class=\"has-text-align-justify\">After this step, our Apache webserver is now running. Open the web browser and go to localhost:8250 and log in. You would see the DAGs in the snapshot below.<\/p>\n\n\n\n<figure class=\"wp-block-image\"><img src=\"https:\/\/lh6.googleusercontent.com\/1swOYnJxzCAzC0OB8mmeRj43PL-lArDuWxwxo36doGVxbPmYhTqoqXmglnwLJg6lqRGHPuikL-NfUN7fJyQhVHA2yDALd-ZuqzvV276MplGWzN9boO2mJx5WkWtl3GO5vlimk4PNBJKcvb-C9bAluBs\" alt=\"\"\/><\/figure>\n\n\n\n<ul><li>Configuring VS Code editor<ul><li>We would use the VS Code editor to write our python scripts.<\/li><li>Install the plugin Remote\u200a\u2014\u200aSSH and connect to the host by typing ssh -p 2222 airflow@localhost<\/li><li>Add the connection configuration to SSH configurations, so select the first option<\/li><li>When prompted for password, enter the same password you have set for Ubuntu airflow user.<\/li><li>Now open the \/home\/airflow folder<\/li><\/ul><\/li><\/ul>\n\n\n\n<p>After setup, you will see:<\/p>\n\n\n\n<figure class=\"wp-block-image\"><img src=\"https:\/\/lh4.googleusercontent.com\/vSOzZDFDWPtYUQKRdzgGvOUvttIyePZZ3cBNZC7NMcEuFYKearuSKlNZVebPdXT0M0j8zc0coMgjXHDOeZtoLwj6WmX5bPLYsjLx1JZlWPuYHj5quyB-nKgwiYLIAo1QyVKKqYTyJX_VKlqew82hBrM\" alt=\"Easily build ETL Pipeline using Python and Airflow\"\/><\/figure>\n\n\n\n<p class=\"has-text-align-justify\">And that\u2019s it\u200a\u2014\u200ayou\u2019re ready to create your first Airflow DAG. Make sure to put them inside the dags folder (you\u2019ll have to create it first), as that\u2019s where Airflow will try to find them<\/p>\n\n\n\n<h3><strong>Coding the Pipeline<\/strong><\/h3>\n\n\n\n<p class=\"has-text-align-justify\">We would write a python script for extracting, transforming, and loading (ETL) data and running the data pipeline that we have created.<\/p>\n\n\n\n<p>Create a Python file in dags\/cats_<a href=\"http:\/\/pipeline.py\/\">pipeline.py<\/a>.<\/p>\n\n\n\n<p><strong>Extraction<\/strong><\/p>\n\n\n\n<p class=\"has-text-align-justify\">In this example, we would be extracting the data about some of the facts about cats from catfact.ninja API. So we would use the requests library and get the JSON response.<\/p>\n\n\n\n<figure class=\"wp-block-image\"><img src=\"https:\/\/lh5.googleusercontent.com\/mDRvVo-WrW0SL_JMPm8pGS-kbee5PxSsVNRAjhHRS4rxV-_gcZSclpTZpEZUHX65F9zbjJISlD-wY2BlEbzX8PsWt6WZWtNBqPJBVajcTQ39JhqCEfOsNt6AJdhsCmCcJhXodlK2yfmhwgXAdsEaK9o\" alt=\"\"\/><\/figure>\n\n\n\n<figure class=\"wp-block-image\"><img src=\"https:\/\/lh6.googleusercontent.com\/COXn8K5X_n-LcNBMKlnUoZ8R45gU3GCbQm3zakAN8RQDQS41bqlu8s_Kh8h6AyY9eexIu6R2wVMA0bh0w1AfUrZteBRwCCta8SxPWM6zTDETq3mnLVpvBQ_Ow5JURIXd_OEhDQPjDP-WZ5LrGiG14XY\" alt=\"\"\/><\/figure>\n\n\n\n<p>xcom_push will save the fetched results to Airflow\u2019s database.<\/p>\n\n\n\n<p><strong>Transformation<\/strong><\/p>\n\n\n\n<p class=\"has-text-align-justify\">Here, we have created a dummy function to do xcom_pull which will get the data and transform it to our requirements.<\/p>\n\n\n\n<figure class=\"wp-block-image\"><img src=\"https:\/\/lh3.googleusercontent.com\/31x5ppYEy8zXkFz_xyI3LU4GbsQJL6JauNq4ozn4YRcKAqbVFNovdesZpwd3Xykf8QPCGH5Xfe7gSDDRc1HRNvLvUn0lalyZnJPyJmGAaAPJePG-0e8cR5hyRud7gZVDQN-pCzU_Di-I33V9N4Q2K60\" alt=\"Easily build ETL Pipeline using Python and Airflow\"\/><\/figure>\n\n\n\n<p><strong>Loading<\/strong><\/p>\n\n\n\n<p>We shall do the xcom_pull to get the transformed cats data and save it to a CSV file.<\/p>\n\n\n\n<figure class=\"wp-block-image\"><img src=\"https:\/\/lh5.googleusercontent.com\/BAqbMlBn7v-45y6-wuWIJdmnwAHK_tr8p1-Ds4bKqVOrVD4LLp_kD8WLtNEWgHMaIZq5TQ8c2j8ua-Nl9qq5ln9QKLJHOs84UDH_mUMihQydFb4rk3-xhtBip52reHZdMegeBmBONIMJmmGGZt6UWQ4\" alt=\"Easily build ETL Pipeline using Python and Airflow\"\/><\/figure>\n\n\n\n<p>Directed Acyclic Graph using Python<\/p>\n\n\n\n<p class=\"has-text-align-justify\">This code will help us to arrange the extract, transform and load tasks in a DAG workflow format and help us to create a pipeline. Apache Airflow is based on the idea of DAGs (Directed Acyclic Graphs). This means we\u2019ll have to specify tasks for pieces of our pipeline and then arrange them somehow. We would use PythonOperator based tasks.<\/p>\n\n\n\n<p>Code for DAG using PythonOperator:<\/p>\n\n\n\n<figure class=\"wp-block-image\"><img src=\"https:\/\/lh6.googleusercontent.com\/ci6EwMJDYN934HuNyNsz805sDjDDpxNobnMmJFCa2W1L53P4NP2hFz28vIuLWlo1RuFsqh2n06VRf8HDBDp5kWhIA7LLS7wPbANU-hpziKubsiomPOReGYtNK7tOEEp_1IsVGu7h6GmtoOVRYlLW6S4\" alt=\"Easily build ETL Pipeline using Python and Airflow\"\/><\/figure>\n\n\n\n<h3><strong>Run the Pipeline<\/strong><\/h3>\n\n\n\n<p>Now, go to the Airflow UI On the local host and you would see the DAG etl_cats_data.<\/p>\n\n\n\n<p class=\"has-text-align-justify\">Initially, the pipeline is not running so we manually would have to trigger the DAG. Switch on the pipeline and click on the Play button. You can now see the pipeline running.<\/p>\n\n\n\n<p>After the pipeline is completed, we can now check whether the data has been loaded or not.<\/p>\n\n\n\n<figure class=\"wp-block-image\"><img src=\"https:\/\/lh4.googleusercontent.com\/kq6P1seKzeAQ5BTpVhx_YjjC6CYnB9ENjFd-57zG2Ym4ylvFMLT7AAozNCMJRB0Nq7qSAGEkn2KbVpiQfFoilAYptUyD4c2gCG9WF9Zd618aWIRYVaRutpWHTlHURZk2Fa9e0bqa9dRmBqB6ZDLx4Fw\" alt=\"\"\/><\/figure>\n\n\n\n<p>The data load was a success.<\/p>\n\n\n\n<h2>Conclusion<\/h2>\n\n\n\n<p class=\"has-text-align-justify\">Apache Airflow simplifies the creation of data pipelines while also optimizing management and scheduling tasks. It is widely used in the software industry for orchestrating both ETL (Extract Load Transform) and ELT (Extract Load Transform) for data warehouse applications. You will be able to create your first data pipeline using this blog, and you can also use it as a template\/format to create multiple pipelines based on your needs. We will come up with more such use cases in our upcoming blogs.<\/p>\n\n\n\n<p><strong>Meanwhile\u2026<\/strong><\/p>\n\n\n\n<p class=\"has-text-align-justify\">If you are an aspiring Python developer and want to explore more about the above topics, here are a few of our blogs for your reference:<\/p>\n\n\n\n<ul><li><a href=\"https:\/\/www.workfall.com\/learning\/blog\/how-to-build-a-web-scraper-using-python\/\">How to build a Web Scraper using Python?<\/a><\/li><li><a href=\"https:\/\/www.workfall.com\/learning\/blog\/how-to-send-webhooks-using-python-and-receive-via-node-js-applications\/\">How to send Webhooks using Python and receive via Node.js Applications?<\/a><\/li><\/ul>\n\n\n\n<p>Stay tuned to get all the updates about our upcoming blogs on the cloud and the latest technologies.<\/p>\n\n\n\n<p class=\"has-text-align-justify\"><strong>Keep Exploring -> Keep Learning -> Keep Mastering\u00a0<\/strong><br>At <a href=\"https:\/\/www.workfall.com\/\">Workfall<\/a>, we strive to provide the best tech and pay opportunities to kickass coders around the world. If you\u2019re looking to work with global clients, build cutting-edge products and make big bucks doing so, give it a shot at <a href=\"https:\/\/www.workfall.com\/partner\/\">workfall.com\/partner<\/a> today!<\/p>\n\n\n\n<p><\/p>\n\n\n<style type=\"text\/css\"><\/style><section id='' \n                class='helpie-faq accordions faq-toggle open-first groupSettings-530__enabled' \n                data-collection='' \n                data-pagination='0' \n                data-search='0' \n                data-pagination-enabled='0'\n                role='region'\n                aria-label='FAQ Section'\n                aria-live='polite'><h3 class=\"collection-title\">Frequently Asked Questions:<\/h3><article class=\"accordion \"><div class='helpie-faq-row'><div class='helpie-faq-col helpie-faq-col-12' ><ul><li class=\"accordion__item \"><div class=\"accordion__header \" \r\n                id=\"accordion-header-post-3035\"\r\n                role=\"button\"\r\n                aria-expanded=\"false\"\r\n                aria-controls=\"accordion-content-post-3035\"\r\n                data-id=\"post-3035\" \r\n                data-item=\"hfaq-post-3035\" \r\n                style=\"background:transparent;\" \r\n                data-tags=\"\"\r\n                tabindex=\"0\"><div class=\"accordion__title\">Q. What is Apache Airflow, and why is it suited for ETL pipelines?<\/div><\/div><div id=\"accordion-content-post-3035\" \r\n                class=\"accordion__body\" \r\n                role=\"region\"\r\n                aria-labelledby=\"accordion-header-post-3035\"\r\n                style=\"background:transparent;\"><p><span class=\"rt-reading-time\" style=\"display: block;\"><span class=\"rt-label rt-prefix\">Reading Time: <\/span> <span class=\"rt-time\">5<\/span> <span class=\"rt-label rt-postfix\">minutes<\/span><\/span><span style=\"font-weight: 400\">Airflow is an open-source workflow orchestration tool that allows you to define Directed Acyclic Graphs (DAGs) for scheduling and monitoring tasks. For ETL, Airflow is powerful because it supports dependency management, retries, task scheduling, and integrations with databases and cloud services.<\/span><\/p>\n<\/div><\/li><li class=\"accordion__item \"><div class=\"accordion__header \" \r\n                id=\"accordion-header-post-3036\"\r\n                role=\"button\"\r\n                aria-expanded=\"false\"\r\n                aria-controls=\"accordion-content-post-3036\"\r\n                data-id=\"post-3036\" \r\n                data-item=\"hfaq-post-3036\" \r\n                style=\"background:transparent;\" \r\n                data-tags=\"\"\r\n                tabindex=\"0\"><div class=\"accordion__title\">Q. How does Python integrate with Airflow for ETL tasks?<\/div><\/div><div id=\"accordion-content-post-3036\" \r\n                class=\"accordion__body\" \r\n                role=\"region\"\r\n                aria-labelledby=\"accordion-header-post-3036\"\r\n                style=\"background:transparent;\"><p><span class=\"rt-reading-time\" style=\"display: block;\"><span class=\"rt-label rt-prefix\">Reading Time: <\/span> <span class=\"rt-time\">5<\/span> <span class=\"rt-label rt-postfix\">minutes<\/span><\/span><span style=\"font-weight: 400\">\u00a0Python is used to define DAGs and operators in Airflow. You can write Python functions to extract data from APIs\/databases, transform it (cleaning, aggregation), and load it into a destination like AWS S3, Redshift, or a relational DB.<\/span><\/p>\n<\/div><\/li><li class=\"accordion__item \"><div class=\"accordion__header \" \r\n                id=\"accordion-header-post-3037\"\r\n                role=\"button\"\r\n                aria-expanded=\"false\"\r\n                aria-controls=\"accordion-content-post-3037\"\r\n                data-id=\"post-3037\" \r\n                data-item=\"hfaq-post-3037\" \r\n                style=\"background:transparent;\" \r\n                data-tags=\"\"\r\n                tabindex=\"0\"><div class=\"accordion__title\">Q. What are the key components of an ETL DAG in Airflow?<\/div><\/div><div id=\"accordion-content-post-3037\" \r\n                class=\"accordion__body\" \r\n                role=\"region\"\r\n                aria-labelledby=\"accordion-header-post-3037\"\r\n                style=\"background:transparent;\"><p><span class=\"rt-reading-time\" style=\"display: block;\"><span class=\"rt-label rt-prefix\">Reading Time: <\/span> <span class=\"rt-time\">5<\/span> <span class=\"rt-label rt-postfix\">minutes<\/span><\/span><\/p>\n<ul>\n<li><b>Operators<\/b><span style=\"font-weight: 400\"> (e.g., PythonOperator, BashOperator) for task logic<\/span><span style=\"font-weight: 400\">\n<p><\/span><\/li>\n<li><b>Tasks<\/b><span style=\"font-weight: 400\"> \u2014 instances of operators<\/span><span style=\"font-weight: 400\">\n<p><\/span><\/li>\n<li><b>DAGs<\/b><span style=\"font-weight: 400\"> \u2014 workflow definitions<\/span><span style=\"font-weight: 400\">\n<p><\/span><\/li>\n<li><b>Task dependencies<\/b><span style=\"font-weight: 400\"> \u2014 determine execution order<\/span><span style=\"font-weight: 400\">\n<p><\/span><\/li>\n<\/ul>\n<\/div><\/li><li class=\"accordion__item \"><div class=\"accordion__header \" \r\n                id=\"accordion-header-post-3038\"\r\n                role=\"button\"\r\n                aria-expanded=\"false\"\r\n                aria-controls=\"accordion-content-post-3038\"\r\n                data-id=\"post-3038\" \r\n                data-item=\"hfaq-post-3038\" \r\n                style=\"background:transparent;\" \r\n                data-tags=\"\"\r\n                tabindex=\"0\"><div class=\"accordion__title\">Q. How do retries and error handling work in Airflow pipelines?<\/div><\/div><div id=\"accordion-content-post-3038\" \r\n                class=\"accordion__body\" \r\n                role=\"region\"\r\n                aria-labelledby=\"accordion-header-post-3038\"\r\n                style=\"background:transparent;\"><p><span class=\"rt-reading-time\" style=\"display: block;\"><span class=\"rt-label rt-prefix\">Reading Time: <\/span> <span class=\"rt-time\">5<\/span> <span class=\"rt-label rt-postfix\">minutes<\/span><\/span><span style=\"font-weight: 400\">\u00a0Each task can have retry count, delay, and on-failure callbacks. If extraction fails, Airflow can retry automatically or alert the user via email\/SNS.<\/span><\/p>\n<\/div><\/li><li class=\"accordion__item \"><div class=\"accordion__header \" \r\n                id=\"accordion-header-post-3039\"\r\n                role=\"button\"\r\n                aria-expanded=\"false\"\r\n                aria-controls=\"accordion-content-post-3039\"\r\n                data-id=\"post-3039\" \r\n                data-item=\"hfaq-post-3039\" \r\n                style=\"background:transparent;\" \r\n                data-tags=\"\"\r\n                tabindex=\"0\"><div class=\"accordion__title\">Q. How can I monitor ETL jobs in Airflow?<\/div><\/div><div id=\"accordion-content-post-3039\" \r\n                class=\"accordion__body\" \r\n                role=\"region\"\r\n                aria-labelledby=\"accordion-header-post-3039\"\r\n                style=\"background:transparent;\"><p><span class=\"rt-reading-time\" style=\"display: block;\"><span class=\"rt-label rt-prefix\">Reading Time: <\/span> <span class=\"rt-time\">5<\/span> <span class=\"rt-label rt-postfix\">minutes<\/span><\/span><span style=\"font-weight: 400\">Airflow has a web UI where you can view DAG runs, logs, and task success\/failures. You can also enable email\/Slack alerts and integrate with external observability tools.<\/span><\/p>\n<\/div><\/li><\/ul><\/div><\/div><\/article><\/section>\n","protected":false},"excerpt":{"rendered":"<p><span class=\"rt-reading-time\" style=\"display: block;\"><span class=\"rt-label rt-prefix\">Reading Time: <\/span> <span class=\"rt-time\">5<\/span> <span class=\"rt-label rt-postfix\">minutes<\/span><\/span> Apache Airflow is an open-source workflow management platform for authoring, scheduling, and monitoring workflows or data pipelines programmatically. Python is used to write Airflow, and Python scripts are used to create workflows. It was created by Airbnb. In this blog, we will show how to configure airflow on our machine as well as write a [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":1127,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"spay_email":""},"categories":[288],"tags":[329,331,330,174,114,6],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v19.1 - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>Easily build ETL Pipeline using Python and Airflow - The Workfall Blog<\/title>\n<meta name=\"description\" content=\"Apache Airflow is an open-source workflow management platform for authoring, scheduling, and monitoring workflows.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/learning.workfall.com\/learning\/blog\/how-to-easily-build-etl-pipeline-using-python-and-airflow\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Easily build ETL Pipeline using Python and Airflow - The Workfall Blog\" \/>\n<meta property=\"og:description\" content=\"Apache Airflow is an open-source workflow management platform for authoring, scheduling, and monitoring workflows.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/learning.workfall.com\/learning\/blog\/how-to-easily-build-etl-pipeline-using-python-and-airflow\/\" \/>\n<meta property=\"og:site_name\" content=\"The Workfall Blog\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/facebook.com\/workfall\" \/>\n<meta property=\"article:published_time\" content=\"2022-08-16T11:08:17+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2025-09-30T04:54:13+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/18-141-20-153.plesk.page\/learning\/blog\/wp-content\/uploads\/2022\/08\/Cover-Images_Part2-1-2.png\" \/>\n\t<meta property=\"og:image:width\" content=\"1200\" \/>\n\t<meta property=\"og:image:height\" content=\"628\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@workfall\" \/>\n<meta name=\"twitter:site\" content=\"@workfall\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Workfall\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"8 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Organization\",\"@id\":\"https:\/\/learning.workfall.com\/learning\/blog\/#organization\",\"name\":\"Workfall - Hire #Kickass Coders On Demand\",\"url\":\"https:\/\/learning.workfall.com\/learning\/blog\/\",\"sameAs\":[\"https:\/\/www.instagram.com\/workfall\/\",\"https:\/\/www.linkedin.com\/company\/workfall\/\",\"https:\/\/facebook.com\/workfall\",\"https:\/\/twitter.com\/workfall\"],\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/learning.workfall.com\/learning\/blog\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/i1.wp.com\/18.141.20.153\/learning\/blog\/wp-content\/uploads\/2021\/10\/cropped-WF_logo.png?fit=400%2C400\",\"contentUrl\":\"https:\/\/i1.wp.com\/18.141.20.153\/learning\/blog\/wp-content\/uploads\/2021\/10\/cropped-WF_logo.png?fit=400%2C400\",\"width\":400,\"height\":400,\"caption\":\"Workfall - Hire #Kickass Coders On Demand\"},\"image\":{\"@id\":\"https:\/\/learning.workfall.com\/learning\/blog\/#\/schema\/logo\/image\/\"}},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/learning.workfall.com\/learning\/blog\/#website\",\"url\":\"https:\/\/learning.workfall.com\/learning\/blog\/\",\"name\":\"The Workfall Blog\",\"description\":\"#Tech #Remote #Jobs\",\"publisher\":{\"@id\":\"https:\/\/learning.workfall.com\/learning\/blog\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/learning.workfall.com\/learning\/blog\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-US\"},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/learning.workfall.com\/learning\/blog\/how-to-easily-build-etl-pipeline-using-python-and-airflow\/#primaryimage\",\"url\":\"https:\/\/learning.workfall.com\/learning\/blog\/wp-content\/uploads\/2022\/08\/Cover-Images_Part2-1-2.png\",\"contentUrl\":\"https:\/\/learning.workfall.com\/learning\/blog\/wp-content\/uploads\/2022\/08\/Cover-Images_Part2-1-2.png\",\"width\":1200,\"height\":628,\"caption\":\"ETL Pipeline using Python and Airflow\"},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/learning.workfall.com\/learning\/blog\/how-to-easily-build-etl-pipeline-using-python-and-airflow\/#webpage\",\"url\":\"https:\/\/learning.workfall.com\/learning\/blog\/how-to-easily-build-etl-pipeline-using-python-and-airflow\/\",\"name\":\"Easily build ETL Pipeline using Python and Airflow - The Workfall Blog\",\"isPartOf\":{\"@id\":\"https:\/\/learning.workfall.com\/learning\/blog\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/learning.workfall.com\/learning\/blog\/how-to-easily-build-etl-pipeline-using-python-and-airflow\/#primaryimage\"},\"datePublished\":\"2022-08-16T11:08:17+00:00\",\"dateModified\":\"2025-09-30T04:54:13+00:00\",\"description\":\"Apache Airflow is an open-source workflow management platform for authoring, scheduling, and monitoring workflows.\",\"breadcrumb\":{\"@id\":\"https:\/\/learning.workfall.com\/learning\/blog\/how-to-easily-build-etl-pipeline-using-python-and-airflow\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/learning.workfall.com\/learning\/blog\/how-to-easily-build-etl-pipeline-using-python-and-airflow\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/learning.workfall.com\/learning\/blog\/how-to-easily-build-etl-pipeline-using-python-and-airflow\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/learning.workfall.com\/learning\/blog\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Easily build ETL Pipeline using Python and Airflow\"}]},{\"@type\":\"Article\",\"@id\":\"https:\/\/learning.workfall.com\/learning\/blog\/how-to-easily-build-etl-pipeline-using-python-and-airflow\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/learning.workfall.com\/learning\/blog\/how-to-easily-build-etl-pipeline-using-python-and-airflow\/#webpage\"},\"author\":{\"@id\":\"https:\/\/learning.workfall.com\/learning\/blog\/#\/schema\/person\/cab8236044692bc5b27606b13167794a\"},\"headline\":\"Easily build ETL Pipeline using Python and Airflow\",\"datePublished\":\"2022-08-16T11:08:17+00:00\",\"dateModified\":\"2025-09-30T04:54:13+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/learning.workfall.com\/learning\/blog\/how-to-easily-build-etl-pipeline-using-python-and-airflow\/#webpage\"},\"wordCount\":1045,\"publisher\":{\"@id\":\"https:\/\/learning.workfall.com\/learning\/blog\/#organization\"},\"image\":{\"@id\":\"https:\/\/learning.workfall.com\/learning\/blog\/how-to-easily-build-etl-pipeline-using-python-and-airflow\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/learning.workfall.com\/learning\/blog\/wp-content\/uploads\/2022\/08\/Cover-Images_Part2-1-2.png\",\"keywords\":[\"airflow\",\"apacheairflow\",\"datapipeline\",\"ETL\",\"python\",\"workfall\"],\"articleSection\":[\"Backend Development\"],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\/\/learning.workfall.com\/learning\/blog\/#\/schema\/person\/cab8236044692bc5b27606b13167794a\",\"name\":\"Workfall\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/learning.workfall.com\/learning\/blog\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/learning.workfall.com\/learning\/blog\/wp-content\/uploads\/2023\/09\/avatar_user_1_1693914404-96x96.png\",\"contentUrl\":\"https:\/\/learning.workfall.com\/learning\/blog\/wp-content\/uploads\/2023\/09\/avatar_user_1_1693914404-96x96.png\",\"caption\":\"Workfall\"},\"sameAs\":[\"https:\/\/www.workfall.com\"]}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Easily build ETL Pipeline using Python and Airflow - The Workfall Blog","description":"Apache Airflow is an open-source workflow management platform for authoring, scheduling, and monitoring workflows.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/learning.workfall.com\/learning\/blog\/how-to-easily-build-etl-pipeline-using-python-and-airflow\/","og_locale":"en_US","og_type":"article","og_title":"Easily build ETL Pipeline using Python and Airflow - The Workfall Blog","og_description":"Apache Airflow is an open-source workflow management platform for authoring, scheduling, and monitoring workflows.","og_url":"https:\/\/learning.workfall.com\/learning\/blog\/how-to-easily-build-etl-pipeline-using-python-and-airflow\/","og_site_name":"The Workfall Blog","article_publisher":"https:\/\/facebook.com\/workfall","article_published_time":"2022-08-16T11:08:17+00:00","article_modified_time":"2025-09-30T04:54:13+00:00","og_image":[{"width":1200,"height":628,"url":"https:\/\/18-141-20-153.plesk.page\/learning\/blog\/wp-content\/uploads\/2022\/08\/Cover-Images_Part2-1-2.png","type":"image\/png"}],"twitter_card":"summary_large_image","twitter_creator":"@workfall","twitter_site":"@workfall","twitter_misc":{"Written by":"Workfall","Est. reading time":"8 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Organization","@id":"https:\/\/learning.workfall.com\/learning\/blog\/#organization","name":"Workfall - Hire #Kickass Coders On Demand","url":"https:\/\/learning.workfall.com\/learning\/blog\/","sameAs":["https:\/\/www.instagram.com\/workfall\/","https:\/\/www.linkedin.com\/company\/workfall\/","https:\/\/facebook.com\/workfall","https:\/\/twitter.com\/workfall"],"logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/learning.workfall.com\/learning\/blog\/#\/schema\/logo\/image\/","url":"https:\/\/i1.wp.com\/18.141.20.153\/learning\/blog\/wp-content\/uploads\/2021\/10\/cropped-WF_logo.png?fit=400%2C400","contentUrl":"https:\/\/i1.wp.com\/18.141.20.153\/learning\/blog\/wp-content\/uploads\/2021\/10\/cropped-WF_logo.png?fit=400%2C400","width":400,"height":400,"caption":"Workfall - Hire #Kickass Coders On Demand"},"image":{"@id":"https:\/\/learning.workfall.com\/learning\/blog\/#\/schema\/logo\/image\/"}},{"@type":"WebSite","@id":"https:\/\/learning.workfall.com\/learning\/blog\/#website","url":"https:\/\/learning.workfall.com\/learning\/blog\/","name":"The Workfall Blog","description":"#Tech #Remote #Jobs","publisher":{"@id":"https:\/\/learning.workfall.com\/learning\/blog\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/learning.workfall.com\/learning\/blog\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"en-US"},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/learning.workfall.com\/learning\/blog\/how-to-easily-build-etl-pipeline-using-python-and-airflow\/#primaryimage","url":"https:\/\/learning.workfall.com\/learning\/blog\/wp-content\/uploads\/2022\/08\/Cover-Images_Part2-1-2.png","contentUrl":"https:\/\/learning.workfall.com\/learning\/blog\/wp-content\/uploads\/2022\/08\/Cover-Images_Part2-1-2.png","width":1200,"height":628,"caption":"ETL Pipeline using Python and Airflow"},{"@type":"WebPage","@id":"https:\/\/learning.workfall.com\/learning\/blog\/how-to-easily-build-etl-pipeline-using-python-and-airflow\/#webpage","url":"https:\/\/learning.workfall.com\/learning\/blog\/how-to-easily-build-etl-pipeline-using-python-and-airflow\/","name":"Easily build ETL Pipeline using Python and Airflow - The Workfall Blog","isPartOf":{"@id":"https:\/\/learning.workfall.com\/learning\/blog\/#website"},"primaryImageOfPage":{"@id":"https:\/\/learning.workfall.com\/learning\/blog\/how-to-easily-build-etl-pipeline-using-python-and-airflow\/#primaryimage"},"datePublished":"2022-08-16T11:08:17+00:00","dateModified":"2025-09-30T04:54:13+00:00","description":"Apache Airflow is an open-source workflow management platform for authoring, scheduling, and monitoring workflows.","breadcrumb":{"@id":"https:\/\/learning.workfall.com\/learning\/blog\/how-to-easily-build-etl-pipeline-using-python-and-airflow\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/learning.workfall.com\/learning\/blog\/how-to-easily-build-etl-pipeline-using-python-and-airflow\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/learning.workfall.com\/learning\/blog\/how-to-easily-build-etl-pipeline-using-python-and-airflow\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/learning.workfall.com\/learning\/blog\/"},{"@type":"ListItem","position":2,"name":"Easily build ETL Pipeline using Python and Airflow"}]},{"@type":"Article","@id":"https:\/\/learning.workfall.com\/learning\/blog\/how-to-easily-build-etl-pipeline-using-python-and-airflow\/#article","isPartOf":{"@id":"https:\/\/learning.workfall.com\/learning\/blog\/how-to-easily-build-etl-pipeline-using-python-and-airflow\/#webpage"},"author":{"@id":"https:\/\/learning.workfall.com\/learning\/blog\/#\/schema\/person\/cab8236044692bc5b27606b13167794a"},"headline":"Easily build ETL Pipeline using Python and Airflow","datePublished":"2022-08-16T11:08:17+00:00","dateModified":"2025-09-30T04:54:13+00:00","mainEntityOfPage":{"@id":"https:\/\/learning.workfall.com\/learning\/blog\/how-to-easily-build-etl-pipeline-using-python-and-airflow\/#webpage"},"wordCount":1045,"publisher":{"@id":"https:\/\/learning.workfall.com\/learning\/blog\/#organization"},"image":{"@id":"https:\/\/learning.workfall.com\/learning\/blog\/how-to-easily-build-etl-pipeline-using-python-and-airflow\/#primaryimage"},"thumbnailUrl":"https:\/\/learning.workfall.com\/learning\/blog\/wp-content\/uploads\/2022\/08\/Cover-Images_Part2-1-2.png","keywords":["airflow","apacheairflow","datapipeline","ETL","python","workfall"],"articleSection":["Backend Development"],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/learning.workfall.com\/learning\/blog\/#\/schema\/person\/cab8236044692bc5b27606b13167794a","name":"Workfall","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/learning.workfall.com\/learning\/blog\/#\/schema\/person\/image\/","url":"https:\/\/learning.workfall.com\/learning\/blog\/wp-content\/uploads\/2023\/09\/avatar_user_1_1693914404-96x96.png","contentUrl":"https:\/\/learning.workfall.com\/learning\/blog\/wp-content\/uploads\/2023\/09\/avatar_user_1_1693914404-96x96.png","caption":"Workfall"},"sameAs":["https:\/\/www.workfall.com"]}]}},"jetpack_featured_media_url":"https:\/\/learning.workfall.com\/learning\/blog\/wp-content\/uploads\/2022\/08\/Cover-Images_Part2-1-2.png","jetpack-related-posts":[{"id":1498,"url":"https:\/\/learning.workfall.com\/learning\/blog\/how-to-etl-api-data-to-aws-s3-bucket-using-apache-airflow\/","url_meta":{"origin":1126,"position":0},"title":"How to ETL API data to AWS S3 Bucket using Apache Airflow?","date":"November 1, 2022","format":false,"excerpt":"2.5 quintillion bytes of data are produced every day with 90% of it generated solely in the last 2 years (Source: Forbes). Data is pulled, cleaned, transfigured & then presented for analytical purposes & put to use in thousands of applications to fulfill consumer needs & more. While generating insights\u2026","rel":"","context":"In &quot;AWS Cloud Computing&quot;","img":{"alt_text":"How to ETL API data to AWS S3 Bucket using Apache Airflow?","src":"https:\/\/i0.wp.com\/learning.workfall.com\/learning\/blog\/wp-content\/uploads\/2022\/11\/Cover-Images_Part2-2.png?resize=350%2C200&ssl=1","width":350,"height":200},"classes":[]},{"id":2408,"url":"https:\/\/learning.workfall.com\/learning\/blog\/how-to-simplify-data-pipelines-with-dbt-and-airflow\/","url_meta":{"origin":1126,"position":1},"title":"How to Simplify Data Pipelines with DBT and Airflow?","date":"August 14, 2023","format":false,"excerpt":"In today's data-driven world, efficient data pipelines have become the backbone of successful organizations. These pipelines ensure that data flows smoothly from various sources to its intended destinations, enabling businesses to make informed decisions and gain valuable insights. Two powerful tools that have emerged to simplify the management of data\u2026","rel":"","context":"In &quot;Backend Development&quot;","img":{"alt_text":"How To Simplify Data Pipelines With DBT And Airflow?","src":"https:\/\/i2.wp.com\/learning.workfall.com\/learning\/blog\/wp-content\/uploads\/2023\/08\/Cover-Images_Part2-1-2.png?resize=350%2C200&ssl=1","width":350,"height":200},"classes":[]},{"id":2369,"url":"https:\/\/learning.workfall.com\/learning\/blog\/unleashing-the-power-of-cdc-with-snowflake\/","url_meta":{"origin":1126,"position":2},"title":"Unleashing the Power of CDC With Snowflake","date":"June 12, 2023","format":false,"excerpt":"Picture a vibrant landscape of bustling tech hubs, where businesses thrive and decisions are made in the blink of an eye. In this dynamic realm of data engineering, a monumental challenge takes centre stage: efficiently managing the ever-changing tides of real-time data. Data, the lifeblood of organisations, holds the key\u2026","rel":"","context":"In &quot;Backend Development&quot;","img":{"alt_text":"Unleashing the Power of CDC With Snowflake","src":"https:\/\/i0.wp.com\/learning.workfall.com\/learning\/blog\/wp-content\/uploads\/2023\/06\/Cover-Images_Part2-1.png?resize=350%2C200&ssl=1","width":350,"height":200},"classes":[]},{"id":1348,"url":"https:\/\/learning.workfall.com\/learning\/blog\/how-to-connect-snowflake-with-python-and-execute-queries\/","url_meta":{"origin":1126,"position":3},"title":"Connect Snowflake with Python and execute queries","date":"September 27, 2022","format":false,"excerpt":"Snowflake cloud data warehouse is a buzzing trend in managing data these days as it has advantages like cost-effectiveness, auto-scaling, easy-to-transform data, etc. over traditional data warehouses. It is greatly assisting organizations in terms of its critical role in ELT (Extract-Load-Transform). Python is a very popular programming language that is\u2026","rel":"","context":"In &quot;Backend Development&quot;","img":{"alt_text":"Connect Snowflake with Python and execute queries","src":"https:\/\/i0.wp.com\/learning.workfall.com\/learning\/blog\/wp-content\/uploads\/2022\/09\/Cover-Images_Part2-1-3.png?resize=350%2C200&ssl=1","width":350,"height":200},"classes":[]},{"id":2388,"url":"https:\/\/learning.workfall.com\/learning\/blog\/how-to-read-and-write-in-google-spreadsheet-using-python-and-sheety-api\/","url_meta":{"origin":1126,"position":4},"title":"How to Read and Write In Google Spreadsheet Using Python and Sheety API?","date":"July 25, 2023","format":false,"excerpt":"Tired of manual data entry in Google Spreadsheets? Discover a simple and efficient way to automate your data handling using Python and Sheety API. In this blog, we'll demonstrate step-by-step the process of reading and writing data in Google Sheets, empowering you to effortlessly manage your data with the power\u2026","rel":"","context":"In &quot;Backend Development&quot;","img":{"alt_text":"Read and Write In Google Spreadsheet Using Python and Sheety API","src":"https:\/\/i0.wp.com\/learning.workfall.com\/learning\/blog\/wp-content\/uploads\/2023\/07\/Cover-Images_Part2-1-3.png?resize=350%2C200&ssl=1","width":350,"height":200},"classes":[]},{"id":541,"url":"https:\/\/learning.workfall.com\/learning\/blog\/how-to-build-a-serverless-event-driven-workflow-with-aws-glue-and-amazon-eventbridgepart-1\/","url_meta":{"origin":1126,"position":5},"title":"How to build a serverless event-driven workflow with AWS Glue and Amazon EventBridge(Part 1)?","date":"November 10, 2021","format":false,"excerpt":"Have you ever wondered how huge IT companies construct their ETL pipelines for production? Are you curious about how TBs and ZBs of data are effortlessly captured and rapidly processed to a database or other storage for data scientists and analysts to use? The answer is the serverless data integration\u2026","rel":"","context":"In &quot;AWS Cloud Computing&quot;","img":{"alt_text":"AWS Glue","src":"https:\/\/i1.wp.com\/learning.workfall.com\/learning\/blog\/wp-content\/uploads\/2021\/11\/Glue.png?resize=350%2C200&ssl=1","width":350,"height":200},"classes":[]}],"_links":{"self":[{"href":"https:\/\/learning.workfall.com\/learning\/blog\/wp-json\/wp\/v2\/posts\/1126"}],"collection":[{"href":"https:\/\/learning.workfall.com\/learning\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/learning.workfall.com\/learning\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/learning.workfall.com\/learning\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/learning.workfall.com\/learning\/blog\/wp-json\/wp\/v2\/comments?post=1126"}],"version-history":[{"count":4,"href":"https:\/\/learning.workfall.com\/learning\/blog\/wp-json\/wp\/v2\/posts\/1126\/revisions"}],"predecessor-version":[{"id":3041,"href":"https:\/\/learning.workfall.com\/learning\/blog\/wp-json\/wp\/v2\/posts\/1126\/revisions\/3041"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/learning.workfall.com\/learning\/blog\/wp-json\/wp\/v2\/media\/1127"}],"wp:attachment":[{"href":"https:\/\/learning.workfall.com\/learning\/blog\/wp-json\/wp\/v2\/media?parent=1126"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/learning.workfall.com\/learning\/blog\/wp-json\/wp\/v2\/categories?post=1126"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/learning.workfall.com\/learning\/blog\/wp-json\/wp\/v2\/tags?post=1126"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}