#Google cloud composer airflow 2.0 drivers

Of course, there were other issues such as adding in drivers for older databases, and other plugins that were less straightforward. I found this out first hand as I migrated a project over to Cloud Composer. What Cloud Composer and MWAA give you in ease of set-up they take away in the difficulty of editing much more than your Python libraries. What You Lose In Ease Of Set-Up You Gain In Customizability Overall, what you gain in ease of set-up you will lose in customizability. But personally, it’s still far easier than trying to set up your own VMs to scale. It does take about 20-30 minutes to wait for the entire system to spin up.Īstronomer takes a little more lift. It’s just a few clicks and you have a fully managed Airflow instance that is easy to scale at your fingertips. Setting up MWAA or Cloud Composer is easier than setting up Astronomer. So let’s dive into some of the benefits and challenges you will face with all of these solutions. Where you will notice Astronomer shines is as you set up more complex jobs and need more flexibility. They can help set up a POC as well as an MVP without needing to set up too many external logistical components or agreements. The truth is if you’re just getting started. I want to start with the fact that this article may come across as very positive towards one specific product. If your team is small or just needs to prioritize other work, then managed versions of Airflow are often a good choice.īut which managed Airflow service works best for you?Ĭomparing Composer And MWAA to Astronomer On top of that, trying to manage your Airflow instance generally means you need to have a meta database like Postgres, Redis, a logging location like S3, and several other components to manage your workers as they scale.Įach of these components increases technical complexity and the opportunity for failures. We already deal with daily pipeline failures due to the engineering team updating that one column or removing a table altogether without telling us. Maintenance and management of services and databases often hold many data teams back. In this article, I want to discuss some of the benefits and limitations of Cloud Composer and MWAA as well as discuss why Astronomer might be a better fit for many companies. But there is a third option called Astronomer. Of course, both Cloud Composer and MWAA are services provided by two of the larger cloud providers. They help reduce a lot of issues such as scaling workloads across workers and managing all the other various components required to run Airflow at scale. These are two great options when it comes to starting your first Airflow project. Over the last 3 months, I have taken on two different migrations that involved taking companies from manually managing Airflow VMs to going over to using Cloud Composer and MWAA (Managed Workflows For Apache Airflow).

Google cloud composer airflow 2.0