(nmedia/Shutterstock)
Organizations have been engaged on changing into extra data-driven for a few years at this level, with combined outcomes. We perceive that the worth of knowledge is plain. Nonetheless, it has now turn out to be extra essential than ever for organizations to operate as data-driven entities. The urgency of this problem is underscored by the emergence of AI and ML.
The 2024 State of Apache Airflow helps us perceive the expansion, utilization, and person group dynamics of navigating the complexities of knowledge orchestration, information pipelines, and information supply.
The info for the report was compiled by surveying 281 professionals who have been immediately or not directly liable for managing their group’s Airflow cases. This included information engineers, software program engineers, DevOps engineers, and options architects.
The report highlights three important developments – speedy progress in Airflow adoption, growing reliability of Airflow for mission-critical information supply, and Airflow changing into the usual for AI pipelines.
Apache Airflow is an open-source platform for creating, scheduling, and monitoring information and computing workflows. With its graphical UI, integration with main cloud platforms, and ease of use, Airflow is changing into a preferred instrument for orchestrating workflows or pipelines.
One of many key findings of the report is that AL and ML services by Airflow spiked by 24 p.c yr over yr. The highest functions for Airflow included the inner analytics dashboard, GenAI, and ML. There has additionally been a 68 p.c improve in Airflow downloads yr on yr with over 165.7 million downloads. Airflow contributors outpace Apache tasks Spark and Kafka with 2.8K contributors
Whereas the AI market is projected to proceed rising, there are some considerations, together with fragmented instrument stacks, siloed instruments, and a looming menace of stricter compliance and operational necessities.
In line with Apache, a rising variety of companies are addressing these considerations utilizing Astronomer – the unified information platform constructed on Apache Airflow service. Astronomer permits organizations to make sure information privateness, and unify information throughout the cloud, staff, and deployments. It additionally helps speed up improvement by means of seamless integrations.
(Andrey Suslov/Shutterstock)
The report reveals that Apache Airflow has advanced from a distinct segment instrument to a mission-critical asset. Two-thirds (67 p.c) of firms have greater than 6 individuals utilizing Airflow. This means a rising pattern of knowledge groups utilizing Airflow to ship enterprise outcomes.
It isn’t simply the variety of individuals utilizing Airflow that’s spectacular, the frequency of interplay can also be excessive. The report exhibits that 55 p.c of customers work together with Airflow each day, with one other 26 p.c utilizing it at the least as soon as per week. It seems that firms are rising of their confidence in Airflow and their imaginative and prescient of what the platform can accomplish.
Round half (46 p.c) of respondents shared that they really feel Airflows is a “essential” instrument for his or her enterprise operations. Outages and the necessity for scalability have been the highest two considerations of the respondents. The respondents additionally shared that probably the most regarding features of knowledge downtime embody inside programs being negatively affected, throttling of knowledge staff productiveness, and points with revenue-generation apps or providers.
The outcomes of the survey spotlight how Apache Airflow meets the growing challenges of knowledge orchestration, scaling, and facilitation of finest practices. Maybe the best influence of Airflow has been in AI pipeline supply. The momentum positions Airflows as one of many main information orchestration platforms for AI improvement and deployment.
Associated Objects
How Airflow 2.8 Makes Constructing and Operating Information Pipelines Simpler
Astronomer’s Apache Airflow-Powered Astro Outperforms Rivals with 30M Month-to-month Downloads
AWS Pronounces Common Availability of Amazon Managed Workflows for Apache Airflow