Astronomer opens AMA on Airflow and running AI work
The Astronomer team opened a Reddit AMA about Airflow, orchestration, and running AI workload in production. The post says many teams are trying to put AI work into real systems without creating a mess. Astronomer also mentions Airflow 3 features such as DAG versioning, human-in-the-loop, event-driven scheduling, UI refresh, and backfills. The team says people can ask about Otto, its new data engineering agent for Airflow.
Key points
- Astronomer is hosting an AMA in r/dataengineering.
- The AMA covers Airflow, orchestration, and AI workload in production.
- The post says teams are trying to run AI work without creating operational mess.
- Airflow 3 features mentioned include DAG versioning, human-in-the-loop, event-driven scheduling, and backfills.
- Astronomer says people can ask about Otto, its data engineering agent for Airflow.
Quick term guide
- orchestration
- Coordinating multiple AI agents or steps to run in a specific order or in parallel to complete a task
- AI workload
- A set of tasks that use an AI model or process AI results.
- production
- The live version of a service that real users use.
- DAG versioning
- A way to track versions of a workflow as it changes.
- human-in-the-loop
- A design pattern where a human provides input or confirmation within an automated process.
- AI agents
- AI agents are AI tools that can carry out steps toward a goal, not just answer once.
- model call
- One request sent to an AI model to get an answer.
- token cost
- The money or usage spent when sending text to an AI model and getting text back.