Senior Software Engineer, Assistant Engineering

Airbnb was born in 2007 when two hosts welcomed three guests to their San Francisco home, and has since grown to over 5 million hosts who have welcomed over 2 billion guest arrivals in almost every country across the globe. Every day, hosts offer unique stays and experiences that make it possible for guests to connect with communities in a more authentic way.

The CommunityYouWill Join:

Artificial Intelligence is at the heart of Airbnb’s product strategy. The Conversational Product team within Assistant Engineering is responsible for building and scaling Airbnb’s AI Assistant experience, expanding it beyond customer support to empower product domains across Airbnb.

Within the Conversational Product team, the AI Assistant Product Evaluation team builds the evaluation systems, data foundations, and observability tools that help Airbnb develop reliable, high-quality agentic AI products. Our work enables teams to measure AI product quality, understand model and agent behavior, identify regressions, and accelerate the feedback loop between model development and product iteration.

The Difference You Will Make:

As a Senior Software Engineer on the AI Assistant Product Evaluation team, you will design and build scalable data systems and evaluation platforms for Airbnb’s agentic AI products. You will work at the intersection of backend engineering, data systems, AI evaluation, and product quality.

In this role, you will help define how Airbnb measures, monitors, and improves the quality of agentic AI products at scale. You will build data-rich systems that transform product interactions, model outputs, human feedback, and evaluation results into reliable signals for product and model development. Your work will help teams move faster with greater confidence by making AI quality measurable, debuggable, and actionable across the development lifecycle.

You will also partner closely with modeling teams, product managers, data scientists, and operations teams to turn ambiguous AI evaluation needs into robust systems that help Airbnb ship trustworthy AI experiences.

A Typical Day:

• Design and productionize scalable data systems that support AI evaluation, metric computation, observability, and feedback loops for agentic AI products.
• Build data models, schemas, and processing pipelines for agentic AI interactions, supporting reliable logging, retrieval, metric computation, and long-term evaluation dataset management.
• Work closely with Core Modeling engineers to understand pain points in the LLM evaluation process, and develop LLM-as-a-judge solutions and data pipelines to address metric-related challenges in a scalable and efficient way.
• Collabo