Model Behavior Engineer

About the Role

You'll own the quality bar for Notion's AI products, working across engineering, product, and design teams to define standards, measure progress, and drive improvements that ensure reliable AI experiences for millions of users. This role sits within the AI engineering team and combines operations, strategy, and product thinking—you'll spend your time on context engineering, evaluation system design, and data analysis rather than traditional software development.

Responsibilities

• Design, test, and iterate on system prompts, tool prompts, and context strategies to shape how Notion's AI products behave and respond to user needs
• Analyze production data—transcripts, logs, and user feedback—to reproduce issues, identify root causes, and translate problems into actionable statements
• Design evaluation strategies, build datasets, and run assessments to track quality over time and surface issues before users encounter them
• Evaluate and launch models from OpenAI, Anthropic, Google, and other labs; benchmark across quality, latency, cost, and edge cases to inform Notion's model strategy
• Work embedded with engineering and product teams to surface critical quality issues, own the quality narrative, and guide prioritization decisions
• Manage AI observability and evaluation platforms; develop playbooks and tools that enable teams across Notion to build AI products effectively

Requirements

• Driver mentality—you treat problems as yours and take action to resolve them, even when you didn't cause them
• Curiosity about LLM capabilities and how AI products perform in real-world contexts
• Analytical instinct and ability to find signal in noisy production data
• Comfortable working with data via SQL, coding agents, or similar tools to self-serve insights from large datasets
• Clear communication skills—you can explain complex technical issues simply
• Experience with LLMs, prompting, or AI products

Nice to Have

• Background in engineering, product, data science, research, or consulting
• History of building something independently—side project, startup, tool—to solve a problem

Benefits

• Competitive cash compensation and equity
• Base salary range for New York-based roles: $98,000–$140,000 per year (based on experience, expertise, and role scope)