Software Engineer III, AI/ML, AI Data

In this role you will help invent and prototype evaluation methods for model behavior and failure modes, translating research insights into robust benchmarks and data pipelines. You work at the intersection of research and software engineering, running empirical analyses and building scalable infrastructure. You collaborate with researchers and engineers across Google DeepMind and AI Data to measure frontier model capabilities in code, math, and specialized domains. This is a hands-on position that shapes how we assess and improve AI evaluation at scale.

• Research and prototype novel approaches to analyze model behavior and failure patterns
• Conduct targeted empirical investigations to inform benchmark design and trust in evaluation
• Implement, test, and refine benchmark suites for complex agentic reasoning across code, math, and specialized domains
• Develop and maintain pipelines that transform benchmark/ sandbox execution logs, agent traces, and runtime artifacts into high-fidelity evaluation datasets and training data
• Partner with research scientists, senior engineers, and cross-functional teams to translate research insights into evaluation infrastructure
• Run empirical analyses, build programmatic verifiers, and help measure frontier model capabilities across domains

• Bachelor’s degree or equivalent practical experience
• 2 years of software development experience, or 1 year with advanced degree
• 1 year of experience with end-to-end Machine Learning (deployment, evaluation, data processing, debugging)
• Experience with GenAI techniques (LLMs, Multi-Modal, Large Vision Models) or GenAI concepts (language modeling, computer vision)
• Experience in Generative AI
• Experience in Python

• collaboration with cross-functional teams
• versatility and leadership qualities
• ability to translate research into practical infrastructure
• GenAI techniques
• end-to-end ML
• Python