IT & Software

AI Agent Evaluation Engineer

JoinStartup

Winnipeg · Manitoba · Canada

Share this job: X Facebook WhatsApp Save job : AI Agent Evaluation Engineer
Define conversational AI quality remotely from Canada or the United States using Python, LLM APIs, and rigorous evaluation systems. As Cresta’s mid-level AI Agent Evaluation Engineer, you will own quality measurement for production LLM-based conversational agents. You will combine conversation analysis, annotation, rubric design, and statistical sampling to create an accurate picture of agent performance. Your work will support Cresta’s AI Receptionist, which handles millions of conversations monthly across hundreds of customers. Key Responsibilities:

Define rubrics across adherence, outcome, and conversation experience Review production conversations and classify failure modes weekly Build automated judges, golden sets, and regression suites Calibrate judges against human labels and measure agreement Report prioritized quality findings to engineering and product owners Requirements:

Three or more years in software engineering, applied ML, or research One to two years working with production LLM systems Python programming skills and experience with LLM APIs Experience building end-to-end evaluation systems Degree in a related field or equivalent practical experience Bring Python, annotation, rubric design, and LLM evaluation expertise to measure Cresta agent quality accurately.

#J-18808-Ljbffr

Reference: WJ-3875_16356394 · Posted 17 Nov 2026, 01:42 AM

Find more on GigBlow

This role is listed on GigBlow for discovery and search. Hiring decisions and applications are handled by the employer or their chosen application system.

AI Agent jobs in other states

Browse jobs