Senior AI Quality Engineer
Roofr is seeking a Senior AI Quality Engineer to lead the development and implementation of evaluation frameworks, testing standards, and quality gates for AI integrations across its product suite. This role is embedded within the AI Platform team and involves collaboration with various engineering teams to ensure the delivery of high-quality AI features. The position is remote, based in Canada.
Key responsibilities include defining AI testing standards, building and owning Roofr's LLM evaluation framework using tools like Promptfoo, DeepEval, and Braintrust, integrating quality gates into CI/CD pipelines, and designing human-in-the-loop review processes for AI outputs. The role also involves coaching QA engineers and developers on AI evaluation patterns and staying updated with evolving AI quality techniques and tools.
Candidates should have 5–8 years of experience in software engineering or quality assurance, hands-on experience with evaluation frameworks for LLM-powered features, strong engineering fundamentals, experience integrating automated quality checks into CI/CD pipelines, familiarity with LLM APIs and agent frameworks, and experience designing human review workflows. Strong collaboration skills and the ability to operate in an early-stage environment are also essential.
The compensation range for this position is $140,000 - $160,000 CAD annually. Benefits include mandatory PTO during the first week of employment, one Friday off per month, a company-wide paid shutdown between Christmas and New Year's, flexible time off, employer-paid benefits, RRSP/401k match, generous parental leave policy, annual company retreats, learning and development opportunities, home office setup stipend, internet and phone allowance, and a remote-first culture.
Roofr fosters a customer-centric culture, emphasizing innovation and collaboration. The company is in a growth phase, offering significant opportunities for career advancement and the chance to make a substantial impact within the organization.