AI Test Manager

Job Title: AI Solutions Evaluation & Experience Manager – Contractor

Location: [Greenwood Village, CO]

Contract Duration: [6 months, with potential for extension]

Job Overview:

• We are seeking a highly technical AI Solutions Evaluation & Experience Manager to architect the framework that determines the real-world intelligence and accuracy of our AI and analytic products and perform ongoing solution testing and monitoring. In this role, you will move beyond traditional QA to implement LLM-as-a-Judge and SLM patterns to evaluate performance at scale.


• You will be responsible for managing a team that will calculate and interpret critical technical metrics, such as RAGas scores (Faithfulness, Answer Relevance), Perplexity, and Hallucination rates, to determine model accuracy.


• Additionally, you will analyze adoption and utilization m...