Job Title: Senior / Lead Agentic AI Engineer – LLM Evaluation & Observability
Location: Dallas, TX (Hybrid) 
Duration: Long‑Term 
Experience: 8+ years 
Interview: Client round, in‑person 

Important: This is a hands‑on engineering role. Candidates with practical experience building and evaluating AI applications are preferred over those whose experience is limited to AI strategy, architecture documentation, conventional QA automation, or using AI coding assistants. 

We seek a Senior/Lead AI Engineer with experience building, orchestrating, monitoring, and evaluating production‑grade Generative AI and Agentic AI applications. The ideal candidate is expert in Amazon Bedrock, AgentCore, LangGraph, LangChain, RAG, LLM observability, and evaluation frameworks. The role focuses on creating reliable, scalable, production‑ready AI applications, emphasizing agent orchestration, AI testing, LLM evaluation, and continuous quality monitoring. 

The candidate will collaborate with Engineering, QA, and platform teams to establish best practices for building, testing, monitoring, and improving AI‑powered applications. The testing focus is on GenAI evaluation, RAG pipeline assessment, agent behavior validation, LLM quality benchmarking, and automated evaluation frameworks, not conventional UI test automation. 

Key Responsibilities  

1. Agentic AI & GenAI Engineering
   - Design, develop, and deploy production‑grade agentic and generative AI applications. 
   - Build and orchestrate AI agents using LangGraph, LangChain, Amazon Bedrock, and AgentCore. 
   - Implement multi‑agent workflows, tool calling, state management, memory, and orchestration. 
   - Create scalable LLM‑powered applications and reusable AI components. 
   - Deploy agentic workflows with Amazon Bedrock AgentCore, handling runtime management, tool integration, and execution orchestration. 

2. RAG & LLM Application Development 
   - Design and implement Retrieval‑Augmented Generation (RAG) solutions. 
   - Develop pipelines for document ingestion, chunking, embedding, retrieval, reranking, and context management. 
   - Optimize retrieval accuracy, response quality, latency, and cost. 
   - Apply grounding, hallucination reduction, and context‑relevance techniques. 

3. LLM Evaluation & AI Testing
   - Build automated evaluation frameworks with RAGAS, DeepEval, and similar tools. 
   - Evaluate RAG pipelines using metrics such as Faithfulness, Answer Relevancy, Context Precision, Context Recall, and Answer Correctness. 
   - Implement LLM‑as‑a‑Judge techniques to assess generated responses, retrieval quality, agent behavior, and workflow execution. 
   - Establish evaluation datasets, benchmarks, and regression‑testing pipelines for continuous quality assessment. 

   3a. Agent Evaluation and Observability
   - Implement end‑to‑end tracing of multi‑agent workflows, including tool invocation, agent handoffs, execution paths, and failure handling. 
   - Establish observability practices to detect hallucinations, reasoning failures, tool‑calling errors, and execution inconsistencies. 

4. AI Observability & Monitoring
   - Implement tracing, monitoring, and observability with OpenTelemetry, LangSmith, and Langfuse. 
   - Track LLM requests, agent execution flows, tool calls, latency, token consumption, errors, and performance. 
   - Develop debugging mechanisms for complex multi‑step agentic workflows. 
   - Create monitoring practices to detect performance degradation and operational issues. 

5. AWS AI Platform & Production Engineering
   - Deploy AI applications using Amazon Bedrock and Bedrock AgentCore. 
   - Integrate foundation models, agent frameworks, APIs, and enterprise data sources. 
   - Apply production‑readiness practices covering scalability, reliability, security, and performance. 
   - Support CI/CD integration and automated evaluation of AI applications. 
   - Optimize model selection, inference costs, latency, and resource utilization. 
 
6. Technical Leadership & Collaboration
   - Provide technical guidance to engineering and QA teams. 
   - Establish reusable frameworks, implementation patterns, and best practices. 
   - Collaborate with architecture, DevOps, QA, and engineering teams. 
   - Participate in design discussions, code reviews, proof‑of‑concept development, and implementation. 
   - Mentor engineers and help teams adopt effective AI development and testing practices. 

Required Technical Skills 

Agentic AI & Orchestration
- LangGraph 
- LangChain 
- Amazon Bedrock 
- Amazon Bedrock AgentCore 
- AI agents, multi‑agent orchestration, tool calling, workflow management 

Generative AI & RAG
- Large Language Models (LLMs) 
- Retrieval‑Augmented Generation (RAG) 
- Embeddings and vector search 
- Prompt engineering and context management 
- LLM application development 

LLM Evaluation & Testing
- DeepEval 
- LLM‑as‑a‑Judge techniques 
- Evaluation metrics and benchmarking 
- Automated evaluation and regression testing 
- Assessment of hallucination, faithfulness, relevance, and response quality 
- Hands‑on experience with RAGAS, DeepEval, or equivalent frameworks 
- Strong understanding of RAG evaluation metrics (Faithfulness, Answer Relevancy, Context Precision, Context Recall, Answer Correctness) 
- Experience building automated evaluation pipelines for RAG‑based applications 

AI Observability & Monitoring
- OpenTelemetry 
- LangSmith 
- Langfuse 
- LLM tracing, debugging, and performance monitoring 

Production Engineering 
- Deploying and operating production‑grade GenAI applications 
- API integration and backend engineering 
- CI/CD and automated testing integration 
- Scalability, reliability, security, and performance optimization
- Strong hands‑on programming experience in Python.
- Experience developing REST APIs and integrating LLM frameworks with enterprise applications.
- Familiarity with asynchronous programming, API integrations, and backend development. 

Preferred Qualifications 
- 8+ years of software engineering or IT experience. 
- Demonstrated hands‑on experience building and deploying GenAI and Agentic AI applications. 
- Recent experience in LLM engineering, agent orchestration, and AI evaluation. 
- Experience with enterprise‑scale AI implementations. 
- Familiarity with AWS cloud services and enterprise AI security practices. 
- Experience establishing AI quality standards and evaluation methodologies. 
- Strong problem‑solving, communication, and technical collaboration skills. 

What Success Looks Like  
- Development and deployment of production‑grade agentic AI applications.  
- Reliable orchestration and execution of AI agents and workflows.  
- Effective LLM evaluation and automated AI testing frameworks.  
- Improved AI application accuracy, reliability, observability, and performance.  
- Reusable engineering practices for building and maintaining enterprise GenAI solutions.