AI Solution and Platform Sr. Engineer

1. Observability Platform Engineering & OTEL Integration (25%)

• Design and implement OpenTelemetry (OTEL) instrumentation within one or two assigned agent frameworks or platforms — including custom exporters, span enrichers, semantic convention tagging, and distributed trace context propagation — with the ability to work independently from requirements through to production deployment.
• Build and maintain telemetry pipeline components (collectors, processors, exporters) that reliably route metrics, logs, traces, and semantic signals to observability backends — owning the full lifecycle of assigned pipeline components including testing, deployment, and on-call support.
• Contribute to the integration of OTEL with enterprise agentic platforms (Salesforce AgentForce, ServiceNow, Microsoft Agent 365, or internal frameworks) within the assigned scope — implementing instrumentation to architecture patterns established by the L11.
• Develop and maintain observability dashboards, alerting rules, and SLO/SLA definitions for assigned sub-domains — validating signal quality and tuning alert thresholds to achieve low false-positive rates.
• Participate in on-call rotations and production incident response — contributing to RCA documentation, runbook authoring, and post-incident improvement actions.
• Write comprehensive unit, integration, and end-to-end tests for all telemetry components owned; maintain >80% test coverage across assigned services and proactively identify gaps in existing coverage.

2. Safety, Security & Red Teaming Observability (15%)

• Implement safety-critical signal capture within assigned telemetry pipelines — building reliable instrumentation for guardrail trigger rates, policy violation events, adversarial detection flags, hallucination indicators, and trust boundary crossing alerts.
• Build observability components that support red team exercises — instrumenting assigned agent systems to capture adversarial test events, behavioral deviations, and attack surface signals in a measurable, repeatable way.
• Implement secure trace handling patterns within assigned pipelines — applying data masking, PII redaction, and audit-log retention configurations as specified by the security architecture.
• Contribute to the Security Observability Playbook — documenting assigned instrumentation patterns, updating escalation procedures based on observed incidents, and maintaining accuracy of the playbook sections within scope.
• Monitor agent-to-agent protocol traffic (A2A, UCP, AP2) within the assigned domain for anomalous patterns — flagging deviations for review in a timely manner with sufficient diagnostic context.

3. Responsible AI (RAI) & Governance Signal Instrumentation (10%)

• Implement RAI signal collectors within assigned agent workflows — building reliable pipelines that capture fairness indicators, bias detection outputs, explainability scores, and content safety classifications with validated data quality.
• Maintain RAI telemetry pipelines within scope — ensuring completeness, accuracy, and timeliness of governance signals that feed into compliance dashboards, and resolving data quality issues proactively.
• Ensure all AI decision traces within the assigned domain include required governance metadata and comply with retention policies — contributing to the audit-readiness of the observability platform.
• Identify and document RAI signal coverage gaps within the assigned scope — reporting findings to the L11 with sufficient detail to inform remediation planning.

• Build and maintain quality gate components within CI/CD pipelines for assigned agent services — implementing regression detection logic, performance degradation alerts, and SLA breach notifications using production observability data.
• Instrument and monitor Skill Evaluations (evals) across assigned Memory, Skills, and MCP harness components — collecting eval telemetry, tracking pass/fail trends over time, and alerting on regression thresholds with appropriate context.
• Implement continuous quality monitoring for post-go-live agentic solutions within scope — tracking agent success rates, tool-call fidelity, latency distributions, and user outcome proxies against defined baselines.
• Execute structured testing of new agent capabilities using standardized eval harnesses — documenting results clearly, flagging anomalies, and contributing findings to quality improvement cycles.
• Build and maintain automated quality reports and metric dashboards for assigned areas — ensuring stakeholders have timely, accurate visibility into agent behavioral quality and trend direction.

• Instrument agent memory operations within the assigned scope — building reliable monitoring of read/write latency, cache hit rates, memory staleness, and semantic drift across episodic, semantic, and working memory backends.
• Add trace instrumentation to MCP server interactions within assigned components — implementing OTEL semantic tagging for tool registrations, skill invocations, context injections, and result returns.
• Capture telemetry for self-evolving harness and RL system components as assigned — implementing signal capture for reward distributions, policy update events, environment state transitions, and convergence indicators.
• Monitor eval harness execution within assigned scope — building detection for flaky eval environments, setup failures, and result inconsistencies that could obscure real capability regressions.

• Write production-quality Python for assigned observability components — custom OTEL exporters, signal aggregators, data transformation pipelines, and anomaly detection logic — consistently meeting team engineering standards for code quality, testing, and documentation.
• Apply data science methods to assigned telemetry data — time-series analysis, statistical threshold tuning, distribution characterization — to improve signal accuracy and reduce alerting noise within the assigned domain.
• Contribute to shared Python SDK and library components — implementing well-tested, documented additions that improve OTEL onboarding experience for agent developers.
• Actively participate in code reviews — both receiving feedback from peers and contributing constructive technical review of peer engineers' pull requests within areas of expertise.

7. Agent Fleet, Physical AI & Multi-Modal Observability (5%)

• Implement telemetry for agent fleet coordination components as assigned — building signal capture for spawn/termination events, inter-agent message traces, load distribution metrics, and fleet-level health indicators.
• Contribute to observability instrumentation for physical AI or multi-modal pipelines within the assigned scope — focusing on latency, data quality, and reliability signals as directed by the L11 architecture.
• Document instrumentation patterns for fleet, physical AI, and multi-modal components — ensuring observability approaches are reproducible and transferable to other team members.

8. Agentic Marketplace, Registry & Agent Protocol Observability (5%)

• Instrument assigned Agentic Marketplace and Agent Registry components with usage telemetry — building signal capture for agent invocations, capability health, adoption patterns, and dependency relationships within scope.
• Implement protocol observability for assigned A2A, UCP, and AP2 communication flows — capturing message latency, error rates, retry patterns, and trust boundary events with sufficient granularity for incident diagnosis.
• Contribute to Marketplace Observability Dashboard development — building data connectors, metric calculations, and visualization components for assigned areas as directed.

9. Peer Collaboration, Standards Contribution & Continuous Learning (5%)

• Collaborate actively and constructively with peer Jr. AI Observability Architects — sharing technical knowledge, co-designing solutions to shared problems, and contributing to a high-quality, high-trust team environment.
• Contribute to engineering standards discussions — bringing informed technical perspectives on OTEL conventions, instrumentation patterns, and telemetry design decisions based on hands-on experience in assigned tracks.
• Participate fully in agile ceremonies — sprint planning, stand-ups, retrospectives — contributing accurate estimates, early identification of blockers, and transparent delivery status updates.
• Stay current with evolving OTEL specifications, agent communication protocols, AI safety research, and observability tooling — proactively applying new knowledge to improve the quality and coverage of assigned work.
• Contribute to internal documentation, engineering wikis, and instrumentation guides — ensuring that the approaches used in assigned tracks are clearly documented and accessible to the broader team.

• Bachelor's or Master's degree in Computer Science, Software Engineering, AI/ML, Data Science, or a related technical field.
• 10+ years of professional software and AI/ML engineering or platform engineering experience, with at least 2 years of hands-on observability, distributed systems monitoring, or telemetry pipeline development.
• Demonstrated experience delivering production-grade software end-to-end — from design through deployment and on-call operation — in a collaborative team environment.
• Experience working in or adjacent to AI/ML platform, data engineering, or cloud infrastructure roles; exposure to agentic AI systems or LLM pipelines is a strong plus.
• • Technical Proficiency: Implement monitoring for agent failure modes: tool-call failures, infinite loops, timeouts, hallucination risk signals, retrieval misses, and degraded response quality. Create alerts aligned to operational SLOs (availability, latency, tool reliability) and AI-specific indicators (cost spikes, loop bursts, retrieval anomalies). Support guardrail observability: policy blocks, content filtering events, and safety classifier outcomes (where applicable). Build onboarding automation (IaC, templates, CI checks) that makes observability “default-on” for all agentic services.
  • Problem-Solving: Ability to translate business challenges into technical solutions.
  • Collaboration Skills: Effective at working within cross-functional teams.
  • Agility: Flexibility to adapt to changing requirements and new technologies.
  • Communication Skills: Capable of explaining complex technical concepts to non-technical stakeholders.
  • Observability & OpenTelemetry: Solid hands-on proficiency with OpenTelemetry (OTEL) SDK instrumentation — custom exporters, collector configuration, semantic conventions, and distributed trace propagation. Able to independently instrument a service and validate signal quality end-to-end.
  • Python Engineering: Strong Python development skills — clean, well-tested, production-ready code. Familiarity with async patterns, type hints, testing frameworks (pytest), and CI/CD integration. Able to build and maintain Python-based telemetry tooling with minimal guidance.
  • Distributed Systems: Good working knowledge of microservices, event streaming (Kafka or equivalent), REST/gRPC API design, and containerized deployment (Docker, Kubernetes). Able to reason about distributed failure modes and their observability implications.
  • Cloud Platforms: Hands-on experience with at least one major cloud provider (Azure, AWS, or GCP) — managed services, IAM, storage, and cost awareness sufficient to make responsible deployment decisions.
  • Data Analysis Applied to Telemetry: Ability to query, analyze, and interpret time-series and log data using Grafana, Datadog, Prometheus, Splunk, or equivalent — including threshold tuning and basic statistical interpretation of signal distributions.
  • CI/CD & DevOps: Working experience with CI/CD pipelines, GitOps practices, automated testing, and infrastructure-as-code concepts sufficient to contribute to and extend existing pipeline configurations.
  • AI/ML Awareness: Familiarity with LLM-based workflows, agentic AI concepts, and common agent patterns (tool/function calling, RAG, memory, multi-step planning) — sufficient to understand observability requirements without needing deep ML expertise.
  • Safety & Security Fundamentals: Basic understanding of AI safety concepts (guardrails, policy enforcement, prompt injection) and data security practices (PII handling, access control, audit logging) as applied to telemetry systems.
  • Quality Engineering Basics: Familiarity with software quality concepts — regression detection, eval frameworks, test harnesses — and the ability to implement quality gate components within CI/CD pipelines using observability data.
  • RAI Awareness: Working knowledge of Responsible AI principles — fairness, explainability, bias — sufficient to implement signal capture pipelines for RAI governance requirements as specified.
  • Direct experience with agentic AI frameworks such as LangChain, LangGraph, AutoGen, Semantic Kernel, CrewAI, or Bedrock Agents.
  • Familiarity with MCP (Model Context Protocol), A2A, UCP, or AP2 agent communication protocols.
  • Exposure to reinforcement learning concepts, RL training infrastructure, or self-supervised learning pipelines.
  • Experience contributing to or consuming developer-facing Python SDKs or observability libraries.
  • Background with vector databases (Pinecone, Weaviate, pgvector) or semantic search in the context of RAG pipeline observability.
  • Contributions to open-source observability or AI tooling projects.
  Familiarity with AI safety frameworks, adversarial ML concepts, or red team tooling applied to LLM systems.