From unstable prototype to defined platform roles

A global engineering and networks company was building an LLM-powered support layer for industrial operations. In the lab, the prototype looked promising. In production, it was fragile. Deployments broke unpredictably, monitoring was inconsistent, and the internal team was constantly split between delivering new features and firefighting the LLM platform.

They knew they needed two senior hires: one person to own MLOps across Azure, Kubernetes and MLflow, and another to focus on retrieval-augmented generation, vector databases and evaluation. But the brief blended tools, patterns and “nice-to-haves” into a single overloaded profile. Internal TA only had basic filters for ML roles and limited time to decode what truly mattered for running LLM systems in production.

Calimala stepped in as a specialist Data & AI hiring partner. Working directly with the data platform lead, we translated that broad ambition into two clear, pragmatic role definitions—one for Lead MLOps, one for Senior RAG Engineer. For each, we made explicit trade-offs around seniority, salary bands, and the balance between on-site presence and remote flexibility.

The result: instead of “ML/LLM everything,” the client had a focused hiring plan tied to platform stability and LLM quality, not just a list of buzzwords.

Splitting the problem: platform stability vs LLM quality

The first step was to separate responsibilities that had been blurred for months. Together with the platform lead, we pulled apart an unrealistic “unicorn” profile into two complementary tracks:

• A Lead MLOps role responsible for running ML and LLM workloads in production on Azure and Kubernetes, with strong ownership of CI/CD, observability, security and incident response.

• A Senior RAG Engineer role focused on retrieval-augmented generation: vector stores, data pipelines, evaluation frameworks, and continuous improvement of LLM answer quality.

For both roles, we defined non-negotiables: experience running ML systems in production (not just POCs), fluency with Kubernetes, and awareness of security, monitoring and cost controls in a cloud-native environment. We aligned early on budget, the remote/on-site mix, and expectations around on-call and out-of-hours coverage so there would be no surprises later in the process.

A tight three-step interview flow was then agreed that respected engineers’ time and allowed the team to probe architecture decisions, incident stories and trade-offs instead of redoing basic screening.

Screening ML and LLM talent with people who have built platforms

Once the roles were locked in, the next challenge was quality at depth. Calimala combined its AI-native evaluation engine with human screening led by engineers who have worked on real data and ML platforms—not just demo environments.

Sourcing was global, with a focus on Eastern Europe and Pakistan for senior engineers experienced in distributed teams and complex infrastructure. Every shortlisted candidate went through a structured process:

• A focused pre-screen on previous production systems, incident narratives, and platform design decisions.

• AI-assisted scoring across tools and patterns (Azure ML, Kubernetes, MLflow, vector databases, RAG pipelines), delivery track record and communication.

• A concise 360° profile outlining strengths, gaps, culture fit and risk factors.

• Clear notes on availability, rate expectations, and constraints around relocation or travel.

Instead of raw “ML CVs,” the client saw side-by-side evaluation packs that surfaced a critical distinction: who had actually run ML and LLM systems in production, and who had only worked at the prototype or research layer.

This allowed the hiring manager to spend interview time on architecture trade-offs, scaling strategies and observability approaches—rather than basic filtering.

What changed for the client

With a small but strong shortlist, interviews progressed quickly and decisively. The team could finally separate platform stability from LLM quality and assign clear ownership for each.

Within three weeks, the client hired both a Lead MLOps Engineer and a Senior RAG Engineer. Accountability snapped into place: one leader for the ML/LLM platform’s reliability, observability and release practices, and one for retrieval-augmented generation, vector store design and evaluation.

The impact was tangible:

• Reduced time wasted on weak interviews and “CV scientists” who couldn’t demonstrate production experience.

• A sharper split between platform operations and LLM quality, allowing each area to progress in parallel.

• Fewer incidents on the ML platform as the new hires took ownership of observability, deployment hygiene and incident response.

By combining a boutique, practitioner-led approach with an AI-native evaluation engine, Calimala helped the client move from an unstable LLM prototype to a resilient, production-ready platform—with the right senior MLOps and RAG talent in place to keep it evolving.