Staff Machine Learning Engineer, LLM Inference Optimization
Nebius - Zurich
Description du poste
<div class="content-intro"><p><strong>About Nebius:</strong></p> <p>Nebius is leading a new era in cloud infrastructure for the global AI economy. We are building a full-stack AI cloud platform that supports developers and enterprises from data and model training through to production deployment, without the cost and complexity of building large in-house AI/ML infrastructure.</p> <p>Built by engineers, for engineers. From large-scale GPU orchestration to inference optimization, we own the hard problems across compute, storage, networking and applied AI.</p> <p>Listed on Nasdaq (NBIS) and headquartered in Amsterdam, we have a global footprint with R&D hubs across Europe, the UK, North America and Israel. Our team of 1,500+ includes hundreds of engineers with deep expertise across hardware, software and AI R&D.</p></div><div class="_3iDTPbvLZJ renderer-heading-wrapper" data-testid="renderer-heading-wrapper" data-level="2"> <h2 id="Role-Summary" data-local-id="ad213d8fe4e2" data-renderer-start-pos="1929">Role Summary</h2> </div> <p data-renderer-start-pos="1943" data-local-id="4daa64b4c380">Nebius Token Factory is building production-grade inference for open-source, customer, and internally optimized models. This role owns the path from model artifact to fast, reliable, cost-efficient endpoint. The work sits at the intersection of model internals, inference engines, serving architecture, benchmarking, and production deployment.</p> <div class="_3iDTPbvLZJ renderer-heading-wrapper" data-testid="renderer-heading-wrapper" data-level="2"> <h2 data-local-id="6b105a288faf" data-renderer-start-pos="2508"> </h2> <h2 id="What-You-Will-Do" data-local-id="6b105a288faf" data-renderer-start-pos="2508">What You Will Do</h2> </div> <ul class="ak-ul" data-local-id="ebbff5d80805" data-indent-level="1"> <li> <p data-renderer-start-pos="2528" data-local-id="c5115fb223c0">Own a platform-level optimization pillar such as quantization, speculative decoding, <span data-highlighted="true" data-vc="highlighted-text">KV</span>-cache optimization, engine selection, or long-context serving.</p> </li> <li> <p data-renderer-start-pos="2682" data-local-id="0ffc3414d75f">Create reusable evaluation, benchmarking, and rollout frameworks across multiple model families.</p> </li> <li> <p data-renderer-start-pos="2782" data-local-id="b1093d4c9089">Provide technical direction to MLEs, Applied Scientists, GPU kernel engineers, and infrastructure engineers on shared performance goals.</p> </li> <li> <p data-renderer-start-pos="2922" data-local-id="ba8442a0b3db">Optimize LLM and <span data-highlighted="true" data-vc="highlighted-text"><span class="_1mT5xdRAKS"><span class="_0nmozVaHVx _2V7sgO2M4S _3BoeIDwZ7G _30zrXRyxdf _0ReiIH7zJH" data-testid="definition-highlighter">VLM</span></span></span> endpoints for latency, throughput, memory efficiency, GPU utilization, quality, and cost per token.</p> </li> <li> <p data-renderer-start-pos="3046" data-local-id="295f060f2f89">Deploy, configure, benchmark, and extend inference engines such as vLLM, SGLang, TensorRT-LLM, Triton Inference Server, <span data-highlighted="true" data-vc="highlighted-text">NVIDIA</span> Dynamo, or similar systems.</p> </li> <li> <p data-renderer-start-pos="3204" data-local-id="a92060bc9227">Build and productionize model-compression workflows, including quantization, quantization-aware training, distillation, low-bit serving, and accuracy recovery.</p> </li> <li> <p data-renderer-start-pos="3367" data-local-id="fc94ad87ea7d">Implement or integrate speculative decoding, draft-model approaches, <span data-highlighted="true" data-vc="highlighted-text">KV</span>-cache optimization, prefix caching, chunked prefill, continuous batching, and disaggregated prefill/decode serving.</p> </li> <li> <p data-renderer-start-pos="3558" data-local-id="767163fd69ef">Build reproducible benchmark harnesses for <span data-highlighted="true" data-vc="highlighted-text">TTFT</span>, <span data-highlighted="true" data-vc="highlighted-text">TPOT</span>, tokens per second per GPU, p95/p99 latency, GPU memory, reliability, and cost per token.</p> </li> <li> <p data-renderer-start-pos="3705" data-local-id="08931f048a7e">Partner with GPU kernel engineers and platform engineers to diagnose bottlenecks across model code, kernels, runtime, scheduler, gateway, and cluster layers.</p> </li> <li> <p data-renderer-start-pos="3866" data-local-id="40fc2b50527b">Write clear design docs, performance reports, rollout plans, and customer-facing technical explanations.</p> </li> </ul> <div class="_3iDTPbvLZJ renderer-heading-wrapper" data-testid="renderer-heading-wrapper" data-level="2"> <h2 id="Required-Qualifications" data-local-id="d808b8c02bc9" data-renderer-start-pos="3974">Required Qualifications</h2> </div> <ul class="ak-ul" data-local-id="000c2bf27478" data-indent-level="1"> <li> <p data-renderer-start-pos="4001" data-local-id="8bcf8dd9c7f9">Strong Python and PyTorch engineering skills.</p> </li> <li> <p data-renderer-start-pos="4050" data-local-id="781d2e3f4c3f">Hands-on experience deploying or optimizing LLM, <span data-highlighted="true" data-vc="highlighted-text">VLM</span>, or high-throughput transformer inference systems.</p> </li> <li> <p data-renderer-start-pos="4157" data-local-id="604abc69fbd4">Practical knowledge of at least one modern inference stack such as vLLM, SGLang, TensorRT-LLM, Triton Inference Server, <span data-highlighted="true" data-vc="highlighted-text">NVIDIA</span> Dynamo, Ray Serve, KServe, or equivalent internal systems.</p> </li> <li> <p data-renderer-start-pos="4346" data-local-id="3815fcf1bdd9">Strong understanding of transformer inference bottlenecks, including <span data-highlighted="true" data-vc="highlighted-text">KV</span> cache, attention, memory bandwidth, batching, parallelism, and long-context serving.</p> </li> <li> <p data-renderer-start-pos="4506" data-local-id="e7ade27b40b8">Ability to reason quantitatively about latency, throughput, quality