Senior ML Engineer, Inference Stack Optimization
New
C
Cast AIML inference infrastructure
Location: Austria; France; Germany; Italy; Netherlands; Poland; Spain; United KingdomFull-TimeSenior
SalaryCompetitive salary (depending on the level of experience).
Apply NowOpens the employer's application page
Job Details
- Experience
- 5+ years
- Required Skills
- PythonKubernetesPyTorch
Requirements
- Have 5+ years of experience building real ML systems.
- Bring depth in inference or training infrastructure, beyond model-training notebooks.
- Write production-grade Python for services, not only scripts.
- Have hands-on experience running vLLM, SGLang, or TensorRT-LLM.
- Understand how inference engines perform on different GPUs.
- Have measured quantization regressions, not only compression ratios.
- Be familiar with multi-GPU and multi-node systems, including tensor and expert parallelism, collective communication, and practical failure modes.
- Understand page caches, O_DIRECT, network filesystems, terabyte-scale reads, and data-transfer performance.
- Instrument systems before optimizing and distinguish real improvements from benchmark artifacts.
- Nice to have: experience with LLM routing evaluations, calibration, abstention, conformal methods, or online A/B loops with drift monitoring.
Responsibilities
- Improve throughput through continuous batching, chunked prefill, speculative decoding, CUDA-graph capture, and scheduler tuning across the engine fleet.
- Reduce time to first token and inter-token latency by profiling and addressing compute, memory bandwidth, scheduling, storage, or network bottlenecks.
- Optimize KV caches using hierarchical and paged attention, prefix caching, eviction policies, and quantized KV.
- Reduce cold starts by improving weight loaders, JIT and compile caches, and shared-filesystem read paths.
- Apply INT8/INT4 and FP8/FP4 quantization to weights, activations, and KV, measuring quality on real workloads.
- Scale distributed inference across nodes using network-aware placement, prefill/decode disaggregation, and checkpointing strategies.
- Set technical direction by deciding what to benchmark, adopt, or build, and support decisions with writeups and reproducible experiments.
View Full Description & ApplyYou'll be redirected to the employer's site
