Fullstack Engineer_Backend - GPUOD (英语全栈工程师)

About Rakuten

Rakuten group has almost 100 million customers based in Japan and 1 billion globally as well, providing more than 70 services in a variety such as e-commerce, payment services, financial services, telecommunication, media, sports, etc.

Division Introduction

AI & Data Division (AIDD) spearheads data science & AI initiatives by leveraging data from Rakuten Group. We build a platform for large-scale field experimentations using cutting-edge technologies to provide critical insights that enable faster and better and faster contribution for our business. Our division boasts an international culture created by talented employees from around the world. Following the strategic vision "Rakuten as a data-driven membership company", AIDD is expanding its data & AI related activities across multiple Rakuten Group companies.

About the Role

As a Full-Stack Engineer (Backend-Focused) on the In-House LLM Platform team, you will build and operate the platform that provides Rakuten's in-house LLM API for internal usage across the group. You will design the API gateway, model routing, and serving abstraction layer that exposes Rakuten's LLMs (self-hosted and fine-tuned) as a reliable, secure, and cost-efficient service to internal product teams.

This role is backend-heavy — you will own the LLM API surface, request routing, rate limiting, usage metering, and integration with model servers (vLLM, SGLang, TensorRT-LLM). You will also contribute to the front-end admin console, observability dashboards, and developer playground that let internal users discover, test, and monitor LLM usage.

This role requires strong backend engineering expertise, experience with LLM serving infrastructure, and working knowledge of modern frontend development, with a focus on building a production-grade LLM platform used by hundreds of internal teams.

Key Responsibilities

· Design, implement, and operate the in-house LLM Platform backend — API gateway, model routing, request dispatching, and serving abstraction over vLLM, SGLang, TensorRT-LLM, and other inference engines.

· Build OpenAI-compatible and custom LLM API endpoints (chat completions, embeddings, function calling, streaming) for internal product teams, with versioning, backward compatibility, and SDK generation.

· Implement platform-level capabilities: authentication and tenant isolation, rate limiting and quotas, usage metering and billing, model fallback and failover, prompt logging and audit trails.

· Integrate with the GPU infrastructure and model registry to route requests to the right model endpoint based on availability, latency, cost, and tenant policy.

· Develop cost-control mechanisms (model routing by tier, token-based metering, caching of prompts and embeddings, request throttling) to keep LLM inference cost-efficient at organization scale.

· Build and maintain the front-end admin console and developer playground (React/TypeScript) for model discovery, API key management, usage dashboards, prompt testing, and request inspection.

· Implement observability for the LLM platform — request tracing, latency and error metrics, GPU utilization dashboards, cost analytics — using Prometheus, Grafana, and distributed tracing.

· Enforce security and governance across the platform — API key management, RBAC, PII/secret redaction in prompts, audit logging, and compliance with internal data policies.

· Collaborate with the GPU training/inference optimization team, model fine-tuning team, and internal product teams to align platform capabilities with evolving model and business needs.

· Research and apply cutting-edge techniques (e.g., semantic caching, speculative decoding integration, multi-model routing, MCP) to optimize platform performance, cost, and developer experience.

Mandatory Qualifications

· 3+ years of hands-on experience in backend software engineering, building and operating production APIs and distributed services.

· Deep expertise in Python (or Go) and experience building scalable backend services with frameworks like FastAPI, Flask, Spring Boot, or Go net/http.

· Experience with RESTful API design, streaming APIs (SSE/WebSocket), authentication (OAuth, API keys, JWT), and API versioning.

· Experience with relational databases (PostgreSQL, MySQL), caching (Redis), and message queues (Kafka, RabbitMQ).

· Experience with containerization and orchestration (Docker, Kubernetes) and CI/CD pipelines (Jenkins, GitLab CI, GitHub Actions).

· Working knowledge of modern frontend development with React and TypeScript, including building admin consoles, dashboards, or developer tools.

· Bachelor's or higher degree in Computer Science, Engineering, or related field.

Nice-to-Have Skills

· Experience building or operating LLM serving platforms, AI gateways, or model serving infrastructure (vLLM, SGLang, TensorRT-LLM, Triton, TGI).

· Familiarity with LLM inference concepts — tokenization, KV cache, batching, quantization, streaming completions, function calling, embeddings.

· Experience with OpenAI-compatible API design and LLM SDKs (OpenAI Python/JS SDK, LangChain, LiteLLM).

· Experience with observability and tracing for LLM workloads (Langfuse, OpenTelemetry, prompt logging, evaluation pipelines).

· Familiarity with semantic caching, prompt caching, and cost optimization techniques for LLM APIs at scale.

· Experience with multi-tenant platform design — tenant isolation, quota management, usage metering, and billing.

· Experience with infrastructure-as-code (Terraform, Pulumi) and cloud platforms (AWS, GCP, Azure).

· Contributions to open-source LLM tooling or serving frameworks (vLLM, LiteLLM, LangChain, vLLM ecosystem).

· Experience with GPU workload scheduling on Kubernetes (KubeFlow, Volcano, GPU operator).

· Familiarity with security and compliance for AI platforms (PII redaction, prompt injection defenses, audit logging, RBAC).

Why Join Us?

· Build and operate the in-house LLM Platform that powers AI products across 70+ Rakuten services.

· Work on cutting-edge LLM serving, routing, and cost-optimization challenges at organization scale.

· Backend-heavy role with real full-stack ownership — from GPU model servers to developer-facing admin console.

· Collaborate with global AI/ML, GPU optimization, and platform engineering teams on high-impact challenges.

· Opportunity to shape the foundation of Rakuten's internal AI infrastructure and developer experience.