Problem
Design an online judge (LeetCode/HackerRank-style): run arbitrary, untrusted user code against test cases and return a verdict — pass/fail, runtime, memory. Unlike most system-design problems, the central constraint here is security, not raw throughput.
Requirements
Functional
- Accept a submission (source + language) for a problem, run it against test cases, and return per-test-case results and an overall verdict.
- Support multiple languages, each with different compile/run semantics.
- Enforce time and memory limits per submission.
Non-functional
- Isolation: untrusted code must never reach the host, the network, or other users' data.
- Fairness/consistency: the same submission gets the same verdict regardless of which worker runs it — no noisy-neighbor timing flakiness flipping a correct solution to TLE.
- Throughput: many concurrent submissions, e.g. thousands in a short window during a live contest.
Architecture
- Submission service: accepts and persists a submission, enqueues it on a work queue that's intentionally not partitioned by user, so no single user can starve the queue for everyone else.
- Sandboxed workers: a fleet of workers pulls jobs and runs submitted code inside an isolated sandbox (locked-down container or a microVM like Firecracker) with hard CPU-time, wall-clock, and memory limits enforced by the sandbox itself.
- Compile step: for compiled languages, a separate, also-sandboxed compile stage runs before execution — a malicious "compilation" is itself an attack vector.
- Test-case runner: each test case runs in a fresh process inside the sandbox so state never leaks between test cases within one submission; results (stdout, exit code, time, memory) are captured and compared to expected output.
- Result aggregation: per-test-case results roll up into a verdict (Accepted / Wrong Answer / TLE / MLE / Runtime Error), stored and pushed back to the user.
Key decisions
- OS/hypervisor-level isolation (containers/microVMs) over a language-runtime sandbox — language-level sandboxes have a long history of escape bugs; a VM/container boundary is stronger and simpler to reason about.
- Time/memory limits measured from inside the sandbox's own accounting, not by timing the request from the submission service — network and queueing delay must never count against a user's time limit.
- A queue with no per-user priority for the common case, so a flood of intentionally-slow submissions from one user can't push everyone else's submissions to the back — fairness is a queueing-discipline decision, not just a capacity one.
Tradeoffs
- Stronger isolation (microVMs) costs more per-submission boot overhead than lighter containers — worth it given the whole premise is running arbitrary untrusted code, but users feel it as slower verdicts.
- Fresh-process-per-test-case avoids state leakage but pays process-startup cost per test case rather than per submission — for hundreds of test cases, that adds up against overall judge latency.
- A fairness-first queue means even a legitimate high-priority case (a contest's final tiebreak submission) gets no preferential treatment without adding separate logic for it.
Failure modes
- Worker crash mid-execution: the job needs to be idempotently retryable or explicitly marked a system-side failure, so a transient infra hiccup doesn't wrongly surface as "Wrong Answer."
- Sandbox escape or resource-limit bypass: treat the worker VM/container as compromised after running untrusted code and recycle it rather than reuse it, even if the sandbox is believed sound.
- Queue backlog during a contest spike: submissions should queue visibly ("your submission is queued") rather than silently timing out client-side, so lag doesn't get misread as the user's own code being slow.
Capacity estimates
- A single test case run is typically sub-second for correct, efficient solutions, but a 1–2s time limit means a worker can be tied up for the full limit on an infinite loop — pool sizing has to assume a meaningful fraction of submissions consume their full limit, not the happy-path average.
- A live contest with ~10K concurrent participants submitting every few minutes is on the order of a few dozen to low hundreds of submissions/sec at peak — modest volume, but each submission is far more expensive per unit than, say, an ad-click event.
- Sandbox boot/teardown overhead is often the dominant per-submission cost at this scale, more than actual test-case execution time for typical, well-optimized solutions.
What I'd change at 10x scale
- Move from booting a fresh sandbox per submission to a pool of pre-warmed sandboxes that get reset and reassigned — paying the boot cost once per pool slot instead of once per submission.
- Split the worker fleet by language, since compile/runtime costs differ wildly (a C++ compile vs. starting a Python interpreter), so one language's load spike doesn't starve capacity for another.
- Add a lightweight static pre-check (obvious infinite-loop patterns, disallowed syscalls at a syntax level) before full sandboxed execution, worth the complexity only once sandbox slots are the real bottleneck.