Short definition
An OOM kill is a Linux kernel action that forcibly terminates one or more processes when the system exhausts both physical RAM and swap space. The kernel selects which process to kill using a scoring algorithm based on memory consumption and process age. The terminated process disappears immediately with no user-facing error message.
Extended definition
When a Linux system reaches the point where no memory can be allocated for new or existing operations, the kernel activates the OOM killer as a last resort. Rather than crashing the entire system, the kernel sacrifices one or more processes to reclaim memory and keep the system running. This behavior is intentional and recoverable, but it creates serious problems in production environments where process continuity is expected.
The OOM killer calculates a badness score for each process. Higher scores indicate processes the kernel considers acceptable to kill. Memory-heavy processes with fewer child processes and shorter runtimes tend to score higher. System-critical processes like init are protected by default, and administrators can adjust scores manually using the oom_score_adj unable per process.
In practice, OOM kills surface in environments running inference workloads, large databases, or memory-hungry runtimes. A process killed by the OOM killer produces no application-level error. From the application’s perspective, the process simply stops. This makes OOM kills particularly difficult to diagnose without direct access to kernel logs. Detection relies on commands such as dmesg | grep -i 'killed process' or journalctl -k | grep oom.
OOM kills are preventable with proper capacity planning, memory limits at the container or cgroup level, and proactive monitoring. Ignoring memory pressure signals until the kernel intervenes is a reactive pattern that causes outages and data loss in production systems.
Deep technical explanation
How the OOM killer selects a victim
The kernel evaluates every running process using the oom_badness() function. This function computes a score based on the process’s memory footprint (RSS plus swap usage) relative to total available memory, then applies an adjustment from /proc/[pid]/oom_score_adj. Values range from -1000 (never kill) to +1000 (kill first). Administrators can pin critical daemons to -1000 to protect them entirely.
Once a victim is selected, the kernel sends SIGKILL to that process and any children sharing the same memory space. There is no grace period. The process tree is terminated synchronously. The kernel then logs the event to the ring buffer, including the process name, PID, and the amount of memory freed.
Memory overcommit and its role in OOM events
Linux defaults to memory overcommit: it allows processes to allocate more virtual memory than is physically available, under the assumption that not all allocations will be used simultaneously. The overcommit policy is controlled by /proc/sys/vm/overcommit_memory. Mode 0 (default) uses heuristics; mode 1 always permits allocation; mode 2 restricts allocations to physical RAM plus swap. Overcommit mode 0 or 1 means the system can accept allocations that it cannot fulfill at runtime, making OOM kills a predictable outcome under heavy load.
OOM kills in inference and AI workloads
Inference runtimes such as Ollama are common OOM kill victims. Three primary causes apply. First, KV cache growth during long conversations accumulates state in memory without an explicit cap, and a session that starts within memory bounds can exceed them mid-inference. Second, loading a model whose quantized size is close to available RAM leaves no headroom for the runtime overhead, OS buffers, or concurrent requests. Third, concurrent model loading multiplies peak memory demand and frequently triggers kills when two load operations overlap.
When Ollama is OOM-killed, the inference stops silently. The calling application receives a broken connection or timeout rather than an error message. Without kernel log monitoring, the root cause is invisible to application developers.
Detection and diagnostic commands
The two primary detection commands are dmesg | grep -i 'killed process' and journalctl -k | grep oom. Both read the kernel ring buffer where OOM kill events are recorded. A typical log entry includes the process name, PID, total pages consumed, and the oom_score at time of kill. Persistent kernel logs require that journald is configured to save them across reboots, which is not the default on all distributions.
Edge cases and failure modes
A common edge case is the zombie OOM loop: a process killed by the OOM killer is managed by systemd with a restart policy, restarts, allocates memory again, and is killed again in a cycle. This loop can pin CPU and prevent recovery without manual intervention. Another edge case involves cgroup memory limits: when a container hits its cgroup memory limit before the host exhausts RAM, the container’s OOM killer fires independently of the host kernel, and the host-level logs may show nothing. Operators must check cgroup-level events separately via /sys/fs/cgroup/memory/memory.oom_control.
Practical examples
An API server running on a 16GB instance began returning 502 errors sporadically. No application errors appeared in logs. Running journalctl -k | grep oom revealed that the inference process had been killed three times overnight due to growing KV cache from long-lived sessions. Adding a session memory cap and upgrading to a 32GB instance resolved the issue.
A containerized database was OOM-killed repeatedly under peak load. The container had no explicit memory limit, so it competed directly with the host OS for RAM. Setting a cgroup memory limit and enabling memory.oom_control notifications gave the team visibility into pressure events before they escalated to kills.
A machine learning pipeline loaded multiple model variants concurrently during A/B testing. Overlapping load operations doubled peak memory demand and triggered OOM kills on every deployment. Serializing model loads and pre-checking available RAM before each load operation eliminated the kills entirely.
A monitoring agent on a production server was killed by the OOM killer because it had a high oom_score due to its memory footprint. Setting oom_score_adj to -500 for the agent process protected it while still allowing less critical processes to be killed first under pressure.
Why it matters
- OOM kills terminate processes silently, making them one of the hardest failure modes to diagnose without direct kernel log access.
- Any process on the host is a potential victim, including monitoring agents, security tools, and system daemons, not just the memory-heavy application.
- In containerized environments, cgroup-level OOM events are invisible at the host kernel level and require separate instrumentation.
- Repeated OOM kills combined with systemd restart policies can create a resource-exhaustion loop that destabilizes the entire host.
- Proactive memory monitoring and capacity planning eliminate the conditions that trigger OOM kills before they reach production.
- Compliance frameworks that require availability guarantees treat OOM-driven downtime as an infrastructure control failure requiring documented remediation.