Patch Lifecycle: From CVE to Client Machines

This document outlines the comprehensive lifecycle of kernel live patches, from the initial identification of Common Vulnerabilities and Exposures (CVEs), to their distribution and ongoing monitoring on customer systems. It details the rigorous quality assurance (QA) methods employed to ensure the stability and effectiveness of patches, and how they are delivered and observed in the field.

CVE Triage and Kernel Patching Process

The process begins with the Canonical Security team, which receives initial CVE reports and provides a list of CVEs affecting kernel packages, and passes them on to the kernel team. The kernel team then triages these CVEs, assigning priority levels (e.g., high, critical). The kernel team also writes the patches for the affected kernels to fix these CVEs. Only patches that fix high or critical severity Linux kernel vulnerabilities are eligible for livepatch. The Livepatch team is responsible for developing kernel livepatches based on the prioritized patches. While the vulnerabilities being addressed by the kernel updates and livepatches are the same, a livepatch for a vulnerability can be significantly more complex than an ordinary kernel patch.

Livepatch Testing and Release

Livepatches undergo a meticulous testing process. Easy patches are prioritized first, while more complex ones requiring rework or backports are addressed subsequently. Testing involves:

• Unit tests: Performed on a virtual machine by loading the livepatch against the same kernel it was written for.
• Kernel regression tests: Determine if the livepatch causes any issues while running kernel regression tests. The goal is for the livepatched kernel to pass the same tests as the original, unpatched kernel. These tests are performed on internal machines and monitored for any problems in the patches

Upon successful testing internally, patches are released in phases to the updates tier (e.g., 10%, 20%, 40%, 60%), which consists of free users of Ubuntu Pro. The Livepatch team continuously monitors the metrics for each patch version. This is done using the monitoring support included with the Livepatch client to track patch health and client machine issues. If no crashes are detected and no negative feedback or bug reports are raised, the livepatches are then promoted to the stable tier, which is available to paid Ubuntu Pro users.

Livepatch Installation

Once the livepatch for a kernel has been released for use, the Livepatch client downloads the patch from the configured Livepatch server and patch source. The downloaded payload consists of the Linux kernel module responsible for patching the running kernel, as well as some metadata. The Livepatch client then inserts the kernel module by making the appropriate syscall. This only affects the in-memory kernel code and not the installed kernel.

The Livepatch client is also equipped with mechanisms to prevent crash loops on installation of a faulty livepatch. For more information on patch installation and preventing crash loops, see Patch Installation reference

Livepatch Security

Ensuring the secure transmission and application of a livepatch is paramount to maintaining kernel stability and preventing installation of malicious kernel modules. The Livepatch client ensures that installed livepatches are genuine and authentic before applying them to the kernel. This is achieved by verifying the livepatch content using SHA256 file checksums, verifying the digital signature for the livepatch kernel module using asymmetric encryption and using TLS to communicate with the hosted Canonical Livepatch server.

For more information on how the security of livepatches is ensured, see Patch Security reference.

Livepatch Monitoring and Blocklisting on faulty patches

In addition to preventing crash loops on application of a faulty livepatch , the Livepatch client has comprehensive monitoring support, to report the health of the client machine and any crashes caused by an applied livepatch, to the Livepatch server. After a livepatch is applied, the Livepatch client sends multiple pings to the hosted Canonical Livepatch server. These pings are anonymous and are only used to understand the health and status of machines, after the application of a livepatch.

The insertion pings are sent just before and after the livepatch has been applied. These pings enable the Livepatch team to monitor the number of livepatch module insertions that were attempted. The health-check pings are then sent after a delay of 10 seconds, 1 minute, 5 minutes and 15 minutes from the time of application of a livepatch, to confirm that the client machine is in a healthy state. However, if the application of a livepatch results in a kernel crash or a kernel bug, within 10 seconds of applying the patch, the error state is reported to the Livepatch server detailing the reason for the failure.

These client metrics are continuously monitored by the Livepatch team. In the rare case, where a significant number of client machines report an error state for a livepatch, the Livepatch team analyses the error reports. The Livepatch team then has the mechanism to blocklist a faulty patch. Once a patch has been blocklisted, it will no longer be distributed to new clients or applied in the client machines that already possess the blocklisted patch.