科技
谷歌如何将GKE API超时问题追溯到Linux内核的TCP漏洞
中文译文
谷歌最近详细说明了谷歌 Kubernetes Engine(GKE)、Private Service Connect以及Linux内核中的三个相互关联的漏洞如何导致Kubernetes API连接出现间歇性故障,使得大约800名开发人员无法将应用部署到生产环境。
1月份,有一名客户使用Private Service Connect(PSC)为多个GKE集群提供私有的控制平面访问功能,当时这些集群中的Pod在尝试连接Kubernetes API服务器时出现了间歇性超时现象。
数据包捕获显示,控制平面在响应新的TCP连接请求时,使用了属于先前已关闭连接的过时ACK数据包。根本原因在于不同网络层中存在的三个独立漏洞。谷歌云的高级技术解决方案工程师Akash Thakur介绍了多个谷歌团队中的工程师是如何识别出这三个根本原因并最终解决它们的:在解决这一问题的过程中,我们动员了来自 Google Kubernetes Engine(GKE)、Private Service Connect、网络组件、hostnet、内核以及 Linux 上游社区的工程师们,他们共同找到了三个不同的根本原因。最终的解决方案并非通过配置更改或功能开关来实现,而是通过对 Linux 内核本身的修补来完成的。
调查发现了三个相互关联的原因:Private Service Connect 中的数据包重排序问题、Envoy 的半关闭行为导致 TCP 连接无法正常关闭,以及 Linux 内核中存在会丢弃某些重排序后的 FIN 数据包的缺陷。
来源:Google Cloud 博客
Thakur 描述了一例复杂的故障排查案例,涉及 Private Service Connect、虚拟网络接口以及 Linux TCP 状态机中的 I/O 超时问题,最终发现是内核存在缺陷,使得套接字无限期地停留在 FIN_WAIT_2 状态。Thakur 解释道:大多数网络问题都在某一层得到解决。你可以找到配置错误的防火墙规则、用尽的连接池,或是计算有误的MTU值。修复这些问题后问题便迎刃而解。但这个案例有所不同,因为无论从Private Service Connect层、内核的TCP状态机,还是Envoy的连接管理机制来看,其症状——即连接从未完成时就出现I/O超时——都是一样的。每一层都有看似合理的解释,但每一个解释都不完整。
该问题通过几周内分三步协同解决的方案得以解决:谷歌首先回退了Private Service Connect的一项更改以恢复连接,随后修正了代理协议的处理方式并增加了回归测试。最后,由曾任职于谷歌的软件工程师Eric Dumazet编写的Linux内核补丁解决了底层的TCP问题。谷歌云的董事总经理Rohan Karne评论道:这就是关键任务平台工程中并不光鲜的现实——需要系统地排除那些显而易见的故障原因,深入分析数据包捕获信息,直至找到内核层面的根本原因。
Thakur指出,网络设备、代理服务器和操作系统各自看似合理的行为,却可能共同导致复杂的基础设施故障。在多个网络层进行数据包捕获对于识别问题至关重要,而将网络症状与基础设施的变化相关联,则有助于确定这三个故障之间的关联。香港中文大学的博士后研究员余光巴强调:
对于AIOps和混沌工程而言,可借鉴的方法是同时收集路径状态、协议状态以及部署事件的信息,同时区分症状的恢复与状态机状态的修正。这个经过匿名处理的案例并未提供整体故障率,也无法解释所有的API超时问题。
Sidero的现场首席技术官Justin Garrison指出:起初,看到这篇介绍内核补丁的文章我感到很兴奋,因为这类内容总是很有趣。结果发现GKE使用的仍是3年前的内核版本(6.1),而该漏洞早已在上游版本中被修复,他们不得不进行回迁应用。
该内核补丁已被合并到Linux 6.9.7版本中,并回迁到6.6和6.1 LTS分支,最终通过1.33.5及更高版本的GKE提供给用户。
原文
Google recently detailed how three interacting bugs across Google Kubernetes Engine (GKE), Private Service Connect, and the Linux kernel caused intermittent Kubernetes API connectivity failures, blocking around 800 developers from deploying to production.
In January, a customer running multiple GKE clusters with Private Service Connect (PSC) for private control-plane access experienced intermittent timeouts when pods attempted to reach the Kubernetes API server.
Packet captures revealed that the control plane was responding to new TCP connection requests with stale ACK packets belonging to previously closed connections. The root cause involved three separate bugs across different networking layers. Akash Thakur, senior technical solutions engineer at Google Cloud, describes how engineers across multiple Google teams identified three root causes and ultimately addressed them:
In a race to fix the problem, we involved engineers across Google Kubernetes Engine (GKE), Private Service Connect, networking, hostnet, kernel, and the Linux upstream community, who together tracked down not one but three distinct root causes. And the final fix landed not as a config change or a feature flag, but as a patch to the Linux kernel itself.
The investigation identified three interacting causes: packet reordering in PSC, Envoy's half-close behavior preventing TCP connections from being properly closed, and a Linux kernel bug that discarded certain reordered FIN packets.
Source: Google Cloud blog
Thakur describes a complex troubleshooting case involving I/O timeouts across Private Service Connect, virtual network interfaces, and the Linux TCP state machine, which ultimately revealed a kernel issue leaving sockets stuck indefinitely in the FIN_WAIT_2 state. Thakur explains:
Most networking cases resolve at one layer. You find the misconfigured firewall rule, the exhausted connection pool, the miscalculated MTU. You fix it and move on. This one was different because the symptom — i/o timeout on a connection that never completed — was the same whether you were looking at the Private Service Connect layer, the kernel's TCP state machine, or Envoy's connection management. Every layer had a plausible explanation, and every explanation was incomplete.
The issue was addressed through three coordinated fixes over several weeks: Google first rolled back a Private Service Connect change to restore connectivity, then corrected the proxy protocol handling and added regression tests. Finally, a Linux kernel patch, authored by Eric Dumazet, formerly a software engineer at Google, fixed the underlying TCP issue. Rohan Karne, managing director at Google Cloud, comments:
This is the unglamorous reality of mission-critical platform engineering—methodically ruling out the obvious suspects and digging into packet captures until you find the kernel root cause.
Thakur highlights how individually reasonable behaviors across networking, proxies and operating systems can combine into complex infrastructure failures. Packet captures at multiple network layers were essential to identifying the problem, while correlating network symptoms with infrastructure changes helped establish the connection between the three bugs. Guangba Yu, a postdoctoral researcher at the Chinese University of Hong Kong, highlights:
For AIOps and chaos engineering, the transferable method is organizing simultaneous path observations, protocol state, and rollout events while distinguishing symptom recovery from state-machine correction. This anonymized case supplies no population failure rate and does not explain all API timeouts.
Justin Garrison, field CTO at Sidero, notes:
At first I was excited to read this article that leads to a kernel patch which are always fun. It turns out GKE just runs a kernel from 3 years ago (6.1) and the bug was already fixed upstream and they had to backport it.
The kernel fix was merged into Linux 6.9.7 and backported to the 6.6 and 6.1 LTS branches, reaching GKE customers in version 1.33.5 and later.
