最近客户现场遇到一个私有内核模块触发ABBA死锁的问题:想着在本地的实验室复现,但是这个死锁问题牵扯“APIC-timer中断tracepoint”的运行,所以死锁出现的条件有点难,非常不容易复现。最后想是否可以在这个“LAPIC定时器中断注册Tracepoint处理函数”里,在获取到锁的之后主动sleep一下,为复现创造更多的机会,结果编译后放到虚拟机运行,出现了下面的告警:Aug 10 16:36:30 abc kernel: [ 368.266247] BUG: scheduling while atomic: health/48381/0x00010000Aug 10 16:36:30 abc kernel: [ 368.266250] Modules linked in: abc(OE) Aug 10 16:36:30 abc kernel: [ 368.266354] CPU: 1 PID: 48381 Comm: health Kdump: loaded Tainted: G OE -------- - - 4.18.0-553.111.1.el8_10.x86_64 #1Aug 10 16:36:30 abc kernel: [ 368.266358] Hardware name: Red Hat KVM, 0+19570+14a90618 04/01/2014Aug 10 16:36:30 abc kernel: [ 368.266361] Call Trace:Aug 10 16:36:30 abc kernel: [ 368.266377] <IRQ>Aug 10 16:36:30 abc kernel: [ 368.266382] dump_stack+0x41/0x60Aug 10 16:36:30 abc kernel: [ 368.266391] __schedule_bug.cold.104+0x4c/0x58Aug 10 16:36:30 abc kernel: [ 368.266400] __schedule+0x6a8/0x870Aug 10 16:36:30 abc kernel: [ 368.266407] schedule+0x55/0xf0Aug 10 16:36:30 abc kernel: [ 368.266410] schedule_timeout+0x19f/0x320Aug 10 16:36:30 abc kernel: [ 368.266414] ? __next_timer_interrupt+0xf0/0xf0Aug 10 16:36:30 abc kernel: [ 368.266423] ? printk+0x58/0x73Aug 10 16:36:30 abc kernel: [ 368.266428] msleep+0x28/0x40Aug 10 16:36:30 abc kernel: [ 368.266432] km_timer_entry.cold.16+0x19/0x1e [abc]Aug 10 16:36:30 abc kernel: [ 368.266444] smp_apic_timer_interrupt+0xf9/0x130Aug 10 16:36:30 abc kernel: [ 368.266448] apic_timer_interrupt+0xf/0x20Aug 10 16:36:30 abc kernel: [ 368.266454] </IRQ>
从上面的警告说明是说,在原子操作上下文里发生了调度/要切换task,这个是不被允许的,因为可能会导致各种同步问题。原子上下文(atomic-context),包括,持有自旋锁、中断上下文、软中断、tasklet;绝对不允许睡眠、不能调度、不能msleep/schedule_timeout。一旦调用睡眠函数,内核就会打印这个BUG警告,潜在后果:死锁、Oops、panic、系统卡死。所以见到这种警告不必惊慌,反而是一件好事,让人知道哪里有bug。如果msleep不能用,那还有什么可以用来做一定时间的延迟?答案可能是mdelay,但是这个也不能delay太长的时间,因为mdelay是忙等,会占死 CPU,延时大的时候会触发softlockup警告;/** * mdelay - Inserting a delay based on milliseconds with busy waiting * @n: requested delay in milliseconds * * See udelay() for basic information about mdelay() and it's variants. * * Please double check, whether mdelay() is the right way to go or whether a * refactoring of the code is the better variant to be able to use msleep() * instead. */#define mdelay(n) (\ (__builtin_constant_p(n) && (n)<=MAX_UDELAY_MS) ? udelay((n)*1000) : \ ({unsigned long __ms=(n); while (__ms--) udelay(1000);}))