单核 CFS 只关心"同一 CPU 上实体间的 vruntime 公平",不关心"这个 CPU 上到底有多少负载/利用率"。多核时代(尤其异构大小核、DVFS、EAS)必须回答三个问题:
老内核用 cpu_load[] 指数衰减数组(无实体粒度、与优先级无关、只反映 tick 抽样),无法支撑上述三类决策。PELT 把历史负载建模为一个按 1ms 分段、几何衰减的等比级数,且每个 sched_entity、每个 cfs_rq、每个 rt/dl/irq/thermal 队列各自维护一套指标,实现 O(1) 增量更新。
等比级数模型
p_0(当前) ~ p_N(N ms 前)。u_i 为第 i 段中实体"可运行/运行"的时间占比,则历史负载 = u_0 + u_1·y + u_2·y² + ...。y^32 = 0.5(LOAD_AVG_PERIOD = 32):32ms 前的贡献权重减半,对应"一个调度周期"。load_avg' = u_0' + y·load_avg,O(1) 增量。load_avg = runnable% × scale_load_down(load)runnable_avg = runnable% × 1024util_avg = running% × 1024为加深理解,这里重点介绍上述三个信号:三个量来自同一套 PELT 引擎,只是三种不同的度量口径:
定点数与查表:y^n 用 32 位定点数查表 runnable_avg_yN_inv[](由 Documentation/scheduler/sched-pelt.c 工具生成);LOAD_AVG_MAX = 47742 为等比级数上限。
LOAD_AVG_MAX = 47742(定点饱和值,可理解为单个任务在每个PELT周期内一直运行所对应的load值)定义:PELT 单周期(32ms)衰减系数:y^32 = 0.5每ms的衰减系数: y = 0.5^(1/32)指数加权滑动平均稳态饱和最大值公式:$$(LOAD_AVG_MAX = \sum_{n=0}^{\infty} 1024 \cdot y^{n}) $$根据等比数列关系可以换算为:$$LOAD_AVG_MAX = \sum_{n=0}^{\infty} 1024 \cdot y^{n} = \frac{1024}{1-y},\quad |y|<1$$内核定点整数递推、每周期整数截断,将y定义代入后或计算出实际收敛饱和上限等于 47742。
频率/容量不变量(clock_pelt):util_avg 要能跨频率、跨 CPU 容量比较,因此 PELT 使用的时钟不是裸 clock_task,而是经过 arch_scale_cpu_capacity/arch_scale_freq_capacity 缩放后的 clock_pelt(update_rq_clock_pelt)。update_rq_clock_pelt 在 rq 空闲(is_idle_task)时把 clock_pelt 同步回 clock_task;满载饱和时的"丢失空闲"则由 update_idle_rq_clock_pelt 累计进 lost_idle_time,读取侧 rq_clock_pelt = clock_pelt - lost_idle_time。
lost_idle_time 用于稳定负载变化率:现实中即使 rq 的 util_avg 稳定在 100%,队列也几乎总是非空,依然会出现瞬时空闲(任务切换缝隙、新任务唤醒延迟、负载均衡迁入迁出瞬间、softirq 处理窗口等)。这些空闲在满容量下同样存在,只是微不足道(100% 利用率意味着平均空闲趋近于 0),PELT 的衰减会被随后的 running 迅速补回。
PELT 的时钟输入为 rq->clock_pelt:rq 的 curr 为 idle 时更新 rq->clock_pelt = rq_clock_task(rq);rq->clock_task 是真实时间,不会经 capacity 缩放。

隐藏冲突点:同样的瞬时空闲,在低 capacity CPU 上会被"放大"。放大不是空闲本身变长,而是 clock_pelt 的 idle 同步把整个 busy 段累计的缩放差值一次性吐了出来。
场景:half capacity(容量 512)满载 rq,忙碌 100 个墙钟单元后出现 20 单元的瞬时 idle。


delta 累积。delta >>= 10 截断,量化误差导致 *_avg 在 [1002..1024) 抖动,用 get_pelt_divider()(PELT_MIN_DIVIDER + period_contrib)做除法定标消除该抖动。util_avg 按"CPU 剩余容量的一半"预置(cap = (cpu_scale - cfs_rq->avg.util_avg) / 2,见 post_init_entity_util_avg),防新任务瞬间冲高越过 overutilized 阈值、破坏 EAS 布置。util_avg 衰减慢,唤醒场景会低估任务需求。用 util_est(EWMA + enqueued 峰值)做唤醒时任务利用率的上界估计(task_util_est)。
依赖关系:PELT 核心不依赖上层;上层只读 *.avg.{load,runnable,util}_avg / util_est / avg_rt / avg_dl / avg_irq / avg_thermal。所有写操作都发生在持 rq->lock 的调度路径内,读方多用 READ_ONCE。
/Volumes/workspace/linux/linux-stable-5.10) | ||
|---|---|---|
kernel/sched/pelt.c | ||
kernel/sched/pelt.h | clock_pelt 时钟语义、PELT_MIN_DIVIDER | |
kernel/sched/sched-pelt.h | runnable_avg_yN_inv[]、LOAD_AVG_PERIOD=32、LOAD_AVG_MAX=47742 | |
kernel/sched/sched.h | struct cfs_rqstruct rq(avg_rt/avg_dl/avg_irq/avg_thermal/clock_pelt/lost_idle_time) | |
include/linux/sched.h | struct sched_avgstruct util_est、struct sched_entity | |
kernel/sched/fair.c | ||
kernel/sched/cpufreq_schedutil.c | schedutil_cpu_utilsugov_get_util消费 cpu_util_cfs | |
kernel/sched/core.c | update_rq_clockupdate_rq_clock_task(调用 pelt.h 的 clock 更新)、scheduler_tick | |
include/trace/events/sched.h | pelt_{se,cfs,rt,dl,thermal,irq}_tpsched_util_est_{se,cfs}_tp、sched_overutilized_tp | |
Documentation/scheduler/sched-pelt | sched-pelt.h 的工具(源码注释注明) | |
kernel/sched/Makefile | obj-$(CONFIG_SMP) += ... pelt.o |
uapi 说明:PELT 结构(
struct sched_avg/util_est)全部位于内核私有头include/linux/sched.h、kernel/sched/sched.h,不向用户态暴露任何 uapi 结构/系统调用。用户态只能通过 tracepoint/procfs/debugfs 观测。
相关CONFIG_*配置项及开关影响:
CONFIG_SMP | kernel/sched/Makefile 仅 SMP 编译 pelt.o;非 SMP 下 pelt.h 全为 inline 空实现,update_load_avg 退化为只调 cfs_rq_util_change |
CONFIG_FAIR_GROUP_SCHED | cfs_rq->tg_load_avg_contrib/h_load/propagate 组负载传播,影响 task_h_load 与 update_cfs_group |
CONFIG_CFS_BANDWIDTH | cfs_rq_clock_pelt |
CONFIG_SCHED_THERMAL_PRESSURE | avg_thermal;热降频期间的"容量损失"计入负载 |
CONFIG_HAVE_SCHED_AVG_IRQ | avg_irq;中断/steal 时间作为任务不可见容量损失计入 util |
CONFIG_ENERGY_MODELCONFIG_CPU_FREQ_GOV_SCHEDUTIL | find_energy_efficient_cpu 用 cpu_util_next + compute_energy |
CONFIG_CPU_FREQ_GOV_SCHEDUTIL | cpu_util_cfs(util_avg + util_est) |
CONFIG_NUMA_BALANCING | sg_lb_stats.nr_numa_running |
CONFIG_SCHED_DEBUG | sched_features 可动态开关 UTIL_EST 等调度特性 |
CONFIG_SCHEDSTATS |
本文聚焦于负载统计与更新,暂不展开调频相关部分。
struct sched_avg {u64 last_update_time; /* 上次更新时间(clock_pelt 域) */u64 load_sum; /* 未归一化加权负载累积和(64bit,防溢出,见 395-408) */u64 runnable_sum; /* 未归一化可运行累积和 */u32 util_sum; /* 未归一化利用率累积和(<<10 定标) */u32 period_contrib; /* 当前 1ms 周期内已累计的 us 数 */unsigned long load_avg; /* = load*load_sum/divider */unsigned long runnable_avg; /* = runnable_sum/divider */unsigned long util_avg; /* = util_sum/divider, [0,1024] */struct util_est util_est; /* 唤醒估计 */} ____cacheline_aligned; /* 与实体其它字段隔离缓存行 */
*_avg = *_sum / get_pelt_divider(sa),其中 get_pelt_divider = LOAD_AVG_MAX - 1024 + period_contrib。对 sched_entity 而言:load_sum 不含权重,load_avg 含权重,即 load_avg = se_weight() * load_sum / get_pelt_divider(sa)。last_update_timedelta = now - last_update_time 补算流失周期。avg 随任务迁移/调度类切换在 CPU 间漂移;util_avg 用 WRITE_ONCE 发布,读者(EAS/均衡/调频)用 READ_ONCE;util_est.enqueued 用 MSB UTIL_AVG_UNCHANGED 做脏标记,原子读改写。struct util_est {unsigned int enqueued; /* 最近一次激活结束时的 PELT util_avg 快照(高比特位兼作脏标记) */unsigned int ewma; /* 历史 EWMA,平滑 util 下降 */};#define UTIL_EST_WEIGHT_SHIFT 2 /* EWMA 权重 w=1/4 */#define UTIL_AVG_UNCHANGED 0x80000000
task_util_est(p) = max(task_util(p), max(ue.ewma, ue.enqueued&~MSB))。enqueuedcfs_se_util_change()里被清零以标记 "PELT 已更新",util_est_update() 在任务睡眠时据此决定是否刷新 EWMA。CONFIG_SMP 下成员:struct sched_avg avg(独占缓存行)。相关字段:load.weight(负载权重)、on_rq、sum_exec_runtime、vruntime。任务迁移/睡眠后avg仍保留,用于在新 CPU 上 attach_entity_load_avg 恢复。
struct sched_avg avgstruct { raw_spinlock_t lock; int nr; unsigned long load_avg/util_avg/runnable_avg; } removedremove_entity_load_avg() 在自身 rq->lock 下把 avg 累加到 removed,随后由任意 CPU 的 update_cfs_rq_load_avg() 在拿 removed.lock 后批量减除。这是 32bit 与 CPU 热插拔/跨 CPU 衰减的重要并发设计。tg_load_avg_contribtg->load_avg 的贡献。propagateprop_runnable_sum:组负载向上传播的挂起量。h_loadh_load_next:层级化负载因子(task_h_load 用)。u64 clock_peltunsigned long lost_idle_time:频率/容量不变量时钟。struct sched_avg avg_rtavg_dl、avg_irq(CONFIG_HAVE_SCHED_AVG_IRQ)、avg_thermal(CONFIG_SCHED_THERMAL_PRESSURE)。unsigned long misfit_task_loadupdate_misfit_status维护,表示"当前 CPU 上装不下的大任务"。rq->lock 时写;clock/clock_pelt 由 update_rq_clock()(要求 rq->lock)维护。avg 之间的三层聚合关系:


__update_load_avg_se(now, cfs_rq, se) 持 rq->lock→ ___update_load_sum(now, &se->avg, load, runnable, running)├─ delta = now - sa->last_update_time├─ (s64)delta < 0 → 时钟回拨(sched clock 初始化换源): 仅复位 last_update_time,return 0├─ delta >>= 10(1024ns≈1us 单位); !delta → return 0├─ !load → runnable = running = 0 处理”已出队但 cfs_rq->curr 仍指向它”的边角(idle_balance 时)└─ accumulate_sum(delta, sa, load, runnable, running)├─ periods = (delta + period_contrib) / 1024├─ 跨周期时 Step1: load/runnable/util_sum 全部 decay_load(× y^periods)├─ Step2: __accumulate_pelt_segments(periods, 1024-period_contrib, delta%1024)│ c1 = d1·y^periods; c2 = 1024·Σy^n(n=1..p-1) = LOAD_AVG_MAX - decay(LOAD_AVG_MAX,p) - 1024; c3=d3├─ period_contrib = delta % 1024└─ load_sum += load·contrib; runnable_sum += runnable·contrib<<10; util_sum += contrib<<10return periods(>0 表示跨了周期)→ ___update_load_avg(&se->avg, se_weight(se)) /* 只有跨周期才执行 */divider = PELT_MIN_DIVIDER + period_contribload_avg = load·load_sum/divider; runnable_avg = runnable_sum/dividerWRITE_ONCE(util_avg, util_sum/divider)→ cfs_se_util_change(&se->avg) // 清 UTIL_AVG_UNCHANGED 脏标记→ trace_pelt_se_tp(se)
同理的入口:__update_load_avg_cfs_rq(scale_load_down(cfs_rq->load.weight)/h_nr_running/curr!=NULL)、__update_load_avg_blocked_se(load=runnable=running=0 纯衰减)、update_rt_rq_load_avg、update_dl_rq_load_avg、update_thermal_load_avg(用 capacity 做衰减量)、update_irq_load_avg(先以 clock-running 衰减旧量、再以 clock 加新量)。
enqueue_task_fairenqueue_entity→ update_load_avg(cfs_rq, se, UPDATE_TG|DO_ATTACH) | attach_entity_load_avg,对齐 period_contrib 并把 se 的 *_avg 并入 cfs_rq) | |
dequeue_task_fairdequeue_entity→ update_load_avg(..., UPDATE_TG) | util_est_dequeue 减除 | |
put_prev_entityupdate_load_avg(cfs_rq, prev, 0);set_next_entity→update_load_avg(..., UPDATE_TG) | ||
scheduler_ticktask_tick_fair→ entity_tick→ update_load_avg(cfs_rq, curr, UPDATE_TG) | update_misfit_status/update_overutilized_status | |
run_rebalance_domainssched_balance_newidle→ update_blocked_averages(cpu)→ __update_blocked_others+__update_blocked_fair(迭代 leaf cfs_rq 对阻塞实体衰减;cfs_rq_is_decayed收敛后移出 leaf 链表,update_blocked_load_status记录 has_blocked | ||
task_dead_fairmigrate_task_rq_fair→ remove_entity_load_avg | cfs_rq->removed,由下一次 update_cfs_rq_load_avg 批量减除 | |
switched_to_fairenqueue_entity 的 DO_ATTACH → attach_entity_load_avg | se->avg.last_update_time=0 | |
wake_up_new_taskpost_init_entity_util_avg | attach_entity_cfs_rq |
update_rq_clock(rq)→ delta = sched_clock_cpu(cpu) - rq->clock→ rq->clock += delta→ update_rq_clock_task(rq, delta)→ update_irq_load_avg(rq, irq_delta+steal) [CONFIG_HAVE_SCHED_AVG_IRQ]→ update_rq_clock_pelt(rq, delta)├─ rq 空闲: clock_pelt = rq_clock_task(rq)(同步回真实时间)└─ 忙: clock_pelt += cap_scale(cap_scale(delta, cpu_capacity), freq_capacity)
update_idle_rq_clock_pelt 在 rq 进 idle 时若 util_sum(cfs+rt+dl 三者之和)≥ ((LOAD_AVG_MAX-1024)<<10) - LOAD_AVG_MAX 判定"满载无空闲可偷",把差值累计进 lost_idle_time;读取侧 rq_clock_pelt = clock_pelt - lost_idle_time。CFS 带宽下 cfs_rq_clock_pelt 再扣 throttled 时间。
SCHED_SOFTIRQ → run_rebalance_domains→ update_blocked_averages + rebalance_domains→ load_balance→ find_busiest_groupsched_balance_newidle→ update_blocked_averages+ load_balance(..., CPU_NEWLY_IDLE)。nohz_balancer_kick根据 check_cpu_capacity/check_misfit_status/nr_busy_cpus 决定 kick_ilb,ILB CPU 上 nohz_idle_balance代跑。__update_load_avg_se | ||||||
__update_load_avg_cfs_rq | ||||||
__update_load_avg_blocked_se | sync_entity_load_avg) | |||||
update_rt_rq_load_avg | ||||||
update_dl_rq_load_avg | ||||||
update_thermal_load_avg | ||||||
update_irq_load_avg | ||||||
update_load_avg(cfs_rq,se,flags) | ||||||
attach_entity_load_avg | ||||||
detach_entity_load_avg | ||||||
remove_entity_load_avg | ||||||
update_cfs_rq_load_avg | ||||||
update_blocked_averages | idle_cpu 判定门控 | |||||
cpu_util(cpu) | ||||||
cpu_util_without(cpu,p) | ||||||
task_util_est(p) | ||||||
task_h_load(p) | div64_ul(load_avg × h_load, cfs_rq_load_avg + 1) | |||||
rq_clock_pelt | ||||||
cfs_rq_clock_pelt | ||||||
post_init_entity_util_avg |
导出面说明:PELT 无 EXPORT_SYMBOL 给内核模块的通用接口;update_rq_clock/scheduler_tick 属 core.c 内部。模块观测只能通过 tracepoint 或读 cpu_util_cfs 之类(仅内核内部)。用户态无系统调用接口。
update_sg_lb_stats逐 CPU 累加:
sgs->group_load += cpu_load(rq)cfs_rq->avg.load_avg)→ 经典"重量"均衡。sgs->group_util += cpu_util(i)sgs->group_runnable += cpu_runnable(rq)cfs_rq->avg.runnable_avg)。sgs->group_misfit_task_loadrq->misfit_task_load(来自 task_h_load,PELT load_avg×h_load,非大小核处理器不考虑)。*sg_status |= SG_OVERLOAD(nr_running>1)/SG_OVERUTILIZED(cpu_overutilized(i),基于 cpu_util+uclamp)。group_classify→ group_has_capacity/group_is_overloaded直接比较 group_capacity vs group_util/group_runnable(PELT 信号 vs 容量)。
find_busiest_group决策顺序:
sched_energy_enabled() && !READ_ONCE(rd->overutilized)out_balanced:overutilized(PELT util 判定)直接关闭负载均衡,把布置权交给 EAS。group_type == group_misfit_task → 强制均衡,calculate_imbalance设 env->migration_type = migrate_misfit; env->imbalance = 1。group_imbalancedgroup_asym_packing → 强制搬 1 个/全部任务。migrate_util,imbalance = max(local_capacity, local_util) - local_util——直接拿 PELT 利用率计算要搬的"量";migrate_task 时按 idle_cpus 差搬。migrate_load,按 avg_load(PELT load_avg/容量)差额求 min imbalance。detach_tasks按 migration_type 扣减:
migrate_loadload = task_h_load(p)(PELT),shr_bound(load, nr_balance_failed) > env->imbalance 则跳过。migrate_utilutil = task_util_est(p)(PELT+util_est),同样带 shr_bound 判定。migrate_misfittask_fits_cpu(p, src_cpu) 才搬。select_task_rq_fair(p, prev_cpu, sd_flag, wake_flags)├─ EAS(sched_energy_enabled): find_energy_efficient_cpu(p, prev_cpu)│ sync_entity_load_avg(p->se); util = cpu_util_next(cpu,p,cpu); spare_cap = capacity_of - util│ compute_energy() 底层走 schedutil_cpu_util(ENERGY_UTIL) → schedutil_cpu_util│ 条件: rd->overutilized 则 fail(PELT 信号是 EAS 的开关)├─ fast path: wake_affine(WA_WEIGHT 用 cpu_load,即 PELT load_avg)│ └─ 失败 → select_idle_sibling → SIS_AVG_CPU/SIS_PROP 扫描└─ slow path: find_idlest_cpu → find_idlest_group→ update_sg_wakeup_stats: sgs->group_util += cpu_util_without(i,p)sgs->group_runnable += cpu_runnable_without(i,p); group_load += cpu_load_without→ group_classify + update_pick_idlest 按 group_type/avg_load/idle_cpus/group_util 挑最闲组→ find_idlest_group_cpu(idle 优先,否则按 cpu_load 挑最轻)
scheduler_tick├─ update_rq_clock → (PELT 时间推进)├─ update_thermal_load_avg├─ task_tick_fair → entity_tick → update_load_avg [运行实体 PELT 更新]├─ update_misfit_status / update_overutilized_status [PELT 驱动 rq->misfit_task_load / rd->overutilized]└─ trigger_load_balance(rq)├─ time_after_eq(jiffies, rq->next_balance) → raise_softirq(SCHED_SOFTIRQ)└─ nohz_balancer_kick(rq)├─ nohz.has_blocked && 到点 → NOHZ_STATS_KICK├─ rq->nr_running>=2 → NOHZ_BALANCE_KICK├─ check_cpu_capacity(rq,sd)(PELT util vs 容量)→ NOHZ_BALANCE_KICK├─ check_misfit_status(rq,sd)(PELT misfit)→ NOHZ_BALANCE_KICK└─ sds->nr_busy_cpus>1 → NOHZ_BALANCE_KICK→ kick_ilb → 目标 ILB CPU 的 run_rebalance_domains├─ nohz_idle_balance└─ update_blocked_averages + rebalance_domains → load_balance
load_balance → find_busiest_group → update_sd_lb_stats → update_sg_lb_stats→ {cpu_load, cpu_util, cpu_runnable, misfit_task_load}。group_classify → group_is_overloaded/group_has_capacity(group_util/group_runnable)→ update_sd_pick_busiest → calculate_imbalance(migration_type + imbalance)。load_balance → detach_tasks(task_h_load / task_util_est / task_fits_cpu)→ detach_task → deactivate_task → dequeue_task_fair → update_load_avg → attach_tasks → enqueue_task_fair → update_load_avg(DO_ATTACH)。rebalance_domains / sched_balance_newidle → update_blocked_averages→ __update_blocked_fair → update_cfs_rq_load_avg。cpu_overutilized → rd->overutilized;update_misfit_status → rq->misfit_task_load;find_busiest_group/find_energy_efficient_cpu。nohz_balancer_kick → check_cpu_capacity/check_misfit_status → kick_ilb。CONFIG_SMP | ||
sched_features | /sys/kernel/debug/sched_features | UTIL_EST/UTIL_EST_FASTUP/ATTACH_AGE_LOAD/WA_IDLE/WA_WEIGHT/LB_MIN/SIS_PROP/NONTASK_CAPACITY 等(kernel/sched/features.h) |
kernel.sched_energy_aware | /proc/sys/kernel/sched_energy_aware | |
kernel.sched_util_clamp_min/max | /proc/sys/kernel/sched_util_clamp_minsched_util_clamp_max | util_fits_cpu/overutilized |
kernel.sched_migration_cost_nskernel.sched_nr_migrate | /proc/sys/kernel/ | |
kernel.sched_schedstats | /proc/sys/kernel/sched_schedstats |
# 1) tracepoint:PELT 每次更新打点(需定义 trace 或用 ftrace events 动态挂载)# pelt_*_tp 是 DECLARE_TRACE 空钩子(include/trace/events/sched.h),可被# tracepoint_probe_register 挂接,不直接在 tracefs 暴露;# 常规观测用已定义的 trace 事件:cat /sys/kernel/fs/tracing/available_events | grep sched# sched:sched_switch / sched_stat_* 等# 2) debugfs sched_features 开关 UTIL_EST(观察对均衡行为的影响)echo ”NO_UTIL_EST” > /sys/kernel/debug/sched_featuresecho ”UTIL_EST” > /sys/kernel/debug/sched_features# 3) 各 CPU 的 PELT 聚合值(每 CPU debug 文件,需 CONFIG_SCHED_DEBUG)# /sys/kernel/debug/sched/debug 输出每 rq 的 .avg.load_avg / .avg.util_avg / .avg.runnable_avg# 4) 负载均衡统计计数(schedstat)grep -E ”lb_|group_” /proc/schedstat# 5) 每任务 util:/proc//sched 的 se.util_avg / se.runnable_avg / se.avg.load_avggrep -E ”util_avg|load_avg|runnable” /proc//sched# 6) perf:观测 wakeup 选核与均衡perf sched record -- sleep 5 && perf sched latencyperf trace -e sched:sched_wakeup,sched:sched_switch -- sleep 1# 7) 查看 overutilized / nohz 状态cat /sys/kernel/debug/sched/debug | grep -i -E ”overutil|has_blocked|nohz”