path: root/kernel/rcu/update.c
diff options
authorPaul E. McKenney <>2014-06-20 16:49:01 -0700
committerPaul E. McKenney <>2014-06-23 11:19:32 -0700
commit4a81e8328d3791a4f99bf5b436d050f6dc5ffea3 (patch)
tree9ffd05b47ffd38639ec904d3329eba1060d3acae /kernel/rcu/update.c
parent546a9d8519ed137b2804a3f5a3659003039dd49c (diff)
rcu: Reduce overhead of cond_resched() checks for RCU
Commit ac1bea85781e (Make cond_resched() report RCU quiescent states) fixed a problem where a CPU looping in the kernel with but one runnable task would give RCU CPU stall warnings, even if the in-kernel loop contained cond_resched() calls. Unfortunately, in so doing, it introduced performance regressions in Anton Blanchard's will-it-scale "open1" test. The problem appears to be not so much the increased cond_resched() path length as an increase in the rate at which grace periods complete, which increased per-update grace-period overhead. This commit takes a different approach to fixing this bug, mainly by moving the RCU-visible quiescent state from cond_resched() to rcu_note_context_switch(), and by further reducing the check to a simple non-zero test of a single per-CPU variable. However, this approach requires that the force-quiescent-state processing send resched IPIs to the offending CPUs. These will be sent only once the grace period has reached an age specified by the boot/sysfs parameter rcutree.jiffies_till_sched_qs, or once the grace period reaches an age halfway to the point at which RCU CPU stall warnings will be emitted, whichever comes first. Reported-by: Dave Hansen <> Signed-off-by: Paul E. McKenney <> Cc: Andi Kleen <> Cc: Christoph Lameter <> Cc: Mike Galbraith <> Cc: Eric Dumazet <> Reviewed-by: Josh Triplett <> [ paulmck: Made rcu_momentary_dyntick_idle() as suggested by the ktest build robot. Also fixed smp_mb() comment as noted by Oleg Nesterov. ] Merge with e552592e (Reduce overhead of cond_resched() checks for RCU) Signed-off-by: Paul E. McKenney <>
Diffstat (limited to 'kernel/rcu/update.c')
1 files changed, 0 insertions, 18 deletions
diff --git a/kernel/rcu/update.c b/kernel/rcu/update.c
index 0fb691e63ce6..bc7883570530 100644
--- a/kernel/rcu/update.c
+++ b/kernel/rcu/update.c
@@ -350,21 +350,3 @@ static int __init check_cpu_stall_init(void)
#endif /* #ifdef CONFIG_RCU_STALL_COMMON */
- * Hooks for cond_resched() and friends to avoid RCU CPU stall warnings.
- */
-DEFINE_PER_CPU(int, rcu_cond_resched_count);
- * Report a set of RCU quiescent states, for use by cond_resched()
- * and friends. Out of line due to being called infrequently.
- */
-void rcu_resched(void)
- preempt_disable();
- __this_cpu_write(rcu_cond_resched_count, 0);
- rcu_note_context_switch(smp_processor_id());
- preempt_enable();