]>
Commit | Line | Data |
---|---|---|
4c54005c PM |
1 | Using RCU's CPU Stall Detector |
2 | ||
3 | The CONFIG_RCU_CPU_STALL_DETECTOR kernel config parameter enables | |
4 | RCU's CPU stall detector, which detects conditions that unduly delay | |
5 | RCU grace periods. The stall detector's idea of what constitutes | |
f1d507be | 6 | "unduly delayed" is controlled by a set of C preprocessor macros: |
4c54005c PM |
7 | |
8 | RCU_SECONDS_TILL_STALL_CHECK | |
9 | ||
10 | This macro defines the period of time that RCU will wait from | |
11 | the beginning of a grace period until it issues an RCU CPU | |
f1d507be | 12 | stall warning. This time period is normally ten seconds. |
4c54005c PM |
13 | |
14 | RCU_SECONDS_TILL_STALL_RECHECK | |
15 | ||
16 | This macro defines the period of time that RCU will wait after | |
f1d507be PM |
17 | issuing a stall warning until it issues another stall warning |
18 | for the same stall. This time period is normally set to thirty | |
19 | seconds. | |
4c54005c PM |
20 | |
21 | RCU_STALL_RAT_DELAY | |
22 | ||
f1d507be PM |
23 | The CPU stall detector tries to make the offending CPU print its |
24 | own warnings, as this often gives better-quality stack traces. | |
25 | However, if the offending CPU does not detect its own stall in | |
26 | the number of jiffies specified by RCU_STALL_RAT_DELAY, then | |
27 | some other CPU will complain. This delay is normally set to | |
28 | two jiffies. | |
4c54005c | 29 | |
f1d507be PM |
30 | When a CPU detects that it is stalling, it will print a message similar |
31 | to the following: | |
32 | ||
33 | INFO: rcu_sched_state detected stall on CPU 5 (t=2500 jiffies) | |
34 | ||
35 | This message indicates that CPU 5 detected that it was causing a stall, | |
36 | and that the stall was affecting RCU-sched. This message will normally be | |
37 | followed by a stack dump of the offending CPU. On TREE_RCU kernel builds, | |
38 | RCU and RCU-sched are implemented by the same underlying mechanism, | |
39 | while on TREE_PREEMPT_RCU kernel builds, RCU is instead implemented | |
40 | by rcu_preempt_state. | |
41 | ||
42 | On the other hand, if the offending CPU fails to print out a stall-warning | |
43 | message quickly enough, some other CPU will print a message similar to | |
44 | the following: | |
45 | ||
46 | INFO: rcu_bh_state detected stalls on CPUs/tasks: { 3 5 } (detected by 2, 2502 jiffies) | |
47 | ||
48 | This message indicates that CPU 2 detected that CPUs 3 and 5 were both | |
49 | causing stalls, and that the stall was affecting RCU-bh. This message | |
50 | will normally be followed by stack dumps for each CPU. Please note that | |
51 | TREE_PREEMPT_RCU builds can be stalled by tasks as well as by CPUs, | |
52 | and that the tasks will be indicated by PID, for example, "P3421". | |
53 | It is even possible for a rcu_preempt_state stall to be caused by both | |
54 | CPUs -and- tasks, in which case the offending CPUs and tasks will all | |
55 | be called out in the list. | |
56 | ||
57 | Finally, if the grace period ends just as the stall warning starts | |
58 | printing, there will be a spurious stall-warning message: | |
59 | ||
60 | INFO: rcu_bh_state detected stalls on CPUs/tasks: { } (detected by 4, 2502 jiffies) | |
61 | ||
62 | This is rare, but does happen from time to time in real life. | |
63 | ||
64 | So your kernel printed an RCU CPU stall warning. The next question is | |
65 | "What caused it?" The following problems can result in RCU CPU stall | |
66 | warnings: | |
4c54005c PM |
67 | |
68 | o A CPU looping in an RCU read-side critical section. | |
69 | ||
f1d507be PM |
70 | o A CPU looping with interrupts disabled. This condition can |
71 | result in RCU-sched and RCU-bh stalls. | |
4c54005c | 72 | |
f1d507be PM |
73 | o A CPU looping with preemption disabled. This condition can |
74 | result in RCU-sched stalls and, if ksoftirqd is in use, RCU-bh | |
75 | stalls. | |
76 | ||
77 | o A CPU looping with bottom halves disabled. This condition can | |
78 | result in RCU-sched and RCU-bh stalls. | |
4c54005c PM |
79 | |
80 | o For !CONFIG_PREEMPT kernels, a CPU looping anywhere in the kernel | |
81 | without invoking schedule(). | |
82 | ||
2c96c775 PM |
83 | o A CPU-bound real-time task in a CONFIG_PREEMPT kernel, which might |
84 | happen to preempt a low-priority task in the middle of an RCU | |
85 | read-side critical section. This is especially damaging if | |
86 | that low-priority task is not permitted to run on any other CPU, | |
87 | in which case the next RCU grace period can never complete, which | |
88 | will eventually cause the system to run out of memory and hang. | |
89 | While the system is in the process of running itself out of | |
90 | memory, you might see stall-warning messages. | |
91 | ||
92 | o A CPU-bound real-time task in a CONFIG_PREEMPT_RT kernel that | |
93 | is running at a higher priority than the RCU softirq threads. | |
94 | This will prevent RCU callbacks from ever being invoked, | |
95 | and in a CONFIG_TREE_PREEMPT_RCU kernel will further prevent | |
96 | RCU grace periods from ever completing. Either way, the | |
97 | system will eventually run out of memory and hang. In the | |
98 | CONFIG_TREE_PREEMPT_RCU case, you might see stall-warning | |
99 | messages. | |
100 | ||
4c54005c PM |
101 | o A bug in the RCU implementation. |
102 | ||
103 | o A hardware failure. This is quite unlikely, but has occurred | |
f1d507be | 104 | at least once in real life. A CPU failed in a running system, |
4c54005c PM |
105 | becoming unresponsive, but not causing an immediate crash. |
106 | This resulted in a series of RCU CPU stall warnings, eventually | |
107 | leading the realization that the CPU had failed. | |
108 | ||
f1d507be PM |
109 | The RCU, RCU-sched, and RCU-bh implementations have CPU stall |
110 | warning. SRCU does not have its own CPU stall warnings, but its | |
111 | calls to synchronize_sched() will result in RCU-sched detecting | |
112 | RCU-sched-related CPU stalls. Please note that RCU only detects | |
113 | CPU stalls when there is a grace period in progress. No grace period, | |
114 | no CPU stall warnings. | |
115 | ||
116 | To diagnose the cause of the stall, inspect the stack traces. | |
117 | The offending function will usually be near the top of the stack. | |
118 | If you have a series of stall warnings from a single extended stall, | |
119 | comparing the stack traces can often help determine where the stall | |
120 | is occurring, which will usually be in the function nearest the top of | |
121 | that portion of the stack which remains the same from trace to trace. | |
122 | If you can reliably trigger the stall, ftrace can be quite helpful. | |
4c54005c PM |
123 | |
124 | RCU bugs can often be debugged with the help of CONFIG_RCU_TRACE. |