J’ai dû le redémarrer de force car il ne répondait plus du tout (ping, ssh, complètement HS).
Il avait un uptime d’une vingtaine de jours lorsque je l’ai redémarré.
Ce n’est pas la première fois que cela arrive (pas de cause concrète, ici c’est htop mais ça peut arriver n’importe comment).
Le matériel est plutôt récent, voire neuf (plus de précisions si nécessaire).
En regardant dans les logs, et plus précisément le fichier kern.log.1, on trouve des dizaines de lignes relatives à cet incident. Problème: je n’y comprends rien!
Merci d’avance à toute personne susceptible de m’aider à déchiffrer ce log et identifier la cause (matériel? software?).
Actuellement, je n’arrive plus à faire tourner cette machine plus d’une vingtaine de jours sans qu’elle plante. Je commence à douter de la stabilité légendaire de debian
Voici le log en entier (en deux messages, car j’atteins la limite):
Ca pourrait aider de connaitre le noyau, pour régler un problème de noyau…
Ca ne serait pas un noyau preempt ?
Quel usage pour la machine (serveur ? desktop ?).
D’ailleurs, as tu essayé de booter sur un autre noyau, pour savoir si ce n’est pas le noyau actuel qui est bugué ?
Dans tous les cas de plantage, c’est toujours le même CPU qui cause le « soft lockup » ?
Ca fait un peu plus de 15j.
D’où ma question: est ce que ça fait à peu prés le même intervalle entre chaque plantage ?
As tu accés physique à la machine ? As tu moyen d’executer des choses dans une console ?
Toutes les "Call Trace"s sont identique, et on voit que le proc rentre dans un _raw_spin_lock (la boucle du soft lockup) au moment d’un proc_pid_make_inode, il me semble.
Mais je t’avoue que je ne sais pas trop à quoi ça correspond, en tout cas, pas à un probléme de module foireux (pas les mêmes messages a priori).
Ah non, ça, c’est soit le matos, soit le kernel, mais ce n’est pas debian.
Oui, effectivement un peu plus de 15 j. Jusqu’ici je n’avais pas trop creusé, aujourd’hui en regardant les logs kern.log, je constate que j’ai eu le même problème le 17 février et le 4 décembre. Je ne me souviens pas précisément si j’ai dû rebooter de force la machine pour sortir du problème.
Trouvaille intéressante: il s’agissait à chaque fois d’un problème avec htop:
cat kern.log* grep -i "soft lockup"
Mar 4 19:53:26 home-server kernel: [1317256.350378] NMI watchdog: BUG: soft lockup - CPU#5 stuck for 22s! [htop:13364]
Mar 4 19:53:54 home-server kernel: [1317284.352266] NMI watchdog: BUG: soft lockup - CPU#5 stuck for 22s! [htop:13364]
Mar 4 19:54:30 home-server kernel: [1317320.354691] NMI watchdog: BUG: soft lockup - CPU#5 stuck for 23s! [htop:13364]
Mar 4 19:54:58 home-server kernel: [1317348.356576] NMI watchdog: BUG: soft lockup - CPU#5 stuck for 23s! [htop:13364]
Mar 4 19:55:34 home-server kernel: [1317384.359000] NMI watchdog: BUG: soft lockup - CPU#5 stuck for 22s! [htop:13364]
Mar 4 19:56:02 home-server kernel: [1317412.360884] NMI watchdog: BUG: soft lockup - CPU#5 stuck for 22s! [htop:13364]
Mar 4 19:56:38 home-server kernel: [1317448.363304] NMI watchdog: BUG: soft lockup - CPU#5 stuck for 22s! [htop:13364]
Dec 5 23:04:55 home-server kernel: [181207.357589] NMI watchdog: BUG: soft lockup - CPU#7 stuck for 22s! [htop:6630]
Dec 5 23:05:23 home-server kernel: [181235.358135] NMI watchdog: BUG: soft lockup - CPU#7 stuck for 22s! [htop:6630]
Dec 5 23:05:59 home-server kernel: [181271.358838] NMI watchdog: BUG: soft lockup - CPU#7 stuck for 23s! [htop:6630]
Dec 5 23:06:27 home-server kernel: [181299.359384] NMI watchdog: BUG: soft lockup - CPU#7 stuck for 23s! [htop:6630]
Dec 5 23:07:03 home-server kernel: [181335.359811] NMI watchdog: BUG: soft lockup - CPU#7 stuck for 22s! [htop:6630]
Dec 5 23:07:31 home-server kernel: [181363.360137] NMI watchdog: BUG: soft lockup - CPU#7 stuck for 22s! [htop:6630]
Dec 5 23:08:03 home-server kernel: [181395.360520] NMI watchdog: BUG: soft lockup - CPU#7 stuck for 22s! [htop:6630]
Dec 5 23:08:31 home-server kernel: [181423.360866] NMI watchdog: BUG: soft lockup - CPU#7 stuck for 22s! [htop:6630]
Dec 5 23:09:07 home-server kernel: [181459.361324] NMI watchdog: BUG: soft lockup - CPU#7 stuck for 23s! [htop:6630]
Dec 5 23:09:35 home-server kernel: [181487.361689] NMI watchdog: BUG: soft lockup - CPU#7 stuck for 23s! [htop:6630]
Dec 5 23:10:11 home-server kernel: [181523.362172] NMI watchdog: BUG: soft lockup - CPU#7 stuck for 22s! [htop:6630]
Dec 5 23:10:15 home-server kernel: [181527.350227] NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [top:6768]
Dec 5 23:10:39 home-server kernel: [181551.362555] NMI watchdog: BUG: soft lockup - CPU#7 stuck for 22s! [htop:6630]
Dec 5 23:10:43 home-server kernel: [181555.350610] NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [top:6768]
Dec 5 23:11:11 home-server kernel: [181583.351001] NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [top:6768]
Dec 5 23:11:15 home-server kernel: [181587.363058] NMI watchdog: BUG: soft lockup - CPU#7 stuck for 22s! [htop:6630]
Dec 5 23:11:39 home-server kernel: [181611.351399] NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [top:6768]
Dec 5 23:12:07 home-server kernel: [181615.363457] NMI watchdog: BUG: soft lockup - CPU#7 stuck for 22s! [htop:6630]
Dec 5 23:12:08 home-server kernel: [181639.351803] NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [top:6768]
Dec 5 23:12:20 home-server kernel: [181647.363919] NMI watchdog: BUG: soft lockup - CPU#7 stuck for 23s! [htop:6630]
Dec 5 23:12:35 home-server kernel: [181667.352212] NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [top:6768]
Dec 5 23:12:43 home-server kernel: [181675.364330] NMI watchdog: BUG: soft lockup - CPU#7 stuck for 22s! [htop:6630]
Dec 5 23:13:03 home-server kernel: [181695.352629] NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [top:6768]
Dec 5 23:13:19 home-server kernel: [181711.364868] NMI watchdog: BUG: soft lockup - CPU#7 stuck for 22s! [htop:6630]
Dec 5 23:13:31 home-server kernel: [181723.353049] NMI watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [top:6768]
Dec 5 23:13:47 home-server kernel: [181739.365292] NMI watchdog: BUG: soft lockup - CPU#7 stuck for 22s! [htop:6630]
Dec 5 23:13:59 home-server kernel: [181751.353475] NMI watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [top:6768]
Dec 5 23:14:23 home-server kernel: [181775.365843] NMI watchdog: BUG: soft lockup - CPU#7 stuck for 22s! [htop:6630]
Dec 5 23:14:27 home-server kernel: [181779.353905] NMI watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [top:6768]
Dec 5 23:14:51 home-server kernel: [181803.366278] NMI watchdog: BUG: soft lockup - CPU#7 stuck for 22s! [htop:6630]
Dec 5 23:14:55 home-server kernel: [181807.354341] NMI watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [top:6768]
Dec 5 23:15:23 home-server kernel: [181835.354779] NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [top:6768]
Dec 5 23:15:27 home-server kernel: [181839.366842] NMI watchdog: BUG: soft lockup - CPU#7 stuck for 23s! [htop:6630]
Dec 5 23:15:51 home-server kernel: [181863.355222] NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [top:6768]
Dec 5 23:15:55 home-server kernel: [181867.367286] NMI watchdog: BUG: soft lockup - CPU#7 stuck for 23s! [htop:6630]
Dec 5 23:16:19 home-server kernel: [181891.355669] NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [top:6768]
Dec 5 23:16:27 home-server kernel: [181899.367797] NMI watchdog: BUG: soft lockup - CPU#7 stuck for 22s! [htop:6630]
Dec 5 23:16:47 home-server kernel: [181919.356121] NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [top:6768]
Dec 5 23:16:55 home-server kernel: [181927.368248] NMI watchdog: BUG: soft lockup - CPU#7 stuck for 23s! [htop:6630]
Dec 5 23:17:15 home-server kernel: [181947.356573] NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [top:6768]
Dec 5 23:17:31 home-server kernel: [181963.368834] NMI watchdog: BUG: soft lockup - CPU#7 stuck for 22s! [htop:6630]
Dec 5 23:17:43 home-server kernel: [181975.357030] NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [top:6768]
Feb 17 12:25:29 home-server kernel: [701367.120480] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [htop:13245]
Feb 17 12:25:57 home-server kernel: [701395.123473] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [htop:13245]
Feb 17 12:26:29 home-server kernel: [701427.126892] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [htop:13245]
Feb 17 12:26:57 home-server kernel: [701455.129883] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [htop:13245]
Feb 17 12:27:33 home-server kernel: [701491.133728] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [htop:13245]
Feb 17 12:28:01 home-server kernel: [701519.136717] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [htop:13245]
Feb 17 12:28:37 home-server kernel: [701555.140559] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [htop:13245]
Feb 17 12:29:05 home-server kernel: [701583.143547] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [htop:13245]
Feb 17 12:29:41 home-server kernel: [701619.147387] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [htop:13245]
Feb 17 12:30:09 home-server kernel: [701647.150373] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [htop:13245]
Feb 17 12:30:41 home-server kernel: [701679.153785] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [htop:13245]
Feb 17 12:31:09 home-server kernel: [701707.156770] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [htop:13245]
Feb 17 12:31:45 home-server kernel: [701743.160607] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [htop:13245]
Feb 17 12:32:13 home-server kernel: [701771.163591] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [htop:13245]
Feb 17 12:32:49 home-server kernel: [701807.167426] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [htop:13245]
Feb 17 12:33:17 home-server kernel: [701835.170409] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [htop:13245]
Feb 17 12:33:53 home-server kernel: [701871.174243] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [htop:13245]
Feb 17 12:34:21 home-server kernel: [701899.177225] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [htop:13245]
Feb 17 12:34:53 home-server kernel: [701931.180632] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [htop:13245]
Feb 17 12:35:21 home-server kernel: [701959.183613] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [htop:13245]
Feb 17 12:35:57 home-server kernel: [701995.187446] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [htop:13245]
Feb 17 12:36:25 home-server kernel: [702023.190426] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [htop:13245]
Feb 17 12:37:01 home-server kernel: [702059.194258] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [htop:13245]
Feb 17 12:37:29 home-server kernel: [702087.197238] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [htop:13245]
Feb 17 12:38:05 home-server kernel: [702123.201069] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [htop:13245]
Feb 17 12:38:33 home-server kernel: [702151.204048] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [htop:13245]
Feb 17 12:39:05 home-server kernel: [702183.207452] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [htop:13245]
Feb 17 12:39:33 home-server kernel: [702211.210431] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [htop:13245]
Feb 17 12:40:09 home-server kernel: [702247.214261] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [htop:13245]
Feb 17 12:40:37 home-server kernel: [702275.217239] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [htop:13245]
Feb 17 12:41:13 home-server kernel: [702311.221068] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [htop:13245]
Feb 17 12:41:41 home-server kernel: [702339.224046] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [htop:13245]
Feb 17 12:42:17 home-server kernel: [702375.227875] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [htop:13245]
Feb 17 12:42:45 home-server kernel: [702403.230853] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [htop:13245]
Feb 17 12:43:17 home-server kernel: [702435.234256] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [htop:13245]
Feb 17 12:43:45 home-server kernel: [702463.237233] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [htop:13245]
Feb 17 12:44:21 home-server kernel: [702499.241061] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [htop:13245]
Feb 17 12:44:49 home-server kernel: [702527.244038] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [htop:13245]
Feb 17 12:45:25 home-server kernel: [702563.247866] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [htop:13245]
Feb 17 12:45:53 home-server kernel: [702591.250843] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [htop:13245]
Feb 17 12:46:29 home-server kernel: [702627.254670] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [htop:13245]
Feb 17 12:46:57 home-server kernel: [702655.257647] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [htop:13245]
Feb 17 12:47:29 home-server kernel: [702687.261048] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [htop:13245]
Feb 17 12:47:57 home-server kernel: [702715.264025] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [htop:13245]
Feb 17 12:48:33 home-server kernel: [702751.267852] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [htop:13245]
Feb 17 12:49:01 home-server kernel: [702779.270828] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [htop:13245]
Feb 17 12:49:37 home-server kernel: [702815.274655] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [htop:13245]
Feb 17 12:50:05 home-server kernel: [702843.277632] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [htop:13245]
Feb 17 12:50:41 home-server kernel: [702879.281458] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [htop:13245]
Feb 17 12:51:09 home-server kernel: [702907.284434] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [htop:13245]
Feb 17 12:51:41 home-server kernel: [702939.287835] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [htop:13245]
Feb 17 12:52:09 home-server kernel: [702967.290811] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [htop:13245]
Feb 17 12:52:45 home-server kernel: [703003.294637] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [htop:13245]
Feb 17 12:53:13 home-server kernel: [703031.297613] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [htop:13245]
Feb 17 12:53:49 home-server kernel: [703067.301439] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [htop:13245]
Feb 17 12:54:17 home-server kernel: [703095.304428] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [htop:13245]
Feb 17 12:54:53 home-server kernel: [703131.308454] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [htop:13245]
Feb 17 12:55:21 home-server kernel: [703159.311577] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [htop:13245]
Feb 17 12:55:53 home-server kernel: [703191.315139] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [htop:13245]
Feb 17 12:56:21 home-server kernel: [703219.318249] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [htop:13245]
Feb 17 12:56:57 home-server kernel: [703255.322238] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [htop:13245]
Feb 17 12:57:25 home-server kernel: [703283.325335] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [htop:13245]
Feb 17 12:58:01 home-server kernel: [703319.329309] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [htop:13245]
Feb 17 12:58:29 home-server kernel: [703347.332395] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [htop:13245]
Feb 17 12:59:05 home-server kernel: [703383.336356] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [htop:13245]
Feb 17 12:59:33 home-server kernel: [703411.339431] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [htop:13245]
Feb 17 13:00:05 home-server kernel: [703443.342941] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [htop:13245]
Feb 17 13:00:33 home-server kernel: [703471.346008] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [htop:13245]
Feb 17 13:01:09 home-server kernel: [703507.349947] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [htop:13245]
Feb 17 13:01:37 home-server kernel: [703535.353006] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [htop:13245]
Feb 17 13:02:13 home-server kernel: [703571.356935] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [htop:13245]
Feb 17 13:02:41 home-server kernel: [703599.359988] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [htop:13245]
Feb 17 13:03:17 home-server kernel: [703635.363909] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [htop:13245]
Feb 17 13:03:45 home-server kernel: [703663.366955] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [htop:13245]
Feb 17 13:04:17 home-server kernel: [703695.370434] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [htop:13245]
Feb 17 13:04:45 home-server kernel: [703723.373475] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [htop:13245]
Feb 17 13:05:21 home-server kernel: [703759.377382] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [htop:13245]
Feb 17 13:05:49 home-server kernel: [703787.380419] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [htop:13245]
Feb 17 13:06:25 home-server kernel: [703823.384320] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [htop:13245]
Feb 17 13:06:53 home-server kernel: [703851.387353] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [htop:13245]
Feb 17 13:07:29 home-server kernel: [703887.391249] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [htop:13245]
Feb 17 13:07:57 home-server kernel: [703915.394277] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [htop:13245]
Feb 17 13:08:33 home-server kernel: [703951.398169] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [htop:13245]
Feb 17 13:09:01 home-server kernel: [703979.401194] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [htop:13245]
Feb 17 13:09:33 home-server kernel: [704011.404650] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [htop:13245]
Feb 17 13:10:01 home-server kernel: [704039.407673] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [htop:13245]
Feb 17 13:10:37 home-server kernel: [704075.411557] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [htop:13245]
Feb 17 13:11:05 home-server kernel: [704103.414578] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [htop:13245]
Feb 17 13:11:41 home-server kernel: [704139.418459] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [htop:13245]
Feb 17 13:12:09 home-server kernel: [704167.421477] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [htop:13245]
Feb 17 13:12:45 home-server kernel: [704203.425355] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [htop:13245]
Feb 17 13:13:13 home-server kernel: [704231.428371] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [htop:13245]
Feb 17 13:13:45 home-server kernel: [704263.431817] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [htop:13245]
Feb 17 13:14:13 home-server kernel: [704291.434831] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [htop:13245]
Feb 17 13:14:49 home-server kernel: [704327.438705] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [htop:13245]
Feb 17 13:15:17 home-server kernel: [704355.441717] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [htop:13245]
Feb 17 13:15:53 home-server kernel: [704391.445590] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [htop:13245]
Feb 17 13:16:21 home-server kernel: [704419.448601] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [htop:13245]
Feb 17 13:16:57 home-server kernel: [704455.452471] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [htop:13245]
Feb 17 13:17:25 home-server kernel: [704483.455481] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [htop:13245]
Feb 17 13:17:57 home-server kernel: [704515.458920] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [htop:13245]
Feb 17 13:18:25 home-server kernel: [704543.461929] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [htop:13245]
Feb 17 13:19:01 home-server kernel: [704579.465797] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [htop:13245]
Feb 17 13:19:29 home-server kernel: [704607.468805] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [htop:13245]
Feb 17 13:20:05 home-server kernel: [704643.472672] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [htop:13245]
Feb 17 13:20:33 home-server kernel: [704671.475679] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [htop:13245]
Feb 17 13:21:09 home-server kernel: [704707.479544] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [htop:13245]
Feb 17 13:21:37 home-server kernel: [704735.482551] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [htop:13245]
Feb 17 13:22:09 home-server kernel: [704767.485986] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [htop:13245]
Feb 17 13:22:37 home-server kernel: [704795.488992] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [htop:13245]
Feb 17 13:23:13 home-server kernel: [704831.492856] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [htop:13245]
Feb 17 13:23:41 home-server kernel: [704859.495861] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [htop:13245]
Feb 17 13:24:17 home-server kernel: [704895.499724] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [htop:13245]
Feb 17 13:24:45 home-server kernel: [704923.502729] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [htop:13245]
Feb 17 13:25:21 home-server kernel: [704959.506591] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [htop:13245]
Feb 17 13:25:49 home-server kernel: [704987.509595] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [htop:13245]
Feb 17 13:26:21 home-server kernel: [705019.513028] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [htop:13245]
Feb 17 13:26:49 home-server kernel: [705047.516032] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [htop:13245]
Feb 17 13:27:25 home-server kernel: [705083.519894] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [htop:13245]
Feb 17 13:27:53 home-server kernel: [705111.522897] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [htop:13245]
Feb 17 13:28:29 home-server kernel: [705147.526679] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [htop:13245]
Feb 17 13:28:57 home-server kernel: [705175.529328] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [htop:13245]
Feb 17 13:29:33 home-server kernel: [705211.532759] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [htop:13245]
Feb 17 13:30:01 home-server kernel: [705239.535444] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [htop:13245]
Feb 17 13:30:33 home-server kernel: [705271.538532] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [htop:13245]
Feb 17 13:31:01 home-server kernel: [705299.541248] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [htop:13245]
Feb 17 13:31:37 home-server kernel: [705335.544759] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [htop:13245]
Feb 17 13:32:05 home-server kernel: [705363.547504] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [htop:13245]
Feb 17 13:32:41 home-server kernel: [705399.551050] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [htop:13245]
Feb 17 13:33:09 home-server kernel: [705427.553820] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [htop:13245]
Feb 17 13:33:45 home-server kernel: [705463.557397] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [htop:13245]
Feb 17 13:34:13 home-server kernel: [705491.560189] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [htop:13245]
Feb 17 13:34:45 home-server kernel: [705523.563392] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [htop:13245]
Feb 17 13:35:13 home-server kernel: [705551.566203] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [htop:13245]
Feb 17 13:35:49 home-server kernel: [705587.569829] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [htop:13245]
Feb 17 13:36:17 home-server kernel: [705615.572657] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [htop:13245]
Feb 17 13:36:53 home-server kernel: [705651.576305] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [htop:13245]
Feb 17 13:37:21 home-server kernel: [705679.579149] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [htop:13245]
Feb 17 13:37:57 home-server kernel: [705715.582814] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [htop:13245]
Feb 17 13:38:25 home-server kernel: [705743.585672] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [htop:13245]
Feb 17 13:38:57 home-server kernel: [705775.588945] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [htop:13245]
Feb 17 13:39:25 home-server kernel: [705803.591814] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [htop:13245]
Feb 17 13:40:01 home-server kernel: [705839.595510] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [htop:13245]
Feb 17 13:40:29 home-server kernel: [705867.598390] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [htop:13245]
Feb 17 13:41:05 home-server kernel: [705903.602099] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [htop:13245]
Feb 17 13:41:33 home-server kernel: [705931.604988] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [htop:13245]
Feb 17 13:42:09 home-server kernel: [705967.608708] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [htop:13245]
Feb 17 13:42:37 home-server kernel: [705995.611606] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [htop:13245]
Feb 17 13:43:09 home-server kernel: [706027.614922] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [htop:13245]
Feb 17 13:43:37 home-server kernel: [706055.617826] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [htop:13245]
Feb 17 13:44:13 home-server kernel: [706091.621566] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [htop:13245]
Feb 17 13:44:41 home-server kernel: [706119.624476] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [htop:13245]
Feb 17 13:45:17 home-server kernel: [706155.628223] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [htop:13245]
Feb 17 13:45:45 home-server kernel: [706183.631140] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [htop:13245]
Feb 17 13:46:21 home-server kernel: [706219.634894] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [htop:13245]
Feb 17 13:46:49 home-server kernel: [706247.637816] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [htop:13245]
Feb 17 13:47:21 home-server kernel: [706279.641158] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [htop:13245]
Feb 17 13:47:49 home-server kernel: [706307.644084] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [htop:13245]
Feb 17 13:48:25 home-server kernel: [706343.647849] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [htop:13245]
Feb 17 13:48:53 home-server kernel: [706371.650779] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [htop:13245]
Feb 17 13:49:29 home-server kernel: [706407.654549] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [htop:13245]
Feb 17 13:49:57 home-server kernel: [706435.657482] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [htop:13245]
Feb 17 13:50:33 home-server kernel: [706471.661257] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [htop:13245]
Feb 17 13:51:01 home-server kernel: [706499.664194] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [htop:13245]
Feb 17 13:51:33 home-server kernel: [706531.667551] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [htop:13245]
Feb 17 13:52:01 home-server kernel: [706559.670491] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [htop:13245]
Feb 17 13:52:37 home-server kernel: [706595.674272] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [htop:13245]
Feb 17 13:53:05 home-server kernel: [706623.677214] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [htop:13245]
Feb 17 13:53:41 home-server kernel: [706659.680998] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [htop:13245]
Feb 17 13:54:09 home-server kernel: [706687.683942] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [htop:13245]
Feb 17 13:54:45 home-server kernel: [706723.687728] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [htop:13245]
Feb 17 13:55:13 home-server kernel: [706751.690674] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [htop:13245]
Feb 17 13:55:45 home-server kernel: [706783.694042] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [htop:13245]
Feb 17 13:56:13 home-server kernel: [706811.696989] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [htop:13245]
Feb 17 13:56:49 home-server kernel: [706847.700780] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [htop:13245]
Feb 17 13:57:17 home-server kernel: [706875.703729] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [htop:13245]
Feb 17 13:57:53 home-server kernel: [706911.707522] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [htop:13245]
Feb 17 13:58:21 home-server kernel: [706939.710472] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [htop:13245]
Feb 17 13:58:57 home-server kernel: [706975.714266] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [htop:13245]
Feb 17 13:59:25 home-server kernel: [707003.717218] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [htop:13245]
Oui, mais le problème c’est que je n’ai pas d’écran connecté et le serveur n’est pas très pratique d’accès…
Non, apparemment 1, 3, 5, et 7 (cf. ci-dessus logs).
Celle sur laquelle tu tournes a été compilée le 02/02/2019, donc j’imagine qu’avant tu tournais sur la version précédente.
Or il y avait déjà des problèmes en décembre, avec ton autre noyau.
Il faudrait essayer avec une version de noyau vraiment différente.
Je tourne personnellement avec un noyau linux-image-4.19.0-0.bpo.2-amd64 venu de stretch-backports:
sa stabilité mérite peut être d’être testée, en attendant de savoir d’où vient l’instabilité sur le 4.9.
Il y a aussi du top, intéressant et cohérent avec la stack trace qui parle de proc_pid_lookup à chaque fois: ce sont deux commandes qui consultent fortement les PID.
Mais j’arrive à mes limites, à part te recommander de tester un autre noyau pour résoudre le problème, je ne vois pas trop quoi faire de plus:
ok, il y a un problème quand *top tente de verrouiller un truc sur les PIDs, mais bon, d’où ça vient…
Je vais essayer un noyau plus récent dès que j’ai un moment pour le faire. Le problème, c’est que ça peut être stable pendant des jours avant de planter donc le test risque d’être long
Bon pour le moment j’ai un autre souci… Qui s’était déjà présenté à deux reprises dernièrement. Le système s’est remonté en read-only suite à une erreur de checksum sur une inode:
[28944.730610] EXT4-fs error (device dm-1): ext4_iget:4527: inode #109320614: comm du: checksum invalid
[28944.773196] Aborting journal on device dm-1-8.
[28944.831727] EXT4-fs (dm-1): Remounting filesystem read-only
[28944.874572] EXT4-fs error (device dm-1) in ext4_do_update_inode:4975: Journal has aborted
[28944.906541] EXT4-fs error (device dm-1) in ext4_da_write_end:3076: IO failure
Est-ce que ce serait le disque qui est en train de mourir? Pourtant il n’est pas très vieux.
En faisant quelques recherches sur le net, j’ai vu des suggestions de supprimer la fonction metadata_csum pour régler ce problème:
Bon, avant de crier au loup, possibilité de reconstruction du journal:
systemctl isolate rescue.target
remount -n -o remount,ro /dev/dm-1 #j ai un doute sur le vrai nom de la partition, mais bon
fsck -y /dev/dm-1
tune2fs -O ^has_journal /dev/dm-1 #hop on transforme l ext4 en ext3
mount -n -o remount,rw /dev/dm-1
tune2fs -j /dev/dm-1 #on en refait une ext4
remount -n -o remount,ro /dev/dm-1
fsck -f /dev/dm-1
reboot
MAIS
Bizarrement,on retrouve cette histoire d’inode, qu’il y avait déjà sur le soft lockup.
Serait ce lié ?
que dit df -i ?
Je réalise que je n’étais jamais revenu sur cette discussion (honte à moi).
Bilan: passer au noyau linux-image-4.19.0-0.bpo.2-amd64 comme le suggérait @mattotop a résolu le problème (dont la cause restera inconnue).
Pour le problème du disque, il n’est plus reproduit. De mémoire, la commande df -i ne retournait pas de résultats alarmants.
à l’aide de la petite coche que tu trouvera en cliquant sur les trois petits points d’une ligne ayant effectivement répondu à ton problème je viens de le faire pour toi.