auditd和kswapd0导致kylin环境内存CPU占用过高
环境信息:
cat /etc/kylin-release Kylin Linux Advanced Server release V10 (Tercel) cat /etc/os-release NAME="Kylin Linux Advanced Server" VERSION="V10 (Tercel)" ID="kylin" VERSION_ID="V10" PRETTY_NAME="Kylin Linux Advanced Server V10 (Tercel)" ANSI_COLOR="0;31" cat /etc/.productinfo Kylin Linux Advanced Server release V10 (SP1) /(Tercel)-aarch64-Build20/20210518
探查发现环境内kswapd0几乎吃满了所有cpu资源,导致执行shell命令都变慢
top top - 13:08:34 up 97 days, 16:39, 2 users, load average: 66.60, 65.91, 79.80 Tasks: 324 total, 1 running, 323 sleeping, 0 stopped, 0 zombie %Cpu(s): 0.3 us, 11.7 sy, 0.0 ni, 0.4 id, 86.9 wa, 0.3 hi, 0.4 si, 0.0 st MiB Mem : 31354.6 total, 1849.8 free, 27422.8 used, 2082.0 buff/cache MiB Swap: 0.0 total, 0.0 free, 0.0 used. 0.0 avail Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 68 root 20 0 0 0 0 S 100.0 0.0 3113:12 kswapd0 1092324 etcd 20 0 10.8g 91968 0 S 2.0 0.3 0:04.62 etcd 1012554 root 20 0 751808 21824 0 D 0.9 0.1 23:43.11 kube-scheduler 1032523 root 20 0 2659520 69760 0 S 0.9 0.2 8:46.04 kubelet 1084081 root 20 0 2456704 103040 0 D 0.9 0.3 0:26.40 linux_client 1085770 root 20 0 814592 21632 0 S 0.9 0.1 0:11.68 kube-controller 1094746 root 20 0 744448 2496 384 D 0.9 0.0 0:00.03 kubectl
通过网上查阅发现,当环境内存不足时,kswapd0会频繁进行换页操作,此操作会导致kswapd0占用过多cpu资源
Linux的进程kswapd0占用CPU过高导致卡顿问题
此时通过top命令和shift+M查到了占用内存资源最多的auditd进程
auditd部分版本可能存在内存泄漏的问题,导致占用过多资源
银河麒麟V10 SP2 auditd服务内存泄露问题
尝试重启auditd进程
systemctl restart auditd
重启后,环境内存和cpu恢复正常。