auditd和kswapd0导致kylin环境内存CPU占用过高

auditd和kswapd0导致kylin环境内存CPU占用过高

环境信息:

cat /etc/kylin-release 
Kylin Linux Advanced Server release V10 (Tercel)

cat /etc/os-release
NAME="Kylin Linux Advanced Server"
VERSION="V10 (Tercel)"
ID="kylin"
VERSION_ID="V10"
PRETTY_NAME="Kylin Linux Advanced Server V10 (Tercel)"
ANSI_COLOR="0;31"

cat /etc/.productinfo
Kylin Linux Advanced Server
release V10 (SP1) /(Tercel)-aarch64-Build20/20210518

探查发现环境内kswapd0几乎吃满了所有cpu资源,导致执行shell命令都变慢

top
top - 13:08:34 up 97 days, 16:39,  2 users,  load average: 66.60, 65.91, 79.80
Tasks: 324 total,   1 running, 323 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.3 us, 11.7 sy,  0.0 ni,  0.4 id, 86.9 wa,  0.3 hi,  0.4 si,  0.0 st
MiB Mem :  31354.6 total,   1849.8 free,  27422.8 used,   2082.0 buff/cache
MiB Swap:      0.0 total,      0.0 free,      0.0 used.      0.0 avail Mem 

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND                                                                                                                                
     68 root      20   0       0      0      0 S 100.0   0.0   3113:12 kswapd0                                                                                                                                
1092324 etcd      20   0   10.8g  91968      0 S   2.0   0.3   0:04.62 etcd                                                                                                                                   
1012554 root      20   0  751808  21824      0 D   0.9   0.1  23:43.11 kube-scheduler                                                                                                                         
1032523 root      20   0 2659520  69760      0 S   0.9   0.2   8:46.04 kubelet                                                                                                                                
1084081 root      20   0 2456704 103040      0 D   0.9   0.3   0:26.40 linux_client                                                                                                                           
1085770 root      20   0  814592  21632      0 S   0.9   0.1   0:11.68 kube-controller                                                                                                                        
1094746 root      20   0  744448   2496    384 D   0.9   0.0   0:00.03 kubectl      

通过网上查阅发现,当环境内存不足时,kswapd0会频繁进行换页操作,此操作会导致kswapd0占用过多cpu资源
Linux的进程kswapd0占用CPU过高导致卡顿问题

此时通过top命令和shift+M查到了占用内存资源最多的auditd进程

auditd部分版本可能存在内存泄漏的问题,导致占用过多资源
银河麒麟V10 SP2 auditd服务内存泄露问题

尝试重启auditd进程

systemctl restart auditd

重启后,环境内存和cpu恢复正常。