服务器内存不足导致LINUXKILLMYSQLD故障

OS操作系统：SUSE Linux 11

目前创新互联公司已为上千的企业提供了网站建设、域名、网络空间、网站托管、服务器托管、企业网站设计、天祝藏族自治网站维护等服务，公司将坚持客户导向、应用为本的策略，正道将秉承"和谐、参与、激情"的文化，与客户和合作伙伴齐心协力一起成长，共同发展。

数据库版 本：MySQL 5.6

客户反映MYSQL数据库因为内存不足被重启。

1、根据MYSQL报警日志确认问题：

2016-10-24 12:28:49 29226 [Note] Plugin 'FEDERATED' is disabled.

2016-10-24 12:28:49 29226 [Note] InnoDB: Using atomics to ref count buffer pool pages

2016-10-24 12:28:49 29226 [Note] InnoDB: The InnoDB memory heap is disabled

2016-10-24 12:28:49 29226 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins

2016-10-24 12:28:49 29226 [Note] InnoDB: Compressed tables use zlib 1.2.3

2016-10-24 12:28:49 29226 [Note] InnoDB: Using Linux native AIO

2016-10-24 12:28:49 29226 [Note] InnoDB: Using CPU crc32 instructions

2016-10-24 12:28:49 29226 [Note] InnoDB: Initializing buffer pool, size = 2.0G

2016-10-24 12:28:50 29226 [Note] InnoDB: Completed initialization of buffer pool

2016-10-24 12:28:50 29226 [Note] InnoDB: Highest supported file format is Barracuda.

2016-10-24 12:28:50 29226 [Note] InnoDB: Log scan progressed past the checkpoint lsn 27852365485

2016-10-24 12:28:50 29226 [Note] InnoDB: Database was not shutdown normally!

2016-10-24 12:28:50 29226 [Note] InnoDB: Starting crash recovery.

2016-10-24 12:28:50 29226 [Note] InnoDB: Reading tablespace information from the .ibd files...

2016-10-24 12:28:50 29226 [Note] InnoDB: Restoring possible half-written data pages

2016-10-24 12:28:50 29226 [Note] InnoDB: from the doublewrite buffer...

InnoDB: Doing recovery: scanned up to log sequence number 27852370215

2016-10-24 12:28:51 29226 [Note] InnoDB: Starting an apply batch of log records to the database...

InnoDB: Progress in percent: 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99

InnoDB: Apply batch completed

InnoDB: Last MySQL binlog file position 0 401322767, file name TSPTONLDB01-it_mgmad-bin.000023

2016-10-24 12:28:52 29226 [Note] InnoDB: 128 rollback segment(s) are active.

2016-10-24 12:28:52 29226 [Note] InnoDB: 5.6.19 started; log sequence number 27852370215

2016-10-24 12:28:52 29226 [Note] Recovering after a crash using /dbdata/it_mgmad/binlog/it_mgmad/TSPTONLDB01-it_mgmad-bin

2016-10-24 12:29:00 29226 [Note] Starting crash recovery...

2016-10-24 12:29:00 29226 [Note] Crash recovery finished.

2016-10-24 12:29:06 29226 [Note] Server hostname (bind-address): '*'; port: 60002

2016-10-24 12:29:06 29226 [Note] IPv6 is available.

2016-10-24 12:29:06 29226 [Note]  - '::' resolves to '::';

2016-10-24 12:29:06 29226 [Note] Server socket created on IP: '::'.

2016-10-24 12:29:06 29226 [Warning] 'proxies_priv' entry '@ root@tsptonldb01' ignored in --skip-name-resolve mode.

2016-10-24 12:29:06 29226 [Note] Event Scheduler: Loaded 0 events

2016-10-24 12:29:06 29226 [Note] /usr/local/mysql/bin/mysqld: ready for connections.

Version: '5.6.19-upsql-0.1.1-log' socket: '/etc/mysock/it_mgmad.sock' port: 60002 UpSQL Server

2016-10-24 12:30:21 29226 [ERROR] /usr/local/mysql/bin/mysqld: Out of memory (Needed 8160 bytes)

2016-10-24 12:30:21 29226 [ERROR] Out of memory; check if mysqld or some other process uses all available memory; if not, you may have to use 'ulimit' to allow mysqld to use more memory or you can add more swap space

161024 12:31:35 mysqld_safe Number of processes running now: 0

161024 12:31:35 mysqld_safe mysqld restarted

2016-10-24 12:31:37 29659 [Warning] Buffered warning: Could not increase number of max_open_files to more than 10240 (request: 25000)

从上面的信息，我们看到， MSYQL报内存不足的错误无法启动。

2.操作系统日志提示错误

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317509.996482] Mem-Info:

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317509.996483] Node 0 DMA per-cpu:

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317509.996485] CPU   0: hi:   0, btch:  1 usd:  0

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317509.996486] CPU   1: hi:   0, btch:  1 usd:  0

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317509.996487] Node 0 DMA32 per-cpu:

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317509.996488] CPU   0: hi: 186, btch: 31 usd:  0

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317509.996489] CPU    1: hi: 186, btch: 31 usd:  5

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317509.996490] Node 0 Normal per-cpu:

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317509.996491] CPU   0: hi: 186, btch: 31 usd: 30

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317509.996492] CPU   1: hi: 186, btch: 31 usd: 144

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317509.996495] active_anon:658762 inactive_anon:260562 isolated_anon:0

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317509.996496] active_file:353 inactive_file:1716 isolated_file:0

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317509.996496] unevictable:0 dirty:6 writeback:5 unstable:0

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317509.996497] free:21800 slab_reclaimable:2214 slab_unreclaimable:10742

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317509.996498] mapped:765 shmem:456 pagetables:6664 bounce:0

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317509.996499] Node 0 DMA free:15880kB min:256kB low:320kB high:384kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15688kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317509.996505] lowmem_reserve[]: 0 3000 4010 4010

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317509.996507] Node 0 DMA32 free:54356kB min:50368kB low:62960kB high:75552kB active_anon:2275764kB inactive_anon:569032kB active_file:12kB inactive_file:88kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3072160kB mlocked:0kB dirty:0kB writeback:0kB mapped:32kB shmem:12kB slab_reclaimable:1896kB slab_unreclaimable:7060kB kernel_stack:2904kB pagetables:6588kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:611 all_unreclaimable? yes

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317509.996513] lowmem_reserve[]: 0 0 1010 1010

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317509.996515] Node 0 Normal free:16964kB min:16956kB low:21192kB high:25432kB active_anon:359284kB inactive_anon:473216kB active_file:1400kB inactive_file:6776kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:1034240kB mlocked:0kB dirty:24kB writeback:20kB mapped:3028kB shmem:1812kB slab_reclaimable:6960kB slab_unreclaimable:35908kB kernel_stack:3624kB pagetables:20068kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:12878 all_unreclaimable? yes

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317509.996520] lowmem_reserve[]: 0 0 0 0

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317509.996522] Node 0 DMA: 0*4kB 1*8kB 0*16kB 0*32kB 2*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15880kB

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317509.996544] Node 0 DMA32: 1308*4kB 973*8kB 448*16kB 181*32kB 91*64kB 43*128kB 11*256kB 8*512kB 8*1024kB 1*2048kB 0*4096kB = 54456kB

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317509.996549] Node 0 Normal: 439*4kB 303*8kB 169*16kB 83*32kB 18*64kB 5*128kB 4*256kB 3*512kB 3*1024kB 0*2048kB 0*4096kB = 16964kB

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317509.996553] 38343 total pagecache pages

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317509.996554] 35726 pages in swap cache

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317509.996556] Swap cache stats: add 5506519, delete 5470793, find 22920240/23085658

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317509.996557] Free swap = 0kB

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317509.996557] Total swap = 2097148kB

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.006955] 1048560 pages RAM

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.006957] 67010 pages reserved

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.006958] 1978 pages shared

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.006959] 953768 pages non-shared

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.006960] [ pid ]  uid tgid total_vm     rss cpu oom_adj oom_score_adj name

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.006974] [ 571]    0  571    4223       1  1    -17        -1000 udevd

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.006981] [ 1689]    0 1689     998       1  0      0            0 acpid

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.006983] [ 1704]  100 1704    5444     117  0      0            0 dbus-daemon

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.006986] [ 1722]    0 1722    4824      62  0      0            0 syslog-ng

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.006988] [ 1725]    0 1725    1065       0  0      0            0 klogd

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.006991] [10038]    0 10038   31835     168  1      0            0 vmtoolsd

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.006994] [10096]    0 10096   15193       1  0      0            0 VGAuthService

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.006996] [10185]  101 10185    8608     146  0      0            0 hald

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.006999] [10188]    0 10188   27278       0  0      0            0 console-kit-dae

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007001] [10189]    0 10189    4505       2  0      0            0 hald-runner

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007004] [10284]    0 10284    8690       0  0      0            0 hald-addon-inpu

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007007] [10308]    0 10308    8690      17  0      0            0 hald-addon-stor

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007009] [10312]  101 10312    9129       0  1      0            0 hald-addon-acpi

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007012] [10313]    0 10313    8690      16  0      0            0 hald-addon-stor

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007014] [12046]    0 12046    7409       8  0    -17        -1000 auditd

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007017] [12048]    0 12048    3606       0  0      0            0 audispd

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007020] [12081]    0 12081    4723      16  0      0            0 rpcbind

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007022] [12083]    0 12083    2088      17  0      0            0 haveged

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007025] [12140]    0 12140    4216       1  1    -17        -1000 udevd

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007027] [12141]    0 12141    4216       1  0    -17        -1000 udevd

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007029] [12224]    0 12224    6759       1  1      0            0 mcelog

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007031] [12701]    0 12701   17296      28  0    -17        -1000 sshd

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007034] [12715]    0 12715   17933       0  1      0            0 cupsd

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007036] [12730]    0 12730   59557      71  0      0            0 nscd

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007038] [12798]    0 12798    9552      28  0      0            0 master

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007041] [12825]   51 12825   10052      31  1      0            0 qmgr

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007043] [12864]    0 12864   16711      47  1      0            0 gdm

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007046] [12878]    0 12878   27377     121  0      0            0 X

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007048] [12885]    0 12885    3697      15  0      0            0 cron

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007051] [12924]    0 12924    1121       2  1      0            0 mingetty

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007055] [12925]    0 12925    1121       2  1      0            0 mingetty

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007058] [12926]    0 12926    1121       2  0      0            0 mingetty

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007061] [12927]    0 12927    1121       2  1      0            0 mingetty

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007064] [12928]    0 12928    1121       2  1      0            0 mingetty

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007067] [12929]    0 12929    1121       2  0      0            0 mingetty

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007071] [13064]  107 13064    5548       0  1      0            0 dbus-launch

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007074] [13065]  107 13065    3362       0  1      0            0 dbus-daemon

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007077] [13066]  107 13066   39551      82  0      0            0 gnome-session

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007080] [13079]  107 13079    9863      67  0      0            0 gconfd-2

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007083] [13083]  107 13083   31861       0  1      0            0 at-spi-registry

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007086] [13084]  107 13084   70831    2912  0      0            0 gnome-settings-

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007090] [13086]  107 13086   20207       0  1      0            0 bonobo-activati

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007092] [13094]  107 13094   57536     135  0      0            0 gnome-power-man

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007095] [13096]  107 13096   10854       0  0      0            0 gvfsd

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007098] [13097]  107 13097   54246       0  0      0            0 metacity

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007101] [21349] 1018 21349    2857       2  1      0            0 mysqld_safe

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007105] [12777] 1016 12777    2857       2  0      0            0 mysqld_safe

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007109] [18450]    0 18450   13384      49  1      0            0 gdm-simple-slav

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007112] [18452]    0 18452   66809   13462  1      0            0 X

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007115] [18489]  107 18489    5548       0  1      0            0 dbus-launch

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007118] [18518]    0 18518   24613       0  1      0            0 gdm-session-wor

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007121] [18540]    0 18540   39584      89  1      0            0 gnome-session

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007124] [18613]    0 18613    4178      19  0      0            0 gpg-agent

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007126] [18619]    0 18619    5548       0  1      0            0 dbus-launch

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007129] [18620]    0 18620    3429       0  0      0            0 dbus-daemon

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007132] [18623]    0 18623    9887     123  1      0            0 gconfd-2

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007135] [18627]    0 18627   91696     332  0      0            0 gnome-settings-

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007138] [18631]    0 18631   23814       1  1      0            0 gnome-keyring-d

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007141] [18635]    0 18635   10908       0  1      0            0 gvfsd

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007144] [18639]    0 18639   16822      19  0      0            0 gvfs-fuse-daemo

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007147] [18645]    0 18645   51782     400  1      0            0 metacity

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007149] [18650]    0 18650   93659     621  1      0            0 gnome-panel

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007152] [18651]    0 18651  135905   13563  1      0            0 nautilus

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007155] [18653]    0 18653   36624       0  0      0            0 bonobo-activati

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007160] [18659]    0 18659   82105     455  1      0            0 main-menu

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007171] [18662]    0 18662   12083       0  1      0            0 gvfsd-trash

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007174] [18666]    0 18666   13609       1  0      0            0 gvfs-hal-volume

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007178] [18668]    0 18668   15774       1  1      0            0 gvfs-gphoto2-vo

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007181] [18677]    0 18677   75456       1  1      0            0 gnome-volume-co

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007184] [18681]    0 18681   56947       1  0      0            0 python

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007187] [18686]    0 18686   39298     113  0      0            0 vmtoolsd

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007189] [18688]    0 18688   57564     125  0      0            0 gnome-power-man

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007192] [18692]    0 18692   38020     114  1      0            0 gnome-screensav

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007195] [18693]    0 18693   48186       0  1      0            0 pulseaudio

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007198] [18718]    0 18718   10962       0  0      0            0 gvfsd-burn

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007201] [21417]    0 21417   59168     142  1      0            0 application-bro

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007204] [21420]    0 21420   57129     199  0      0            0 baobab

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007208] [21463]    0 21463   35883       1  0      0            0 gnomesu

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007211] [21464]    0 21464   13445       4  0      0            0 gnomesu-pam-bac

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007214] [21471]    0 21471    3221       1  0      0            0 yast2

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007217] [21501]    0 21501  118060     114  0      0            0 y2base

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007221] [21621]    0 21621   55974     123  0      0            0 gnome-terminal

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007224] [21622]    0 21622    1616       0  0      0            0 gnome-pty-helpe

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007227] [21623]    0 21623    3505       1  0      0            0 bash

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007230] [21649]    0 21649    9244       1  0      0            0 gvfsd-metadata

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007233] [31142] 1016 31142 1100222  219744  1      0            0 mysqld

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007236] [31181] 1018 31181 1039509  163992  1      0            0 mysqld

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007239] [ 699] 1017  699    2856      98  0      0            0 mysqld_safe

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007242] [ 1625] 1017 1625 1076677  414298  1      0             0 mysqld

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007246] [ 1861] 1019 1861  641372   48223  1      0            0 java

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007284] [14971]   51 14971   10014     128  0      0            0 pickup

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007287] Out of memory: Kill process 31142 (mysqld) score 283 or sacrifice child

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007358] Killed process 31142 (mysqld) total-vm:4400888kB, anon-rss:878976kB, file-rss:0kB

Oct 25 13:45:45 TSPTONLDB01 kernel: [14317788.351765] vmtoolsd invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0, oom_score_adj=0

Oct 25 13:45:45 TSPTONLDB01 kernel: [14317788.351772] vmtoolsd cpuset=/ mems_allowed=0

Oct 25 13:45:45 TSPTONLDB01 kernel: [14317788.351777] Pid: 10038, comm: vmtoolsd Tainted: G           X 3.0.13-0.27-default #1

Oct 25 13:45:45 TSPTONLDB01 kernel: [14317788.351780] Call Trace:

Oct 25 13:45:45 TSPTONLDB01 kernel: [14317788.351803] [] dump_trace+0x75/0x300

Oct 25 13:45:45 TSPTONLDB01 kernel: [14317788.351814] [] dump_stack+0x69/0x6f

Oct 25 13:45:45 TSPTONLDB01 kernel: [14317788.351825] [] dump_header+0x8f/0x1f0

Oct 25 13:45:45 TSPTONLDB01 kernel: [14317788.351834] [] oom_kill_process+0x95/0x180

Oct 25 13:45:45 TSPTONLDB01 kernel: [14317788.351841] [] out_of_memory+0x14c/0x230

Oct 25 13:45:45 TSPTONLDB01 kernel: [14317788.351848] [] __alloc_pages_slowpath+0x7ab/0x7f0

Oct 25 13:45:45 TSPTONLDB01 kernel: [14317788.351854] [] __alloc_pages_nodemask+0x1b1/0x1c0

Oct 25 13:45:45 TSPTONLDB01 kernel: [14317788.351863] [] alloc_pages_current+0xa5/0x120

Oct 25 13:45:45 TSPTONLDB01 kernel: [14317788.351871] [] filemap_fault+0x172/0x3d0

Oct 25 13:45:45 TSPTONLDB01 kernel: [14317788.351878] [] __do_fault+0x7a/0x5d0

Oct 25 13:45:45 TSPTONLDB01 kernel: [14317788.351884] [] handle_pte_fault+0x9d/0x230

Oct 25 13:45:45 TSPTONLDB01 kernel: [14317788.351893] [] do_page_fault+0x207/0x4c0

Oct 25 13:45:45 TSPTONLDB01 kernel: [14317788.351900] [] page_fault+0x25/0x30

Oct 25 13:45:45 TSPTONLDB01 kernel: [14317788.353040] DWARF2 unwinder stuck at page_fault+0x25/0x30

Oct 25 13:45:45 TSPTONLDB01 kernel: [14317788.353042]

通过日志发现发现在10月24日到11月份出现多次内存不足，操作系统 KILL掉MYSQLD进程的情况。

在该服务器上有3个MYSQL实例，每个MYSQL实例的 INNODB缓冲池设置为2GB，但操作系统的物理内存

只有4GB，另外在服务器上还有一个数据同步的 JAVA进程。结合日志分析，MYSQL内存分配不合理，建议把服务器内存扩展到10GB。

建议MYSQL INNODB池计算方法： INNODB缓冲池内存=操作系统物理内存 -（预留给操作系统的内存1-2GB）