前段時(shí)間，有測(cè)試人員聯(lián)系我，尋求幫助，讓我?guī)兔纯?#xff0c;測(cè)試環(huán)境的一臺(tái)mysql數(shù)據(jù)庫(kù)不斷的重啟，導(dǎo)致他們的測(cè)試無(wú)法進(jìn)行，時(shí)間拖的長(zhǎng)了，會(huì)影響上線進(jìn)度。

下面就來(lái)說(shuō)說(shuō)現(xiàn)象

在一個(gè)測(cè)試環(huán)境，mysql5.7部署在centos7.4的系統(tǒng)上，測(cè)試人員用LoadRunner進(jìn)行應(yīng)用的一個(gè)壓測(cè)測(cè)試，壓測(cè)并發(fā)為128，啟動(dòng)LoadRunner過(guò)幾秒，LoadRunner日志就報(bào)錯(cuò)，顯示mysql服務(wù)斷開(kāi)了，過(guò)幾秒又恢復(fù)，然后又?jǐn)嚅_(kāi)，異常詭異。

排查定位

于是就登錄到服務(wù)器，看一下數(shù)據(jù)庫(kù)服務(wù)器的內(nèi)存，內(nèi)存已經(jīng)用完了，緊接著就看centos的系統(tǒng)日志(/var/log/messages)，在這個(gè)日志里發(fā)現(xiàn)mysql服務(wù)因?yàn)镺OM，導(dǎo)致進(jìn)程被系統(tǒng)給kill掉了，然后因?yàn)閙ysql服務(wù)有守護(hù)進(jìn)程存在，又被自動(dòng)啟動(dòng)了。

一個(gè)經(jīng)驗(yàn)豐富的mysql數(shù)據(jù)庫(kù)運(yùn)維人員，應(yīng)該很快就知道什么原因?qū)е翺OM了。
第一：mysql數(shù)據(jù)庫(kù)的buffer pool內(nèi)存參數(shù)配置不合理
第二：mysql數(shù)據(jù)庫(kù)的session初始化內(nèi)存參數(shù)配置不合理
第三：數(shù)據(jù)庫(kù)連接總數(shù)配置不合理

測(cè)試環(huán)境的數(shù)據(jù)庫(kù)服務(wù)器內(nèi)存為4G，在這里就不談，為什么測(cè)試人員要在這個(gè)服務(wù)器上做壓測(cè)，其實(shí)做應(yīng)用壓測(cè)，應(yīng)該要用準(zhǔn)用的壓測(cè)環(huán)境，而不是隨便找個(gè)環(huán)境就壓測(cè)，不展開(kāi)說(shuō)了。

模擬復(fù)盤

測(cè)試環(huán)境
mysql5.7，centos7.7，內(nèi)存2G

為了更好的模擬復(fù)盤上述故障，這里關(guān)閉掉模擬環(huán)境的swap，關(guān)閉步驟如下所示

[root@localhost data]# free -m total used free shared buff/cache availableMem: 2124 565 640 9 917 1398Swap: 2047 0 2047[root@localhost data]# swapon -s文件名 類型 大小 已用 權(quán)限/dev/dm-1 partition 2097148 264 -2[root@localhost data]# swapoff /dev/dm-1[root@localhost data]# free -m total used free shared buff/cache availableMem: 2124 254 944 9 925 1709Swap: 0 0 0

從上面free -m結(jié)果中，可以看到swap已經(jīng)變成0了。

釋放cache占用的內(nèi)存

[root@localhost data]# sync[root@localhost data]# echo 1 > /proc/sys/vm/drop_caches ; [root@localhost data]# free -m total used free shared buff/cache availableMem: 2124 254 1784 9 84 1748Swap: 0 0 0

可以看到目前剩余的內(nèi)存為1784M，mysql數(shù)據(jù)庫(kù)占用內(nèi)存主要有2大塊，第一是：buffer pool占用，第二是：初始化連接占用的內(nèi)存

在這里設(shè)置mysql的buffer pool為1500M，會(huì)話的參數(shù)設(shè)置如下

read_buffer_size = 32Mread_rnd_buffer_size = 32Msort_buffer_size = 32Mtmp_table_size = 32Mmax_heap_table_size=32Mjoin_buffer_size=32M

然后5個(gè)連接，開(kāi)始做大查詢操作，沒(méi)過(guò)多久，mysql進(jìn)程就因?yàn)镺OM被kill了

Aug 31 05:37:40 localhost kernel: Out of memory: Kill process 2534 (mysqld) score 658 or sacrifice childAug 31 05:37:40 localhost kernel: Killed process 2534 (mysqld), UID 1001, total-vm:1825792kB, anon-rss:654388kB, file-rss:0kB, shmem-rss:0kB

mysql守護(hù)進(jìn)程就開(kāi)始啟動(dòng)mysql服務(wù)

/u02/mysql/bin/mysqld_safe: 行 198: 2534 已殺死 nohup /u02/mysql/bin/mysqld --defaults-file=/u02/conf/my3308.cnf --basedir=/u02/mysql --datadir=/u02/data/3308 --plugin-dir=/u02/mysql/lib/plugin --user=mysql --log-error=/u02/log/3308/error.log --open-files-limit=65535 --pid-file=/u02/run/3308/mysqld.pid --socket=/u02/run/3308/mysql.sock --port=3308 < /dev/null > /dev/null 2>&12020-08-30T21:37:40.375749Z mysqld_safe Number of processes running now: 02020-08-30T21:37:40.407781Z mysqld_safe mysqld restarted2020-08-30T21:37:40.666886Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).2020-08-30T21:37:40.667059Z 0 [Note] --secure-file-priv is set to NULL. Operations related to importing and exporting data are disabled2020-08-30T21:37:40.667112Z 0 [Note] /u02/mysql/bin/mysqld (mysqld 5.7.26-log) starting as process 2954 ...2020-08-30T21:37:40.782412Z 0 [Warning] InnoDB: Using innodb_file_format is deprecated and the parameter may be removed in future releases. See http://dev.mysql.com/doc/refman/5.7/en/innodb-file-format.html2020-08-30T21:37:40.782684Z 0 [Note] InnoDB: PUNCH HOLE support available2020-08-30T21:37:40.782729Z 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins2020-08-30T21:37:40.782754Z 0 [Note] InnoDB: Uses event mutexes2020-08-30T21:37:40.782772Z 0 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier2020-08-30T21:37:40.782788Z 0 [Note] InnoDB: Compressed tables use zlib 1.2.112020-08-30T21:37:40.782841Z 0 [Note] InnoDB: Adjusting innodb_buffer_pool_instances from 8 to 1 since innodb_buffer_pool_size is less than 1024 MiB2020-08-30T21:37:40.784518Z 0 [Note] InnoDB: Number of pools: 12020-08-30T21:37:40.784865Z 0 [Note] InnoDB: Using CPU crc32 instructions2020-08-30T21:37:40.789314Z 0 [Note] InnoDB: Initializing buffer pool, total size = 512M, instances = 1, chunk size = 128M2020-08-30T21:37:40.834948Z 0 [Note] InnoDB: Completed initialization of buffer pool2020-08-30T21:37:40.843612Z 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().2020-08-30T21:37:40.859028Z 0 [Note] InnoDB: Highest supported file format is Barracuda.2020-08-30T21:37:40.863176Z 0 [Note] InnoDB: Log scan progressed past the checkpoint lsn 57073942292020-08-30T21:37:40.863221Z 0 [Note] InnoDB: Doing recovery: scanned up to log sequence number 57073942382020-08-30T21:37:40.863231Z 0 [Note] InnoDB: Database was not shutdown normally!2020-08-30T21:37:40.863239Z 0 [Note] InnoDB: Starting crash recovery.2020-08-30T21:37:40.901955Z 0 [Note] InnoDB: Last MySQL binlog file position 0 43848, file name binlog.0000252020-08-30T21:37:41.075805Z 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1"2020-08-30T21:37:41.075860Z 0 [Note] InnoDB: Creating shared tablespace for temporary tables2020-08-30T21:37:41.075952Z 0 [Note] InnoDB: Setting file '/u02/log/3308/iblog/ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...2020-08-30T21:37:41.254016Z 0 [Note] InnoDB: File '/u02/log/3308/iblog/ibtmp1' size is now 12 MB.2020-08-30T21:37:41.255390Z 0 [Note] InnoDB: 96 redo rollback segment(s) found. 96 redo rollback segment(s) are active.2020-08-30T21:37:41.255421Z 0 [Note] InnoDB: 32 non-redo rollback segment(s) are active.2020-08-30T21:37:41.256171Z 0 [Note] InnoDB: Waiting for purge to start2020-08-30T21:37:41.307237Z 0 [Note] InnoDB: 5.7.26 started; log sequence number 57073942382020-08-30T21:37:41.308291Z 0 [Note] Plugin 'FEDERATED' is disabled.2020-08-30T21:37:41.310625Z 0 [Note] InnoDB: Loading buffer pool(s) from /u02/log/3308/iblog/ib_buffer_pool2020-08-30T21:37:41.310785Z 0 [Note] InnoDB: Buffer pool(s) load completed at 200831 5:37:41 (/u02/log/3308/iblog/ib_buffer_pool was empty)2020-08-30T21:37:41.314568Z 0 [Note] Recovering after a crash using /u02/log/3308/binlog/binlog2020-08-30T21:37:41.314730Z 0 [Note] Starting crash recovery...2020-08-30T21:37:41.314842Z 0 [Note] Crash recovery finished.2020-08-30T21:37:41.346280Z 0 [Note] Found ca.pem, server-cert.pem and server-key.pem in data directory. Trying to enable SSL support using them.2020-08-30T21:37:41.346337Z 0 [Note] Skipping generation of SSL certificates as certificate files are present in data directory.2020-08-30T21:37:41.349079Z 0 [Warning] CA certificate ca.pem is self signed.2020-08-30T21:37:41.349341Z 0 [Note] Skipping generation of RSA key pair as key files are present in data directory.2020-08-30T21:37:41.350297Z 0 [Note] Server hostname (bind-address): '0.0.0.0'; port: 33082020-08-30T21:37:41.350399Z 0 [Note] - '0.0.0.0' resolves to '0.0.0.0';2020-08-30T21:37:41.350475Z 0 [Note] Server socket created on IP: '0.0.0.0'.2020-08-30T21:37:41.376794Z 0 [Note] Failed to start slave threads for channel ''2020-08-30T21:37:41.397237Z 0 [Note] Event Scheduler: Loaded 0 events2020-08-30T21:37:41.397480Z 0 [Note] /u02/mysql/bin/mysqld: ready for connections.Version: '5.7.26-log' socket: '/u02/run/3308/mysql.sock' port: 3308 Source distribution

正在連接的會(huì)話自動(dòng)中斷

ERROR 2013 (HY000): Lost connection to MySQL server during querymysql>

在配置mysql參數(shù)，一定要考慮以下3個(gè)因素
1.業(yè)務(wù)連接預(yù)期總數(shù)
2.會(huì)話初始化內(nèi)存
3.buffer pool緩沖器大小

下面先用sql查詢一下會(huì)話的內(nèi)存總大小和數(shù)據(jù)庫(kù)buffer pool大小

mysql> select (@@innodb_buffer_pool_size+@@innodb_log_buffer_size+@@key_buffer_size) / 1024 /1024 AS MEMORY_MB;+--------------+| MEMORY_MB |+--------------+| 584.00000000 |+--------------+1 row in set (0.00 sec)mysql> select (@@read_buffer_size+@@read_rnd_buffer_size+@@sort_buffer_size+@@tmp_table_size+@@join_buffer_size+@@binlog_cache_size)/1024/1024 as MB;+--------------+| MB |+--------------+| 160.03125000 |+--------------+1 row in set (0.00 sec)

mysql的buffer pool加上會(huì)話內(nèi)存總大小不超過(guò)服務(wù)器內(nèi)存總大小的80%。

假設(shè)，服務(wù)器內(nèi)存為32G，mysql數(shù)據(jù)庫(kù)內(nèi)存設(shè)置就不要超過(guò)26G，如果buffer pool設(shè)置為20G，每個(gè)會(huì)話內(nèi)存為160M，則6G能提供6 *1024 /160 =38 個(gè)安全連接，超過(guò)了，可能造成內(nèi)存不夠，用swap分區(qū)。

總結(jié)

以上是生活随笔為你收集整理的mysql服务重启后不见了_一次诡异的Mysql服务不断重启故障排查的全部?jī)?nèi)容，希望文章能夠幫你解決所遇到的問(wèn)題。

如果覺(jué)得生活随笔網(wǎng)站內(nèi)容還不錯(cuò)，歡迎將生活随笔推薦給好友。