本文介紹gpdb的master故障及恢復(fù)測(cè)試以及segment故障恢復(fù)測(cè)試。

環(huán)境介紹：
Gpdb版本：5.5.0 二進(jìn)制版本
操作系統(tǒng)版本： centos linux 7.0
Master segment: 192.168.1.225/24 hostname: mfsmaster
Stadnby segemnt: 192.168.1.227/24 hostname: server227
Segment 節(jié)點(diǎn)1： 192.168.1.227/24 hostname: server227
Segment 節(jié)點(diǎn)2： 192.168.1.17/24 hostname: server17
Segment 節(jié)點(diǎn)3： 192.168.1.11/24 hostname: server11
每個(gè)segment節(jié)點(diǎn)上分別運(yùn)行一個(gè)primary segment和一個(gè)mirror segment

一、查看原始狀態(tài)

select * from gp_segment_configuration;

$ gpstate -f 20180320:13:50:38:021814 gpstate:mfsmaster:gpadmin-[INFO]:-Starting gpstate with args: -f 20180320:13:50:38:021814 gpstate:mfsmaster:gpadmin-[INFO]:-local Greenplum Version: 'postgres (Greenplum Database) 5.5.0 build commit:67afa18296aa238d53a2dfcc724da60ed2f944f0' 20180320:13:50:38:021814 gpstate:mfsmaster:gpadmin-[INFO]:-master Greenplum Version: 'PostgreSQL 8.3.23 (Greenplum Database 5.5.0 build commit:67afa18296aa238d53a2dfcc724da60ed2f944f0) on x86_64-pc-linux-gnu, compiled by GCC gcc (GCC) 6.2.0, 64-bit compiled on Feb 17 2018 15:23:55' 20180320:13:50:38:021814 gpstate:mfsmaster:gpadmin-[INFO]:-Obtaining Segment details from master... 20180320:13:50:38:021814 gpstate:mfsmaster:gpadmin-[INFO]:-Standby master details 20180320:13:50:38:021814 gpstate:mfsmaster:gpadmin-[INFO]:----------------------- 20180320:13:50:38:021814 gpstate:mfsmaster:gpadmin-[INFO]:- Standby address = server227 20180320:13:50:38:021814 gpstate:mfsmaster:gpadmin-[INFO]:- Standby data directory = /home/gpadmin/master/gpseg-1 20180320:13:50:38:021814 gpstate:mfsmaster:gpadmin-[INFO]:- Standby port = 5432 20180320:13:50:38:021814 gpstate:mfsmaster:gpadmin-[INFO]:- Standby PID = 22279 20180320:13:50:38:021814 gpstate:mfsmaster:gpadmin-[INFO]:- Standby status = Standby host passive 20180320:13:50:38:021814 gpstate:mfsmaster:gpadmin-[INFO]:-------------------------------------------------------------- 20180320:13:50:38:021814 gpstate:mfsmaster:gpadmin-[INFO]:--pg_stat_replication 20180320:13:50:38:021814 gpstate:mfsmaster:gpadmin-[INFO]:-------------------------------------------------------------- 20180320:13:50:38:021814 gpstate:mfsmaster:gpadmin-[INFO]:--WAL Sender State: streaming 20180320:13:50:38:021814 gpstate:mfsmaster:gpadmin-[INFO]:--Sync state: sync 20180320:13:50:38:021814 gpstate:mfsmaster:gpadmin-[INFO]:--Sent Location: 0/CF2C470 20180320:13:50:38:021814 gpstate:mfsmaster:gpadmin-[INFO]:--Flush Location: 0/CF2C470 20180320:13:50:38:021814 gpstate:mfsmaster:gpadmin-[INFO]:--Replay Location: 0/CF2C470 20180320:13:50:38:021814 gpstate:mfsmaster:gpadmin-[INFO]:--------------------------------------------------------------

二、master主從切換
1、模擬當(dāng)前主庫(kù)宕機(jī)，這里直接采用killall gpadmin用戶(hù)下的所有進(jìn)程來(lái)模擬

2、在master standby節(jié)點(diǎn)（227服務(wù)器上）進(jìn)行執(zhí)行切換命令，提升227為master

$ gpactivatestandby -d master/gpseg-1/ 20180320:13:53:20:030558 gpactivatestandby:server227:gpadmin-[INFO]:------------------------------------------------------ 20180320:13:53:20:030558 gpactivatestandby:server227:gpadmin-[INFO]:-Standby data directory = /home/gpadmin/master/gpseg-1 20180320:13:53:20:030558 gpactivatestandby:server227:gpadmin-[INFO]:-Standby port = 5432 20180320:13:53:20:030558 gpactivatestandby:server227:gpadmin-[INFO]:-Standby running = yes 20180320:13:53:20:030558 gpactivatestandby:server227:gpadmin-[INFO]:-Force standby activation = no 20180320:13:53:20:030558 gpactivatestandby:server227:gpadmin-[INFO]:------------------------------------------------------ Do you want to continue with standby master activation? Yy|Nn (default=N): > y 20180320:13:53:26:030558 gpactivatestandby:server227:gpadmin-[INFO]:-found standby postmaster process 20180320:13:53:26:030558 gpactivatestandby:server227:gpadmin-[INFO]:-Updating transaction files filespace flat files... 20180320:13:53:26:030558 gpactivatestandby:server227:gpadmin-[INFO]:-Updating temporary files filespace flat files... 20180320:13:53:26:030558 gpactivatestandby:server227:gpadmin-[INFO]:-Promoting standby... 20180320:13:53:26:030558 gpactivatestandby:server227:gpadmin-[DEBUG]:-Waiting for connection... 20180320:13:53:27:030558 gpactivatestandby:server227:gpadmin-[INFO]:-Standby master is promoted 20180320:13:53:27:030558 gpactivatestandby:server227:gpadmin-[INFO]:-Reading current configuration... 20180320:13:53:27:030558 gpactivatestandby:server227:gpadmin-[DEBUG]:-Connecting to dbname='postgres' 20180320:13:53:27:030558 gpactivatestandby:server227:gpadmin-[INFO]:-Writing the gp_dbid file - /home/gpadmin/master/gpseg-1/gp_dbid... 20180320:13:53:27:030558 gpactivatestandby:server227:gpadmin-[INFO]:-But found an already existing file. 20180320:13:53:27:030558 gpactivatestandby:server227:gpadmin-[INFO]:-Hence removed that existing file. 20180320:13:53:27:030558 gpactivatestandby:server227:gpadmin-[INFO]:-Creating a new file... 20180320:13:53:27:030558 gpactivatestandby:server227:gpadmin-[INFO]:-Wrote dbid: 1 to the file. 20180320:13:53:27:030558 gpactivatestandby:server227:gpadmin-[INFO]:-Now marking it as read only... 20180320:13:53:27:030558 gpactivatestandby:server227:gpadmin-[INFO]:-Verifying the file... 20180320:13:53:27:030558 gpactivatestandby:server227:gpadmin-[INFO]:------------------------------------------------------ 20180320:13:53:27:030558 gpactivatestandby:server227:gpadmin-[INFO]:-The activation of the standby master has completed successfully. 20180320:13:53:27:030558 gpactivatestandby:server227:gpadmin-[INFO]:-server227 is now the new primary master. 20180320:13:53:27:030558 gpactivatestandby:server227:gpadmin-[INFO]:-You will need to update your user access mechanism to reflect 20180320:13:53:27:030558 gpactivatestandby:server227:gpadmin-[INFO]:-the change of master hostname. 20180320:13:53:27:030558 gpactivatestandby:server227:gpadmin-[INFO]:-Do not re-start the failed master while the fail-over master is 20180320:13:53:27:030558 gpactivatestandby:server227:gpadmin-[INFO]:-operational, this could result in database corruption! 20180320:13:53:27:030558 gpactivatestandby:server227:gpadmin-[INFO]:-MASTER_DATA_DIRECTORY is now /home/gpadmin/master/gpseg-1 if 20180320:13:53:27:030558 gpactivatestandby:server227:gpadmin-[INFO]:-this has changed as a result of the standby master activation, remember 20180320:13:53:27:030558 gpactivatestandby:server227:gpadmin-[INFO]:-to change this in any startup scripts etc, that may be configured 20180320:13:53:27:030558 gpactivatestandby:server227:gpadmin-[INFO]:-to set this value. 20180320:13:53:27:030558 gpactivatestandby:server227:gpadmin-[INFO]:-MASTER_PORT is now 5432, if this has changed, you 20180320:13:53:27:030558 gpactivatestandby:server227:gpadmin-[INFO]:-may need to make additional configuration changes to allow access 20180320:13:53:27:030558 gpactivatestandby:server227:gpadmin-[INFO]:-to the Greenplum instance. 20180320:13:53:27:030558 gpactivatestandby:server227:gpadmin-[INFO]:-Refer to the Administrator Guide for instructions on how to re-activate 20180320:13:53:27:030558 gpactivatestandby:server227:gpadmin-[INFO]:-the master to its previous state once it becomes available. 20180320:13:53:27:030558 gpactivatestandby:server227:gpadmin-[INFO]:-Query planner statistics must be updated on all databases 20180320:13:53:27:030558 gpactivatestandby:server227:gpadmin-[INFO]:-following standby master activation. 20180320:13:53:27:030558 gpactivatestandby:server227:gpadmin-[INFO]:-When convenient, run ANALYZE against all user databases. 20180320:13:53:27:030558 gpactivatestandby:server227:gpadmin-[INFO]:------------------------------------------------------

3、測(cè)試提升后的主庫(kù)是否正常

$ psql -d postgres -c 'ANALYZE' postgres=# select * from gp_segment_configuration;

4、這里可能需要同步配置一下pg_hba.conf文件，才能通過(guò)客戶(hù)端進(jìn)行遠(yuǎn)程連接

到這里我們已經(jīng)完成了master節(jié)點(diǎn)的故障切換工作。

三、添加新的master standby
1、 在225服務(wù)器上執(zhí)行g(shù)pstart -a命令啟動(dòng)gpdb數(shù)據(jù)庫(kù)的時(shí)候報(bào)錯(cuò)”error: Standby active, this node no more can act as master”。當(dāng)standby 提升為master的時(shí)候，原master服務(wù)器從故障中恢復(fù)過(guò)來(lái)，需要以standby的角色加入

2、在原master服務(wù)器225上的數(shù)據(jù)進(jìn)行備份

$ cd master/ $ ls gpseg-1 $ mv gpseg-1/ backup-gpseg-1

3、在當(dāng)前master服務(wù)器227上進(jìn)行 gpinitstandby添加225為standby

$ gpinitstandby -s mfsmaster $ gpstate -f

四、primary segment和mirror segment切換
1、首先我們來(lái)捋一下當(dāng)前的數(shù)據(jù)庫(kù)環(huán)境
Master segment: 192.168.1.227/24 hostname: server227
Stadnby segemnt: 192.168.1.225/24 hostname: mfsmaster
Segment 節(jié)點(diǎn)1： 192.168.1.227/24 hostname: server227
Segment 節(jié)點(diǎn)2： 192.168.1.17/24 hostname: server17
Segment 節(jié)點(diǎn)3： 192.168.1.11/24 hostname: server11
每個(gè)segment節(jié)點(diǎn)上分別運(yùn)行一個(gè)primary segment和一個(gè)mirror segment

2、接著我們采用同樣的方式把227服務(wù)器上gpadmin用戶(hù)的所有進(jìn)行殺掉

$ killall -u gpadmin

3、在225服務(wù)器上執(zhí)行切換master命令

$ gpactivatestandby -d master/gpseg-1/

4、完成切換后使用客戶(hù)端工具連接查看segment狀態(tài)，可以看到227服務(wù)器上的server227
的primary和mirror節(jié)點(diǎn)都已經(jīng)宕機(jī)了。

5、這里為了方面查看，我們使用greenplum-cc-web工具來(lái)查看集群狀態(tài)

$ gpcmdr --start hbjy

需要將pg_hba.conf文件還原回去，因?yàn)?27上所有的segment已經(jīng)宕掉，執(zhí)行g(shù)pstop -u命令會(huì)有報(bào)錯(cuò)

在segment status頁(yè)面中可以看到當(dāng)前segment的狀態(tài)是異常的。server11上有兩組的primary segment，這很危險(xiǎn)，如果不幸server11也宕機(jī)了，整個(gè)集群的狀態(tài)就變成不可用了。

6、將server227做為master standby重新加入集群

$ cd master/ $ mv gpseg-1/ backupgpseg-1 $ gpinitstandby -s server227

7、在master上重啟集群

$ gpstop -M immediate $ gpstart -a

8、在master上恢復(fù)集群

$ gprecoverseg

雖然所有的segment均已啟動(dòng)，但server11上有還是有兩組的primary segment

9、在master上恢復(fù)segment節(jié)點(diǎn)分布到原始狀態(tài)

$ gprecoverseg -r

參考文檔：
http://greenplum.org/docs/520/admin_guide/highavail/topics/g-restoring-master-mirroring-after-a-recovery.html

總結(jié)

以上是生活随笔為你收集整理的GreenPlum数据库故障恢复测试的全部?jī)?nèi)容，希望文章能夠幫你解決所遇到的問(wèn)題。

如果覺(jué)得生活随笔網(wǎng)站內(nèi)容還不錯(cuò)，歡迎將生活随笔推薦給好友。