當前位置：首頁 > 运维知识 > 数据库 >内容正文

数据库

binlog2mysql,MySQL 数据恢复工具之binlog2sql

發布時間：2025/4/5 数据库 29 豆豆

生活随笔收集整理的這篇文章主要介紹了 binlog2mysql,MySQL 数据恢复工具之binlog2sql 小編覺得挺不錯的,現在分享給大家,幫大家做個參考.

概述

已測試環境

Python 2.7, 3.4+

MySQL 5.6, 5.7

最后一次更新是2年前的事情了，一些開源工具上線生產的時候還是需要注意。

MySQL8.0的暫時還是不要使用了。

從MySQL binlog解析出你要的SQL。根據不同選項，你可以得到原始SQL、回滾SQL、去除主鍵的INSERT SQL等。

用途

數據快速回滾(閃回)

主從切換后新master丟數據的修復

從binlog生成標準SQL，帶來的衍生功能

一.安裝pip和binlog2sql

安裝依賴包

yum install libffi-devel

yum -y install gcc automake autoconf libtool make

yum install zlib-devel

1.1 安裝步驟

cd /usr/local/src

wget https://bootstrap.pypa.io/get-pip.py

python get-pip.py

pip -V (查看pip是否安裝成功)

yum -y install git

git clone https://github.com/danfengcao/binlog2sql.git && cd binlog2sql

pip install -r requirements.txt

1.2 安裝python3

-- 下載python

wget https://npm.taobao.org/mirrors/python/3.8.0/Python-3.8.0.tar.xz

-- 解壓

tar -xf Python-3.8.0.tar.xz

-- 編譯

cd Python-3.8.0

./configure --prefix=/usr/local/python3 --with-openssl=/usr/local/python3 --with-http_ssl_module && make && make install

-- 備份 2.7

mv /usr/bin/python /usr/bin/python_2.7

-- 軟鏈

ln -sv /usr/local/python3/bin/python3.8 /usr/bin/python

--驗證

輸入 python，查看是否已經升級到 3.8 版本。

-- 解決升級后, yum 無法正常使用的問題

vi /usr/libexec/urlgrabber-ext-down

將頂部的 /usr/bin/python 改為 /usr/bin/python2

vi /usr/bin/yum

將頂部的 /usr/bin/python 改為 /usr/bin/python2

-- 將python的bin目錄加入到環境變量

vi .bash_profile

/usr/local/python3/bin

1.3 安裝新版本openssl

最新的openssl包, 因為 python3.8 需要使用 1.1 以上的包, 否則訪問ssl 地址會報錯

-- 下載軟件

wget https://github.com/openssl/openssl/archive/OpenSSL_1_1_1d.tar.gz

-- 解壓

tar -zxvf OpenSSL_1_1_1d.tar.gz

cd OpenSSL_1_1_1d

-- 編譯安裝

mkdir /usr/local/openssl

./config --prefix=/usr/local/openssl

make

make install

-- 備份老版本openssl，創建新版本軟連接

mv /usr/bin/openssl /usr/bin/openssl.old

mv /usr/include/openssl /usr/include/openssl.old

ln -s /usr/local/openssl/bin/openssl /usr/bin/openssl

ln -s /usr/local/openssl/include/openssl /usr/include/openssl

-- 把openssl lib路徑 /usr/local/openssl/lib 追加到下面到文件中

vi /usr/local/openssl/lib

-- 創建軟連接

ln -s /usr/local/openssl/lib/libssl.so.1.1 /usr/lib64/libssl.so.1.1

-- 驗證

openssl version

安裝記錄

[root@10-31-1-119 openssl-OpenSSL_1_1_1d]# more /etc/ld.so.conf

include ld.so.conf.d/*.conf:/usr/local/openssl/lib

[root@10-31-1-119 openssl-OpenSSL_1_1_1d]# openssl version

OpenSSL 1.1.1d 10 Sep 2019

1.4 pip安裝報錯1

報錯信息

DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7.

安裝pip的時候時候因為默認python是2.7版本，而pip對python2只支持到2020-01-01，故要升級到python3

將python2 升級到 python3解決問題

1.5 pip安裝報錯2

報錯信息:

could not fetch URL https://pypi.org/simple/pip/: There was a problem confirming the ssl certificate: HTTPSConnectionPool(host='pypi.org', port=443): Max retries exceeded with url: /simple/pip/ (Caused by SSLError("Can't connect to HTTPS URL because the SSL module is not available."))

詳細報錯信息

[root@10-31-1-119 src]# python get-pip.py

WARNING: pip is configured with locations that require TLS/SSL, however the ssl module in Python is not available.

WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError("Can't connect to HTTPS URL because the SSL module is not available.")': /simple/pip/

WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError("Can't connect to HTTPS URL because the SSL module is not available.")': /simple/pip/

WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError("Can't connect to HTTPS URL because the SSL module is not available.")': /simple/pip/

WARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError("Can't connect to HTTPS URL because the SSL module is not available.")': /simple/pip/

WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError("Can't connect to HTTPS URL because the SSL module is not available.")': /simple/pip/

Could not fetch URL https://pypi.org/simple/pip/: There was a problem confirming the ssl certificate: HTTPSConnectionPool(host='pypi.org', port=443): Max retries exceeded with url: /simple/pip/ (Caused by SSLError("Can't connect to HTTPS URL because the SSL module is not available.")) - skipping

ERROR: Could not find a version that satisfies the requirement pip (from versions: none)

ERROR: No matching distribution found for pip

WARNING: pip is configured with locations that require TLS/SSL, however the ssl module in Python is not available.

解決方案

加上國內的鏡像源問題解決

[root@10-31-1-119 src]# python get-pip.py -i http://pypi.douban.com/simple --trusted-host pypi.douban.com

Looking in indexes: http://pypi.douban.com/simple

Collecting pip

Downloading http://pypi.doubanio.com/packages/5a/4a/39400ff9b36e719bdf8f31c99fe1fa7842a42fa77432e584f707a5080063/pip-20.2.2-py2.py3-none-any.whl (1.5 MB)

|████████████████████████████████| 1.5 MB 10.7 MB/s

Collecting setuptools

Downloading http://pypi.doubanio.com/packages/c3/a9/5dc32465951cf4812e9e93b4ad2d314893c2fa6d5f66ce5c057af6e76d85/setuptools-49.6.0-py3-none-any.whl (803 kB)

|████████████████████████████████| 803 kB 49.3 MB/s

Collecting wheel

Downloading http://pypi.doubanio.com/packages/a7/00/3df031b3ecd5444d572141321537080b40c1c25e1caa3d86cdd12e5e919c/wheel-0.35.1-py2.py3-none-any.whl (33 kB)

Installing collected packages: pip, setuptools, wheel

Attempting uninstall: pip

Found existing installation: pip 19.2.3

Uninstalling pip-19.2.3:

Successfully uninstalled pip-19.2.3

WARNING: The scripts pip, pip3 and pip3.8 are installed in '/usr/local/python3/bin' which is not on PATH.

Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.

Attempting uninstall: setuptools

Found existing installation: setuptools 41.2.0

Uninstalling setuptools-41.2.0:

Successfully uninstalled setuptools-41.2.0

WARNING: The scripts easy_install and easy_install-3.8 are installed in '/usr/local/python3/bin' which is not on PATH.

Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.

WARNING: The script wheel is installed in '/usr/local/python3/bin' which is not on PATH.

Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.

Successfully installed pip-20.2.2 setuptools-49.6.0 wheel-0.35.1

[root@10-31-1-119 src]#

1.6 pip安裝binlog2sql報錯

報錯信息:

ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

[root@10-31-1-119 ~]# cd /usr/local/src/

[root@10-31-1-119 src]# git clone https://github.com/danfengcao/binlog2sql.git && cd binlog2sql

正克隆到 'binlog2sql'...

remote: Enumerating objects: 323, done.

remote: Total 323 (delta 0), reused 0 (delta 0), pack-reused 323

接收對象中: 100% (323/323), 151.51 KiB | 10.00 KiB/s, done.

處理 delta 中: 100% (170/170), done.

[root@10-31-1-119 binlog2sql]#

[root@10-31-1-119 binlog2sql]# pip install -r requirements.txt

bash: pip: 未找到命令...

[root@10-31-1-119 binlog2sql]# source /root/.bash_profile

[root@10-31-1-119 binlog2sql]#

[root@10-31-1-119 binlog2sql]# pip install -r requirements.txt -i http://pypi.douban.com/simple --trusted-host pypi.douban.com

Looking in indexes: http://pypi.douban.com/simple

Collecting PyMySQL==0.7.11

Downloading http://pypi.doubanio.com/packages/c6/42/c54c280d8418039bd2f61284f99cb6d9e0eae80383fc72ceb6eac67855fe/PyMySQL-0.7.11-py2.py3-none-any.whl (78 kB)

|████████████████████████████████| 78 kB 16.5 MB/s

Collecting wheel==0.29.0

Downloading http://pypi.doubanio.com/packages/8a/e9/8468cd68b582b06ef554be0b96b59f59779627131aad48f8a5bce4b13450/wheel-0.29.0-py2.py3-none-any.whl (66 kB)

|████████████████████████████████| 66 kB 10.9 MB/s

Collecting mysql-replication==0.13

Downloading http://pypi.doubanio.com/packages/dd/23/384047702e694139e9fe75a8ba7ad007e8942fd119ebadabc32ce19f70f2/mysql-replication-0.13.tar.gz (33 kB)

ERROR: Command errored out with exit status 1:

command: /usr/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-_og_eq1j/mysql-replication/setup.py'"'"'; __file__='"'"'/tmp/pip-install-_og_eq1j/mysql-replication/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-fezrihiv

cwd: /tmp/pip-install-_og_eq1j/mysql-replication/

Complete output (11 lines):

Traceback (most recent call last):

File "", line 1, in

File "/usr/local/python3/lib/python3.8/site-packages/setuptools/__init__.py", line 21, in

from setuptools.dist import Distribution

File "/usr/local/python3/lib/python3.8/site-packages/setuptools/dist.py", line 34, in

from setuptools import windows_support

File "/usr/local/python3/lib/python3.8/site-packages/setuptools/windows_support.py", line 2, in

import ctypes

File "/usr/local/python3/lib/python3.8/ctypes/__init__.py", line 7, in

from _ctypes import Union, Structure, Array

ModuleNotFoundError: No module named '_ctypes'

----------------------------------------

ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

[root@10-31-1-119 binlog2sql]#

解決方案

-- 安裝依賴包

yum install libffi-devel

-- 重新安裝python3

-- 編譯

cd Python-3.8.0

./configure --prefix=/usr/local/python3 --with-openssl=/usr/local/python3 --with-http_ssl_module && make && make install

二.binlog2sql參數相關

2.1 binlog2sql參數

# python binlog2sql/binlog2sql.py --help

選項

mysql連接配置

-h host; -P port; -u user; -p password

解析模式

--stop-never 持續解析binlog。可選。，默認False，同步至執行命令時最新的binlog位置。

-K, --no-primary-key 對INSERT語句去除主鍵。可選。默認False

-B, --flashback 生成回滾SQL，可解析大文件，不受內存限制。可選。默認False。與stop-never或no-primary-key不能同時添加。

--back-interval -B模式下，每打印一千行回滾SQL，加一句SLEEP多少秒，如不想加SLEEP，請設為0。可選。默認1.0。

解析范圍控制

--start-file 起始解析文件，只需文件名，無需全路徑。必須。

--start-position/--start-pos 起始解析位置。可選。默認為start-file的起始位置。

--stop-file/--end-file 終止解析文件。可選。默認為start-file同一個文件。若解析模式為stop-never，此選項失效。

--stop-position/--end-pos 終止解析位置。可選。默認為stop-file的最末位置；若解析模式為stop-never，此選項失效。

--start-datetime 起始解析時間，格式'%Y-%m-%d %H:%M:%S'。可選。默認不過濾。

--stop-datetime 終止解析時間，格式'%Y-%m-%d %H:%M:%S'。可選。默認不過濾。

對象過濾

-d, --databases 只解析目標db的sql，多個庫用空格隔開，如-d db1 db2。可選。默認為空。

-t, --tables 只解析目標table的sql，多張表用空格隔開，如-t tbl1 tbl2。可選。默認為空。

--only-dml 只解析dml，忽略ddl。可選。默認TRUE。

--sql-type 只解析指定類型，支持INSERT, UPDATE, DELETE。多個類型用空格隔開，如--sql-type INSERT DELETE。可選。默認為增刪改都解析。用了此參數但沒填任何類型，則三者都不解析。

2.2 mysql server相關參數

# 在配置文件my.cnf的mysqld這個區下設置

[mysqld]

server_id = 1

log_bin = /var/log/mysql/mysql-bin.log

max_binlog_size = 1G

binlog_format = row

binlog_row_image = full

# 在運行中的mysql中查看

show variables like 'server_id';

show variables like 'log_bin%';

show variables like 'max_binlog_size';

show variables like 'binlog_format';

show variables like 'binlog_row_image';

2.3 用來閃回數據的user需要的最小權限集合

# 建議授權

# select, super/replication client, replication slave

GRANT SELECT, REPLICATION SLAVE, REPLICATION CLIENT ON *.* TO repl@'localhost' identified by '123456';

# select：需要讀取server端information_schema.COLUMNS表，獲取表結構的元信息，拼接成可視化的sql語句

# super/replication client：兩個權限都可以，需要執行'SHOW MASTER STATUS', 獲取server端的binlog列表

# replication slave：通過BINLOG_DUMP協議獲取binlog內容的權限

三.binlog2sql使用案例

3.1 數據準備

首先我往表t3中錄入了3條數據，然后進行了update和delete

現在的需求是恢復到最開始的3條insert的數據

mysql> create table t3(id int,name varchar(100));

Query OK, 0 rows affected (0.01 sec)

mysql> select current_timestamp();

+---------------------+

| current_timestamp() |

+---------------------+

| 2020-08-27 13:47:04 |

+---------------------+

1 row in set (0.01 sec)

mysql>

mysql> insert into t3 values (1,'abc');

Query OK, 1 row affected (0.01 sec)

mysql>

mysql> insert into t3 values (2,'def');

Query OK, 1 row affected (0.00 sec)

mysql> insert into t3 values (3,'ghi');

Query OK, 1 row affected (0.01 sec)

mysql>

mysql> update t3 set name = 'test' where id <2;

Query OK, 1 row affected (0.01 sec)

Rows matched: 1 Changed: 1 Warnings: 0

mysql> delete from t3 where id = 3;

Query OK, 1 row affected (0.07 sec)

mysql> select * from t3;

+------+------+

| id | name |

+------+------+

| 1 | test |

| 2 | def |

+------+------+

2 rows in set (0.00 sec)

mysql>

mysql> select current_timestamp();

+---------------------+

| current_timestamp() |

+---------------------+

| 2020-08-27 13:48:46 |

+---------------------+

1 row in set (0.00 sec)

3.2 使用binlog2sql找到恢復sql

cd /usr/local/src/binlog2sql/binlog2sql/

python binlog2sql.py -h 10.31.1.119 -u root -p abc123 -d test -t t3 --start-file='10-31-1-119-bin.000007' --start-datetime='2020-08-27 13:47:04' --stop-datetime='2020-08-27 13:48:46'

運行記錄

[root@10-31-1-119 binlog2sql]# cd /usr/local/src/binlog2sql/binlog2sql/

[root@10-31-1-119 binlog2sql]# pwd

/usr/local/src/binlog2sql/binlog2sql

[root@10-31-1-119 binlog2sql]# python binlog2sql.py -h 10.31.1.119 -u root -p abc123 -d test -t t3 --start-file='10-31-1-119-bin.000007' --start-datetime='2020-08-27 13:47:04' --stop-datetime='2020-08-27 13:48:46'

INSERT INTO `test`.`t3`(`id`, `name`) VALUES (1, 'abc'); #start 16058170 end 16058400 time 2020-08-27 13:47:31

INSERT INTO `test`.`t3`(`id`, `name`) VALUES (2, 'def'); #start 16058431 end 16058661 time 2020-08-27 13:47:46

INSERT INTO `test`.`t3`(`id`, `name`) VALUES (3, 'ghi'); #start 16058692 end 16058922 time 2020-08-27 13:47:52

UPDATE `test`.`t3` SET `id`=1, `name`='test' WHERE `id`=1 AND `name`='abc' LIMIT 1; #start 16058953 end 16059195 time 2020-08-27 13:48:04

DELETE FROM `test`.`t3` WHERE `id`=3 AND `name`='ghi' LIMIT 1; #start 16059226 end 16059456 time 2020-08-27 13:48:14

[root@10-31-1-119 binlog2sql]#

可以看到最后一個刪除的 #start 16059226 end 16059456

python binlog2sql.py \

-h10.31.1.119 -P3306 -uroot -p'abc123' \

--start-file='10-31-1-119-bin.000007' \

-dtest -tt3 \

--start-position=16059226 \

--stop-position=16059456 \

-B > rollback.sql

[root@10-31-1-119 binlog2sql]# python binlog2sql.py \

> -h10.31.1.119 -P3306 -uroot -p'abc123' \

> --start-file='10-31-1-119-bin.000007' \

> -dtest -tt3 \

> --start-position=16059226 \

> --stop-position=16059456 \

> -B > rollback.sql

[root@10-31-1-119 binlog2sql]#

[root@10-31-1-119 binlog2sql]# more rollback.sql

INSERT INTO `test`.`t3`(`id`, `name`) VALUES (3, 'ghi'); #start 16059226 end 16059456 time 2020-08-27 13:48:14

測試一次性生產多個回滾sql

python binlog2sql.py \

-h10.31.1.119 -P3306 -uroot -p'abc123' \

--start-file='10-31-1-119-bin.000007' \

-dtest -tt3 \

--start-position=16058953 \

--stop-position=16059456 \

-B > rollback.sql

[root@10-31-1-119 binlog2sql]# python binlog2sql.py \

> -h10.31.1.119 -P3306 -uroot -p'abc123' \

> --start-file='10-31-1-119-bin.000007' \

> -dtest -tt3 \

> --start-position=16058953 \

> --stop-position=16059456 \

> -B > rollback2.sql

[root@10-31-1-119 binlog2sql]#

[root@10-31-1-119 binlog2sql]# more rollback2.sql

INSERT INTO `test`.`t3`(`id`, `name`) VALUES (3, 'ghi'); #start 16059226 end 16059456 time 2020-08-27 13:48:14

UPDATE `test`.`t3` SET `id`=1, `name`='abc' WHERE `id`=1 AND `name`='test' LIMIT 1; #start 16058953 end 16059195 time 2020-08-27 13:48:04

數據還原

[root@10-31-1-119 binlog2sql]# mysql -uroot -p

Enter password:

Welcome to the MySQL monitor. Commands end with ; or \g.

Your MySQL connection id is 4912

Server version: 5.7.31-log MySQL Community Server (GPL)

Oracle is a registered trademark of Oracle Corporation and/or its

affiliates. Other names may be trademarks of their respective

owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> source rollback2.sql

Query OK, 1 row affected (0.01 sec)

Query OK, 1 row affected (0.00 sec)

Rows matched: 1 Changed: 1 Warnings: 0

mysql> select * from test.t3;

+------+------+

| id | name |

+------+------+

| 1 | abc |

| 2 | def |

| 3 | ghi |

+------+------+

3 rows in set (0.00 sec)

參考文獻:

總結

以上是生活随笔為你收集整理的binlog2mysql,MySQL 数据恢复工具之binlog2sql的全部內容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網站內容還不錯，歡迎將生活随笔推薦給好友。

上一篇： php图文消息带代码详细注释,微信发送图
下一篇： win10安装怎么一键分区工具 Win1