logstash mysql增量_Logstash jdbc 按时间增量更新的一些总结
不同數(shù)據(jù)庫的支持
mysql
數(shù)據(jù)類型
顯示樣例
是否支持timestamp
statement
tracking_column
tracking_column_type
SQL示例
date
2020-10-20
N
select *, datediff(date, '1970-01-01') as days from tbl_time where datediff(date, '1970-01-01') > :sql_last_value
days
numeric
select *, datediff(date, '1970-01-01') as days from tbl_time where datediff(date, '1970-01-01') > 1603244266
datetime timestamp
2020-10-20 06:12:01
Y
select * from tbl_time where time > :sql_last_value
time
timestamp
select * from tbl_time where time > '2020-10-20 06:12:01'
時(shí)間戳
1603244266
Y
select *, FROM_UNIXTIME(shjnch, '%Y-%m-%d %h:%i:%s') as timestamp from tbl_time where FROM_UNIXTIME(shjnch, '%Y-%m-%d %h:%i:%s') > :sql_last_value
timestamp
timestamp
select *, FROM_UNIXTIME(shjnch, '%Y-%m-%d %h:%i:%s') as timestamp from tbl_time where FROM_UNIXTIME(shjnch, '%Y-%m-%d %h:%i:%s') > '2020-10-21 14:00:00'
sqlserver
數(shù)據(jù)類型
顯示樣例
是否支持timestamp
date
2020-10-21
N
time
14:00:00.0000000
N
datetime
2020-10-21 13:59:40.000
Y
datetime2
2020-10-21 14:00:00
Y
smalldatetime
2020-10-21 14:00:00
Y
datetimeoffset
2020-10-21 14:00:00.0000000 +08:00
Y
db2
數(shù)據(jù)類型
顯示樣例
是否支持timestamp
date
2020-10-21
N
time
14:00:00.0000000
N
timestamp
2020-10-21 13:59:40.000
Y
Oracle
數(shù)據(jù)類型
顯示樣例
是否支持timestamp
statement
date
2010-2-12
N
timestamp
12-FEB-10 01.24.52.234123211 PM
Y
select *,to_char(sysdate,'YYYY-MM-DD HH24:MI:SS') as time d1 from dual where to_char(sysdate,'YYYY-MM-DD HH24:MI:SS') > :sql_last_value
Postgre
數(shù)據(jù)類型
顯示樣例
是否支持timestamp
date
1997-01-01
N
timestamp
2020-06-17 10:01:08.03282
Y
time
12:00:00
N
sybase
數(shù)據(jù)類型
顯示樣例
是否支持timestamp
date
Jul 24 2014
N
timestamp
0x00000000000a8b75
N
datetime
待驗(yàn)證
smalldatetime
待驗(yàn)證
總結(jié)
通過測試，基本上可以斷定，只要是時(shí)間格式為2020-10-21 14:00:00[.000]這種格式，都可以通過timestamp來實(shí)現(xiàn)時(shí)間增量更新。
對于不能通過這種方式的，看看有沒有對應(yīng)的函數(shù)進(jìn)行轉(zhuǎn)換成這種格式，如果沒有，則只能采用datediff轉(zhuǎn)換成天數(shù)(或秒數(shù))之后通過numeric實(shí)現(xiàn)增量同步了。
如需要根據(jù)上述的date字段做增量同步，則可配置如下：
statement => "select *, datediff(s, '1970-01-01', date) from tbl_time where datediff(s, '1970-01-01', date) > :sql_last_value"
tracking_column => "datediff(s, '1970-01-01', date)"
tracking_column_type => "numeric"
配置示例
下面給出兩個(gè)具體的配置示例：
MySQL數(shù)據(jù)庫表結(jié)構(gòu)如下：
mysql> desc tbl_time;
+-----------+--------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+-----------+--------------+------+-----+---------+-------+
| id | int(11) | NO | PRI | NULL | |
| bigid | bigint(20) | YES | | NULL | |
| name | varchar(255) | YES | | NULL | |
| date | date | YES | | NULL | |
| time | datetime | YES | | NULL | |
| timestamp | timestamp | YES | | NULL | |
| shjnch | bigint(255) | YES | | NULL | |
+-----------+--------------+------+-----+---------+-------+
7 rows in set (0.00 sec)
表中有一條數(shù)據(jù)：
mysql> select * from tbl_time;
+----+-------+-------+------------+---------------------+---------------------+------------+
| id | bigid | name | date | time | timestamp | shjnch |
+----+-------+-------+------------+---------------------+---------------------+------------+
| 1 | 1 | time1 | 2020-10-21 | 2020-10-21 09:37:31 | 2020-10-21 09:37:34 | 1603244266 |
+----+-------+-------+------------+---------------------+---------------------+------------+
1 row in set (0.00 sec)
為了能夠查詢出數(shù)據(jù)，我將增量查詢的sql中的>號改成
input { jdbc { jdbc_driver_class => "com.mysql.jdbc.Driver" jdbc_connection_string => "jdbc:mysql://127.0.0.1:3306/mytest?useSSL=false" jdbc_user => "root" jdbc_password => "123456" statement => "select *, FROM_UNIXTIME(shjnch, '%Y-%m-%d %h:%i:%s') as timestamp from tbl_time where FROM_UNIXTIME(shjnch, '%Y-%m-%d %h:%i:%s') < :sql_last_value"" schedule => "*/1 * * * *" connection_retry_attempts => 5 connection_retry_attempts_wait_time => 1 tracking_column => "timestamp" tracking_column_type => "timestamp" columns_charset => { "message" => "utf-8" } use_column_value => true lowercase_column_names => false record_last_run => true add_field => { "@topic" => "fc491237449424896" "@tags" => [] "@ip" => "127.0.0.1" } }
}
通過kafka可消費(fèi)到數(shù)據(jù)如下：
{"@topic":"fc491237449424896","timestamp":"2020-10-21 09:37:46","@ip":"127.0.0.1","date":"2020-10-21T00:00:00.000+08:00","bigid":1,"time":"2020-10-21T09:37:31.000+08:00","@timestamp":"2020-10-21T15:36:02.526+08:00","id":1,"shjnch":1603244266,"name":"time1"}
文章來源: segmentfault.com，作者：禹鼎侯，版權(quán)歸原作者所有，如需轉(zhuǎn)載，請聯(lián)系作者。
原文鏈接：segmentfault.com/a/1190000037555838
總結(jié)
以上是生活随笔為你收集整理的logstash mysql增量_Logstash jdbc 按时间增量更新的一些总结的全部內(nèi)容，希望文章能夠幫你解決所遇到的問題。
- 上一篇： jdbc和mysql做游戏排行榜_MyS
- 下一篇： linux cmake编译源码,linu