當前位置：首頁 > 运维知识 > 数据库 >内容正文

数据库

postgresal去重_PostgreSQL数据去重

發布時間：2025/3/20 数据库 30 豆豆

生活随笔收集整理的這篇文章主要介紹了 postgresal去重_PostgreSQL数据去重小編覺得挺不錯的,現在分享給大家,幫大家做個參考.

數據庫中去重的需求比較常見，比較常見的如單列去重、多列去重、行去重等。pg中針對這些不同的去重要求，我們也可以使用不同的去重方法。https://www.cndba.cn/foucus/article/3916

1、單列去重單列去重應該是最常見的了，就是將某一列中重復的記錄去除掉，我們可以根據要求保留最新或者最舊的記錄。

—創建測試數據

bill=# create table test1(id int primary key, c1 int, c2 timestamp);

CREATE TABLE

bill=# insert into test1 select generate_series(1,1000000), random()*1000, clock_timestamp();

INSERT 0 1000000

bill=# create index idx_test1 on test1(c1,id);

CREATE INDEX

—方法1：

聚合，not in

https://www.cndba.cn/foucus/article/3916

bill=# explain delete from test1 where id not in (select max(id) from test1 group by c1);

QUERY PLAN

------------------------------------------------------------------------------------------------------------------

Delete on test1 (cost=30609.23..48515.23 rows=500000 width=6)

-> Seq Scan on test1 (cost=30609.23..48515.23 rows=500000 width=6)

Filter: (NOT (hashed SubPlan 1))

SubPlan 1

-> GroupAggregate (cost=0.42..30606.73 rows=1001 width=8)

Group Key: test1_1.c1

-> Index Only Scan using idx_test1 on test1 test1_1 (cost=0.42..25596.72 rows=1000000 width=8)

(7 rows)

—方法2:

使用窗口查詢，in

bill=# explain select id from (select row_number() over(partition by c1 order by id) as rn, id from test1) t where t.rn<>1;

QUERY PLAN

--------------------------------------------------------------------------------------------------

Subquery Scan on t (cost=0.42..55596.72 rows=995000 width=4)

Filter: (t.rn <> 1)

-> WindowAgg (cost=0.42..43096.72 rows=1000000 width=16)

-> Index Only Scan using idx_test1 on test1 (cost=0.42..25596.72 rows=1000000 width=8)

(4 rows)

—方法3:

使用游標的方式去遍歷，每一條記錄比較一次。

bill=# do language plpgsql $$

bill$# declare

bill$# v_rec record;

bill$# v_c1 int;

bill$# cur1 cursor for select c1,id from test1 order by c1,id for update;

bill$# begin

bill$# for v_rec in cur1 loop

bill$# if v_rec.c1 = v_c1 then

bill$# delete from test1 where current of cur1;

bill$# end if;

bill$# v_c1 := v_rec.c1;

bill$# end loop;

bill$# end;

bill$# $$;

上面三種方式，方法二效率最高，其次是方法三。

https://www.cndba.cn/foucus/article/3916https://www.cndba.cn/foucus/article/3916

2、多列去重

和單列類似，只是變成了去除多個列的重復記錄。

https://www.cndba.cn/foucus/article/3916https://www.cndba.cn/foucus/article/3916

—創建測試數據

bill=# create table test1(id int primary key, c1 int, c2 int, c3 timestamp);

CREATE TABLE

bill=# insert into test1 select generate_series(1,1000000), random()*1000, random()*1000, clock_timestamp();

INSERT 0 1000000

bill=# create index idx_test1 on test1(c1,c2,id);

CREATE INDEX

—方法1:

bill=# explain (analyze,verbose,timing,costs,buffers) delete from test1 where id not in (select max(id) from test1 group by c1,c2);

QUERY PLAN

----------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Delete on public.test1 (cost=37036.38..55906.38 rows=500000 width=6) (actual time=1924.854..1924.854 rows=0 loops=1)

Buffers: shared hit=1373911 read=3834

-> Seq Scan on public.test1 (cost=37036.38..55906.38 rows=500000 width=6) (actual time=1255.586..1672.129 rows=367700 loops=1)

Output: test1.ctid

Filter: (NOT (hashed SubPlan 1))

Rows Removed by Filter: 632300

Buffers: shared hit=1006211 read=3834

SubPlan 1

-> GroupAggregate (cost=0.42..36786.38 rows=100000 width=12) (actual time=0.061..1001.212 rows=632300 loops=1)

Output: max(test1_1.id), test1_1.c1, test1_1.c2

Group Key: test1_1.c1, test1_1.c2

Buffers: shared hit=999841 read=3834

-> Index Only Scan using idx_test1 on public.test1 test1_1 (cost=0.42..28286.38 rows=1000000 width=12) (actual time=0.052..708.625 rows=1000000 loops=1)

Output: test1_1.c1, test1_1.c2, test1_1.id

Heap Fetches: 1000000

Buffers: shared hit=999841 read=3834

Planning Time: 0.345 ms

Execution Time: 1931.117 ms

(18 rows)

—方法2:

bill=# explain (analyze,verbose,timing,costs,buffers) delete from test1 where id in (select id from (select row_number() over(partition by c1,c2 order by id) as rn, id from test1) t where t.rn<>1);

QUERY PLAN

-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Delete on public.test1 (cost=47204.90..79033.85 rows=629138 width=34) (actual time=625.967..625.968 rows=0 loops=1)

Buffers: shared hit=3836

-> Hash Semi Join (cost=47204.90..79033.85 rows=629138 width=34) (actual time=625.966..625.967 rows=0 loops=1)

Output: test1.ctid, t.*

Hash Cond: (test1.id = t.id)

Buffers: shared hit=3836

-> Seq Scan on public.test1 (cost=0.00..12693.00 rows=632300 width=10) (actual time=0.007..0.007 rows=1 loops=1)

Output: test1.ctid, test1.id

Buffers: shared hit=1

-> Hash (cost=35039.68..35039.68 rows=629138 width=32) (actual time=625.801..625.801 rows=0 loops=1)

Output: t.*, t.id

Buckets: 131072 Batches: 8 Memory Usage: 1024kB

Buffers: shared hit=3835

-> Subquery Scan on t (cost=0.42..35039.68 rows=629138 width=32) (actual time=625.800..625.800 rows=0 loops=1)

Output: t.*, t.id

Filter: (t.rn <> 1)

Rows Removed by Filter: 632300

Buffers: shared hit=3835

-> WindowAgg (cost=0.42..27135.92 rows=632300 width=20) (actual time=0.041..574.119 rows=632300 loops=1)

Output: row_number() OVER (?), test1_1.id, test1_1.c1, test1_1.c2

Buffers: shared hit=3835

-> Index Only Scan using idx_test1 on public.test1 test1_1 (cost=0.42..14489.92 rows=632300 width=12) (actual time=0.024..89.633 rows=632300 loops=1)

Output: test1_1.c1, test1_1.c2, test1_1.id

Heap Fetches: 0

Buffers: shared hit=3835

Planning Time: 0.505 ms

Execution Time: 626.029 ms

(27 rows)

—方法3:

bill=# do language plpgsql $$

bill$# declare

bill$# v_rec record;

bill$# v_c1 int;

bill$# v_c2 int;

bill$# cur1 cursor for select c1,c2 from test1 order by c1,c2,id for update;

bill$# begin

bill$# for v_rec in cur1 loop

bill$# if v_rec.c1 = v_c1 and v_rec.c2=v_c2 then

bill$# delete from test1 where current of cur1;

bill$# end if;

bill$# v_c1 := v_rec.c1;

bill$# v_c2 := v_rec.c2;

bill$# end loop;

bill$# end;

bill$# $$;

3、行去重

行去重一般可以使用ctid。https://www.cndba.cn/foucus/article/3916

—創建測試數據：

https://www.cndba.cn/foucus/article/3916

bill=# create table test1(c1 int, c2 int);

CREATE TABLE

bill=# insert into test1 select random()*1000, random()*1000 from generate_series(1,1000000);

INSERT 0 1000000

—方法1:

bill=# explain (analyze,verbose,timing,costs,buffers) delete from test1 where ctid not in (select max(ctid) from test1 group by c1,c2);

QUERY PLAN

-----------------------------------------------------------------------------------------------------------------------------------------------------------

Delete on public.test1 (cost=135831.29..152756.29 rows=500000 width=6) (actual time=2290.808..2290.808 rows=0 loops=1)

Buffers: shared hit=376170, temp read=2944 written=2954

-> Seq Scan on public.test1 (cost=135831.29..152756.29 rows=500000 width=6) (actual time=1643.262..2040.646 rows=367320 loops=1)

Output: test1.ctid

Filter: (NOT (hashed SubPlan 1))

Rows Removed by Filter: 632680

Buffers: shared hit=8850, temp read=2944 written=2954

SubPlan 1

-> GroupAggregate (cost=124581.29..135581.29 rows=100000 width=14) (actual time=732.049..1390.277 rows=632680 loops=1)

Output: max(test1_1.ctid), test1_1.c1, test1_1.c2

Group Key: test1_1.c1, test1_1.c2

Buffers: shared hit=4425, temp read=2944 written=2954

-> Sort (cost=124581.29..127081.29 rows=1000000 width=14) (actual time=732.035..1015.066 rows=1000000 loops=1)

Output: test1_1.c1, test1_1.c2, test1_1.ctid

Sort Key: test1_1.c1, test1_1.c2

Sort Method: external merge Disk: 23552kB

Buffers: shared hit=4425, temp read=2944 written=2954

-> Seq Scan on public.test1 test1_1 (cost=0.00..14425.00 rows=1000000 width=14) (actual time=0.010..138.017 rows=1000000 loops=1)

Output: test1_1.c1, test1_1.c2, test1_1.ctid

Buffers: shared hit=4425

Planning Time: 0.176 ms

Execution Time: 2304.495 ms

(22 rows)

—方法2:

bill=# explain (analyze,verbose,timing,costs,buffers) delete from test1 where ctid = any(array( select ctid from (select row_number() over(partition by c1,c2 order by ctid) as rn, ctid from test1) t where t.rn<>1));

QUERY PLAN

--------------------------------------------------------------------------------------------------------------------------------------------------------

Delete on public.test1 (cost=100501.36..100514.46 rows=10 width=6) (actual time=1092.431..1092.431 rows=0 loops=1)

Buffers: shared hit=4430, temp read=2013 written=2019

InitPlan 1 (returns $0)

-> Subquery Scan on t (cost=78357.55..100501.35 rows=629517 width=6) (actual time=1092.420..1092.420 rows=0 loops=1)

Output: t.ctid

Filter: (t.rn <> 1)

Rows Removed by Filter: 632680

Buffers: shared hit=4430, temp read=2013 written=2019

-> WindowAgg (cost=78357.55..92592.85 rows=632680 width=22) (actual time=459.611..1042.708 rows=632680 loops=1)

Output: row_number() OVER (?), test1_1.ctid, test1_1.c1, test1_1.c2

Buffers: shared hit=4430, temp read=2013 written=2019

-> Sort (cost=78357.55..79939.25 rows=632680 width=14) (actual time=459.598..616.859 rows=632680 loops=1)

Output: test1_1.ctid, test1_1.c1, test1_1.c2

Sort Key: test1_1.c1, test1_1.c2, test1_1.ctid

Sort Method: external merge Disk: 16104kB

Buffers: shared hit=4430, temp read=2013 written=2019

-> Seq Scan on public.test1 test1_1 (cost=0.00..10751.80 rows=632680 width=14) (actual time=0.006..83.917 rows=632680 loops=1)

Output: test1_1.ctid, test1_1.c1, test1_1.c2

Buffers: shared hit=4425

-> Tid Scan on public.test1 (cost=0.01..13.11 rows=10 width=6) (actual time=1092.429..1092.429 rows=0 loops=1)

Output: test1.ctid

TID Cond: (test1.ctid = ANY ($0))

Buffers: shared hit=4430, temp read=2013 written=2019

Planning Time: 0.204 ms

Execution Time: 1096.153 ms

(25 rows)

—方法3:https://www.cndba.cn/foucus/article/3916https://www.cndba.cn/foucus/article/3916

bill=# do language plpgsql $$

bill$# declare

bill$# v_rec record;

bill$# v_c1 int;

bill$# v_c2 int;

bill$# cur1 cursor for select c1,c2 from test1 order by c1,c2,ctid for update;

bill$# begin

bill$# for v_rec in cur1 loop

bill$# if v_rec.c1 = v_c1 and v_rec.c2=v_c2 then

bill$# delete from test1 where current of cur1;

bill$# end if;

bill$# v_c1 := v_rec.c1;

bill$# v_c2 := v_rec.c2;

bill$# end loop;

bill$# end;

bill$# $$;

Time: 2320.653 ms (00:02.321)

bill=# DO

《新程序員》：云原生和全面數字化實踐50位技術專家共同創作，文字、視頻、音頻交互閱讀

總結

以上是生活随笔為你收集整理的postgresal去重_PostgreSQL数据去重的全部內容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網站內容還不錯，歡迎將生活随笔推薦給好友。

上一篇： vmos框架_VMOS虚拟大师独立的安卓
下一篇： linux cmake编译源码,linu