group by

sql 查詢時，我們常將聚合函數和group by 結合起來對某一個或多個字段進行分組查詢，例如：

select addcode,count(distinct sbtid)uv from tb_hive_window group by addcode;+----------+-----+| addcode | uv |+----------+-----+| 0002 | 2 || 000201 | 1 || 000202 | 1 || 000205 | 1 || 000206 | 1 || 000208 | 2 |+----------+-----+

group by fields ... grouping sets ()

有時候因業務需要，我們需要group by 多個字段，例如：

select addcode,count(distinct sbtid)uv from tb_hive_window group by addcode;select rscode,count(distinct sbtid)uv from tb_hive_window group by rscode;select addcode,rscode,count(distinct sbtid)uv from tb_hive_window group by addcode,rscode;//這種情況我們需要寫3條sql 語句

通過grouping sets (), 我們可以通過一條sql 完成，相當于是將上面三條語句執行的結果通過union all 組合起來。

select addcode,rscode,count(*)pv,count(distinct sbtid)uv,grouping__id from tb_hive_window group by addcode,rscode grouping sets ((addcode,rscode),(addcode),(rscode));+----------+---------+-----+-----+---------------+| addcode | rscode | pv | uv | grouping__id |+----------+---------+-----+-----+---------------+| NULL | 34 | 1 | 1 | 2 || NULL | 35 | 5 | 3 | 2 || NULL | 40 | 1 | 1 | 2 || NULL | 65 | 1 | 1 | 2 || NULL | 351 | 1 | 1 | 2 || NULL | 352 | 1 | 1 | 2 || NULL | 395 | 1 | 1 | 2 || 0002 | NULL | 2 | 2 | 1 || 0002 | 34 | 1 | 1 | 3 || 0002 | 352 | 1 | 1 | 3 || 000201 | NULL | 1 | 1 | 1 || 000201 | 35 | 1 | 1 | 3 || 000202 | NULL | 1 | 1 | 1 || 000202 | 35 | 1 | 1 | 3 || 000205 | NULL | 3 | 1 | 1 || 000205 | 35 | 2 | 1 | 3 || 000205 | 395 | 1 | 1 | 3 || 000206 | NULL | 2 | 1 | 1 || 000206 | 40 | 1 | 1 | 3 || 000206 | 65 | 1 | 1 | 3 || 000208 | NULL | 2 | 2 | 1 || 000208 | 35 | 1 | 1 | 3 || 000208 | 351 | 1 | 1 | 3 |+----------+---------+-----+-----+---------------+

注意，

• 當我們沒有統計某一列值時，此時此列的值用null 表示，這可能與該列本身就是null 沖突，沒關系下面我們通過grouping__id 就可以區分。

• 此處需要說明的是grouping __id(兩個下劃線) 是十進制數，將其轉換成二進制表示后可以明確的知道此grouping __id 所對應的是哪些group by 字段。

grouping by 后面的字段排序處理成二進制數，靠近group by 的是低位，遠離group by 的是高位。查詢出的每一行結果中，如果有統計此列即此列值不為null,二進制位用1表示，否則用0表示。

+----------+---------+-----+-----+---------------+| addcode | rscode | pv | uv | grouping__id |+----------+---------+-----+-----+---------------+| NULL | 34 | 1 | 1 | 2 |--->> 轉換成二進制數：01 -->2 (注意遠離group by 的字段是高位)| 0002 | NULL | 2 | 2 | 1 |--->> 轉換成二進制數：10 -->1| 0002 | 34 | 1 | 1 | 3 | --->> 轉換成二進制數：11 -->1+2=3| 000201 | NULL | 1 | 1 | 1 || 000201 | 35 | 1 | 1 | 3 |

注意：此處開源版hive 的grouping__id 計算方式與華為云的MRS 集群hive的grouping__id計算方式不同。mrs 中的grouping__id 計算時是以靠近group by 的字段為高位，遠離的為低位，并且字段值為null 時 二進制位為1，否則為0

| 20200114 | 2020011406 | fengmizhibonew | NULL | NULL | NULL | NULL | NULL | NULL | 836 | 613 | 63

group by fields ... with cube

通過group by fields ... with cube 可是讓hive 實現所有組合維度的查詢，例如

select a,b,c,d,count(e) pv,count(distinct e) uv, grouping__id from tb_test group by a,b,c,d with cube;

根據排列組合計算，最終的組合有 C(4,1) + C(4,2) + C(4,3) + C(4,4) + 1 = 4 + 6 + 4 + 1 + 1 = 16 種組合，如果是通過單條group by 來實現，需要寫16 條sql 才能完成，因此這種方案極大地降低了程序復雜度。

select addcode,rscode,count(*)pv,count(distinct sbtid)uv,grouping__id from tb_hive_window group by addcode,rscode with cube;+----------+---------+-----+-----+---------------+| addcode | rscode | pv | uv | grouping__id |+----------+---------+-----+-----+---------------+| NULL | NULL | 11 | 6 | 0 || NULL | 34 | 1 | 1 | 2 || NULL | 35 | 5 | 3 | 2 || NULL | 40 | 1 | 1 | 2 || NULL | 65 | 1 | 1 | 2 || NULL | 351 | 1 | 1 | 2 || NULL | 352 | 1 | 1 | 2 || NULL | 395 | 1 | 1 | 2 || 0002 | NULL | 2 | 2 | 1 || 0002 | 34 | 1 | 1 | 3 || 0002 | 352 | 1 | 1 | 3 || 000201 | NULL | 1 | 1 | 1 || 000201 | 35 | 1 | 1 | 3 || 000202 | NULL | 1 | 1 | 1 || 000202 | 35 | 1 | 1 | 3 || 000205 | NULL | 3 | 1 | 1 || 000205 | 35 | 2 | 1 | 3 || 000205 | 395 | 1 | 1 | 3 || 000206 | NULL | 2 | 1 | 1 || 000206 | 40 | 1 | 1 | 3 || 000206 | 65 | 1 | 1 | 3 || 000208 | NULL | 2 | 2 | 1 || 000208 | 35 | 1 | 1 | 3 || 000208 | 351 | 1 | 1 | 3 |+----------+---------+-----+-----+---------------+

group by fields ... with rollup

rollup 是cube 的子集，通過group by fields ... with rollup 可以實現以左側維度為準，計算某一層次維度的聚合

select a,b,c,d,count(e) pv,count(distinct e) uv, grouping__id from tb_test group by a,b,c,d with rollup;//等效于select a,b,c,d,count(e) pv,count(distinct e) uv, grouping__id from tb_test group by a,b,c,d grouping sets((a,b,c,d),(a,b,c),(a,b),(a),());select addcode,rscode,count(*)pv,count(distinct sbtid)uv,grouping__id from tb_hive_window group by addcode,rscode with rollup;+----------+---------+-----+-----+---------------+| addcode | rscode | pv | uv | grouping__id |+----------+---------+-----+-----+---------------+| NULL | NULL | 11 | 6 | 0 || 0002 | NULL | 2 | 2 | 1 || 0002 | 34 | 1 | 1 | 3 || 0002 | 352 | 1 | 1 | 3 || 000201 | NULL | 1 | 1 | 1 || 000201 | 35 | 1 | 1 | 3 || 000202 | NULL | 1 | 1 | 1 || 000202 | 35 | 1 | 1 | 3 || 000205 | NULL | 3 | 1 | 1 || 000205 | 35 | 2 | 1 | 3 || 000205 | 395 | 1 | 1 | 3 || 000206 | NULL | 2 | 1 | 1 || 000206 | 40 | 1 | 1 | 3 || 000206 | 65 | 1 | 1 | 3 || 000208 | NULL | 2 | 2 | 1 || 000208 | 35 | 1 | 1 | 3 || 000208 | 351 | 1 | 1 | 3 |+----------+---------+-----+-----+---------------+//通過grouping sets 實現得到同樣的結果select addcode,rscode,count(*)pv,count(distinct sbtid)uv,grouping__id from tb_hive_window group by addcode,rscode grouping sets((addcode,rscode),(addcode),());+----------+---------+-----+-----+---------------+| addcode | rscode | pv | uv | grouping__id |+----------+---------+-----+-----+---------------+| NULL | NULL | 11 | 6 | 0 || 0002 | NULL | 2 | 2 | 1 || 0002 | 34 | 1 | 1 | 3 || 0002 | 352 | 1 | 1 | 3 || 000201 | NULL | 1 | 1 | 1 || 000201 | 35 | 1 | 1 | 3 || 000202 | NULL | 1 | 1 | 1 || 000202 | 35 | 1 | 1 | 3 || 000205 | NULL | 3 | 1 | 1 || 000205 | 35 | 2 | 1 | 3 || 000205 | 395 | 1 | 1 | 3 || 000206 | NULL | 2 | 1 | 1 || 000206 | 40 | 1 | 1 | 3 || 000206 | 65 | 1 | 1 | 3 || 000208 | NULL | 2 | 2 | 1 || 000208 | 35 | 1 | 1 | 3 || 000208 | 351 | 1 | 1 | 3 |+----------+---------+-----+-----+---------------+

總結

以上是生活随笔為你收集整理的sqlite查询乘以某列如果是null就换成_大数据之Hive group by with cube/rollup分组查询...的全部內容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網站內容還不錯，歡迎將生活随笔推薦給好友。