原創(chuàng)：扣釘日記（微信公眾號(hào)ID：codelogs），歡迎分享，轉(zhuǎn)載請(qǐng)保留出處。

簡(jiǎn)介

如果說(shuō)要給Linux文本三劍客(grep、sed、awk)添加一員的話，我覺(jué)得應(yīng)該是jq命令，因?yàn)閖q命令是用來(lái)處理json數(shù)據(jù)的工具，而現(xiàn)如今json幾乎無(wú)所不在！

網(wǎng)上的jq命令分享文章也不少，但大多介紹得非常淺，jq的強(qiáng)大之處完全沒(méi)有介紹出來(lái)，所以就有了這篇文章，安利一下jq這個(gè)命令。

基本用法

格式化

# jq默認(rèn)的格式化輸出 $ echo -n '{"id":1, "name":"zhangsan", "score":[75, 85, 90]}'|jq . {"id": 1,"name": "zhangsan","score": [75,85,90] }# -c選項(xiàng)則是壓縮到1行輸出 $ jq -c . <<eof {"id": 1,"name": "zhangsan","score": [75,85,90] } eof {"id":1,"name":"zhangsan","score":[75,85,90]}

屬性提取

# 獲取id字段 $ echo -n '{"id":1, "name":"zhangsan", "score":[75, 85, 90]}'|jq '.id' 1 # 獲取name字段 $ echo -n '{"id":1, "name":"zhangsan", "score":[75, 85, 90]}'|jq '.name' "zhangsan"# 獲取name字段，-r 解開字符串引號(hào) $ echo -n '{"id":1, "name":"zhangsan", "score":[75, 85, 90]}'|jq -r '.name' zhangsan# 多層屬性值獲取 $ echo -n '{"id":1, "name":"zhangsan", "attr":{"height":1.78,"weight":"60kg"}}'|jq '.attr.height' 1.78# 獲取數(shù)組中的值 $ echo -n '{"id":1, "name":"zhangsan", "score":[75, 85, 90]}'|jq -r '.score[0]' 75$ echo -n '[75, 85, 90]'|jq -r '.[0]' 75# 數(shù)組截取 $ echo -n '[75, 85, 90]'|jq -r '.[1:3]' [85,90 ]# []展開數(shù)組 $ echo -n '[75, 85, 90]'|jq '.[]' 75 85 90# ..展開所有結(jié)構(gòu) $ echo -n '{"id":1, "name":"zhangsan", "score":[75, 85, 90]}'|jq -c '..' {"id":1,"name":"zhangsan","score":[75,85,90]} 1 "zhangsan" [75,85,90] 75 85 90# 從非對(duì)象類型中提取字段，會(huì)報(bào)錯(cuò) $ echo -n '{"id":1, "name":"zhangsan", "attr":{"height":1.78,"weight":"60kg"}}'|jq '.name.alias' jq: error (at <stdin>:0): Cannot index string with string "alias"# 使用?號(hào)可以避免這種報(bào)錯(cuò) $ echo -n '{"id":1, "name":"zhangsan", "attr":{"height":1.78,"weight":"60kg"}}'|jq '.name.alias?'# //符號(hào)用于，當(dāng)前面的表達(dá)式取不到值時(shí)，執(zhí)行后面的表達(dá)式 $ echo -n '{"id":1, "name":"zhangsan", "attr":{"height":1.78,"weight":"60kg"}}'|jq '.alias//.name' "zhangsan"

管道、逗號(hào)與括號(hào)

# 管道可以將值從前一個(gè)命令傳送到后一個(gè)命令 $ echo -n '{"id":1, "name":"zhangsan", "attr":{"height":1.78,"weight":"60kg"}}'|jq '.attr|.height' 1.78# jq中做一些基礎(chǔ)運(yùn)算也是可以的 $ echo -n '{"id":1, "name":"zhangsan", "attr":{"height":1.78,"weight":"60kg"}}'|jq '.attr|.height*100|tostring + "cm"' "178cm"# 逗號(hào)使得可以執(zhí)行多個(gè)jq表達(dá)式，使得一個(gè)輸入可計(jì)算出多個(gè)輸出結(jié)果 $ echo 1 | jq '., ., .' 1 1 1# 括號(hào)用于提升表達(dá)式的優(yōu)先級(jí)，如下：逗號(hào)優(yōu)先級(jí)低于算術(shù)運(yùn)算 $ $ echo '1'|jq '.+1, .*2' 2 2$ echo '1'|jq '(.+1, .)*2' 4 2# 管道優(yōu)先級(jí)低于逗號(hào) $ echo '1'|jq '., .|tostring' "1" "1"$ echo '1'|jq '., (.|tostring)' 1 "1"

理解jq執(zhí)行過(guò)程

表面上jq是用來(lái)處理json數(shù)據(jù)的，但實(shí)際上jq能處理的是任何json基礎(chǔ)元素所形成的流，如integer、string、bool、null、object、array等，jq執(zhí)行過(guò)程大致如下：

jq從流中獲取一個(gè)json元素

jq執(zhí)行表達(dá)式，表達(dá)式生成新的json元素

jq將新的json元素打印輸出

可以看看這些示例，如下：

# 這里jq實(shí)際上將1 2 3 4當(dāng)作4個(gè)integer元素，每找到一個(gè)元素就執(zhí)行+1操作 # jq實(shí)際上是流式處理的，1 2 3 4可以看成流中的4個(gè)元素 $ echo '1 2 3 4'|jq '. + 1' 2 3 4 5# 流中的元素不需要是同種類型，只要是完整的json元素即可 $ jq '"<" + tostring + ">"' <<eof 1 "zhangsan" true {"id":1} [75, 80, 85] eof"<1>" "<zhangsan>" "<true>" "<{\"id\":1}>" "<[75,80,85]>"# -R選項(xiàng)可用于將讀取到的json元素，都當(dāng)作字符串對(duì)待 $ seq 4|jq -R '.' "1" "2" "3" "4"# -s選項(xiàng)將從流中讀取到的所有json元素，變成一個(gè)json數(shù)組元素 # 這里理解為jq從流中只取到了1個(gè)json元素，這個(gè)json元素的類型是數(shù)組 $ seq 4|jq -s . [1,2,3,4 ]

基礎(chǔ)運(yùn)算

jq支持 + - * / % 運(yùn)算，對(duì)于+號(hào)，如果是字符串類型，則是做字符串拼接，如下：

# 做加減乘除運(yùn)算 $ echo 1|jq '.+1, .-1, .*2, ./2, .%2' 2 0 2 0.5 1# 賦值運(yùn)算 $ echo -n '{"id":1,"name":"zhangsan","age":"17","score":"75"}'|jq '.id=2' -c {"id":2,"name":"zhangsan","age":"17","score":"75"}

數(shù)據(jù)構(gòu)造

jq可以很方便的將其它數(shù)據(jù)，轉(zhuǎn)化為json對(duì)象或數(shù)組，如下：

# 使用[]構(gòu)造數(shù)組元素，-n告訴jq沒(méi)有輸入數(shù)據(jù)，直接執(zhí)行表達(dá)式并生成輸出數(shù)據(jù) $ jq -n '[1,2,3,4]' -c [1,2,3,4]$ cat data.txt id name age score 1 zhangsan 17 75 2 lisi 16 80 3 wangwu 18 85 4 zhaoliu 18 90# 每行分割成數(shù)組，[]構(gòu)造新的數(shù)組輸出 $ tail -n+2 data.txt|jq -R '[splits("\\s+")]' -c ["1","zhangsan","17","75"] ["2","lisi","16","80"] ["3","wangwu","18","85"] ["4","zhaoliu","18","90"]$ jq -n '{id:1, name:"zhangsan"}' -c {"id":1,"name":"zhangsan"}# 每行轉(zhuǎn)換為對(duì)象，{}構(gòu)造新的對(duì)象格式輸出 $ tail -n+2 data.txt|jq -R '[splits("\\s+")] | {id:.[0]|tonumber, name:.[1], age:.[2], score:.[3]}' -c {"id":1,"name":"zhangsan","age":"17","score":"75"} {"id":2,"name":"lisi","age":"16","score":"80"} {"id":3,"name":"wangwu","age":"18","score":"85"} {"id":4,"name":"zhaoliu","age":"18","score":"90"}# \()字符串占位變量替換 $ cat data.json {"id":1,"name":"zhangsan","age":"17","score":"75"} {"id":2,"name":"lisi","age":"16","score":"80"} {"id":3,"name":"wangwu","age":"18","score":"85"} {"id":4,"name":"zhaoliu","age":"18","score":"90"}$ cat data.json |jq '"id:\(.id),name:\(.name),age:\(.age),score:\(.score)"' -r id:1,name:zhangsan,age:17,score:75 id:2,name:lisi,age:16,score:80 id:3,name:wangwu,age:18,score:85 id:4,name:zhaoliu,age:18,score:90

基礎(chǔ)函數(shù)

# has函數(shù)，檢測(cè)對(duì)象是否包含key $ echo -n '{"id":1,"name":"zhangsan","age":"17","score":"75"}'|jq 'has("id")' true# del函數(shù)，刪除某個(gè)屬性 $ echo -n '{"id":1,"name":"zhangsan","age":"17","score":"75"}'|jq 'del(.id)' -c {"name":"zhangsan","age":"17","score":"75"}# map函數(shù)，對(duì)數(shù)組中每個(gè)元素執(zhí)行表達(dá)式計(jì)算，計(jì)算結(jié)果組織成新數(shù)組 $ seq 4|jq -s 'map(. * 2)' -c [2,4,6,8]# 上面map函數(shù)寫法，其實(shí)等價(jià)于這個(gè)寫法 $ seq 4|jq -s '[.[]|.*2]' -c [2,4,6,8]# keys函數(shù)，列出對(duì)象屬性 $ echo -n '{"id":1,"name":"zhangsan","age":"17","score":"75"}'|jq 'keys' -c ["age","id","name","score"]# to_entries函數(shù)，列出對(duì)象鍵值對(duì) $ echo -n '{"id":1,"name":"zhangsan","age":"17","score":"75"}'|jq 'to_entries' -c [{"key":"id","value":1},{"key":"name","value":"zhangsan"},{"key":"age","value":"17"},{"key":"score","value":"75"}]# length函數(shù)，計(jì)算數(shù)組或字符串長(zhǎng)度 $ jq -n '[1,2,3,4]|length' 4# add函數(shù)，計(jì)算數(shù)組中數(shù)值之和 $ seq 4|jq -s 'add' 10# tostring與tonumber，類型轉(zhuǎn)換 $ seq 4|jq 'tostring|tonumber' 1 2 3 4# type函數(shù)，獲取元素類型 $ jq 'type' <<eof 1 "zhangsan" true null {"id":1} [75, 80, 85] eof"number" "string" "boolean" "null" "object" "array"

過(guò)濾、排序、分組函數(shù)

$ cat data.json {"id":1,"name":"zhangsan","sex": 0, "age":"17","score":"75"} {"id":2,"name":"lisi","sex": 1, "age":"16","score":"80"} {"id":3,"name":"wangwu","sex": 0, "age":"18","score":"85"} {"id":4,"name":"zhaoliu","sex": 0, "age":"18","score":"90"}# select函數(shù)用于過(guò)濾，類似SQL中的where $ cat data.json |jq 'select( (.id>1) and (.age|IN("16","17","18")) and (.name != "lisi") or (has("attr")|not) and (.score|tonumber >= 90) )' -c {"id":3,"name":"wangwu","sex":0,"age":"18","score":"85"} {"id":4,"name":"zhaoliu","sex":0,"age":"18","score":"90"}# 有一些簡(jiǎn)化的過(guò)濾函數(shù)，如arrays, objects, iterables, booleans, numbers, normals, finites, strings, nulls, values, scalars # 它們根據(jù)類型過(guò)濾，如objects過(guò)濾出對(duì)象，values過(guò)濾出非null值等 $ jq -c 'objects' <<eof 1 "zhangsan" true null {"id":1} [75, 80, 85] eof{"id":1}$ jq -c 'values' <<eof 1 "zhangsan" true null {"id":1} [75, 80, 85] eof1 "zhangsan" true {"id":1} [75,80,85]# 選擇出id與name字段，類似SQL中的select id,name $ cat data.json|jq -s 'map({id,name})[]' -c {"id":1,"name":"zhangsan"} {"id":2,"name":"lisi"} {"id":3,"name":"wangwu"} {"id":4,"name":"zhaoliu"}# 提取前2行，類似SQL中的limit 2 $ cat data.json|jq -s 'limit(2; map({id,name})[])' -c {"id":1,"name":"zhangsan"} {"id":2,"name":"lisi"}# 按照age、id排序，類似SQL中的order by age,id $ cat data.json|jq -s 'sort_by((.age|tonumber), .id)[]' -c {"id":2,"name":"lisi","sex":1,"age":"16","score":"80"} {"id":1,"name":"zhangsan","sex":0,"age":"17","score":"75"} {"id":3,"name":"wangwu","sex":0,"age":"18","score":"85"} {"id":4,"name":"zhaoliu","sex":0,"age":"18","score":"90"}# 根據(jù)sex與age分組，并每組聚合計(jì)算count(*)、avg(score)、max(id) $ cat data.json |jq -s 'group_by(.sex, .age)[]' -c [{"id":1,"name":"zhangsan","sex":0,"age":"17","score":"75"}] [{"id":3,"name":"wangwu","sex":0,"age":"18","score":"85"},{"id":4,"name":"zhaoliu","sex":0,"age":"18","score":"90"}] [{"id":2,"name":"lisi","sex":1,"age":"16","score":"80"}]$ cat data.json |jq -s 'group_by(.sex, .age)[]|{sex:.[0].sex, age:.[0].age, count:length, avg_score:map(.score|tonumber)|(add/length), scores:map(.score)|join(","), max_id:map(.id)|max }' -c {"sex":0,"age":"17","count":1,"avg_score":75,"scores":"75","max_id":1} {"sex":0,"age":"18","count":2,"avg_score":87.5,"scores":"85,90","max_id":4} {"sex":1,"age":"16","count":1,"avg_score":80,"scores":"80","max_id":2}

字符串操作函數(shù)

# contains函數(shù)，判斷是否包含，實(shí)際也可用于判斷數(shù)組是否包含某個(gè)元素 $ echo hello | jq -R 'contains("he")' true# 判斷是否以he開頭 $ echo hello | jq -R 'startswith("he")' true# 判斷是否以llo結(jié)尾 $ echo hello | jq -R 'endswith("llo")' true# 去掉起始空格 $ echo ' hello '|jq -R 'ltrimstr(" ")|rtrimstr(" ")' "hello"# 大小寫轉(zhuǎn)換 $ echo hello|jq -R 'ascii_upcase' "HELLO"$ echo HELLO|jq -R 'ascii_downcase' "hello"# 字符串?dāng)?shù)組，通過(guò)逗號(hào)拼接成一個(gè)字符串 $ seq 4|jq -s 'map(tostring)|join(",")' "1,2,3,4"# json字符串轉(zhuǎn)換為json對(duì)象 $ echo -n '{"id":1,"name":"zhangsan","age":"17","attr":"{\"weight\":56,\"height\":178}"}'|jq '.attr = (.attr|fromjson)' -c {"id":1,"name":"zhangsan","age":"17","attr":{"weight":56,"height":178}}# json對(duì)象轉(zhuǎn)換為json字符串 $ echo -n '{"id":1,"name":"zhangsan","age":"17","attr":{"weight":56,"height":178}}'|jq '.attr = (.attr|tojson)' {"id": 1,"name": "zhangsan","age": "17","attr": "{\"weight\":56,\"height\":178}" }$ cat data.txt id:1,name:zhangsan,age:17,score:75 id:2,name:lisi,age:16,score:80 id:3,name:wangwu,age:18,score:85 id:4,name:zhaoliu,age:18,score:90# 正則表達(dá)式過(guò)濾，jq使用的是PCRE $ cat data.txt|jq -R 'select(test("id:\\d+,name:\\w+,age:\\d+,score:8\\d+"))' -r id:2,name:lisi,age:16,score:80 id:3,name:wangwu,age:18,score:85# 正則拆分字符串 $ cat data.txt|jq -R '[splits(",")]' -cr ["id:1","name:zhangsan","age:17","score:75"] ["id:2","name:lisi","age:16","score:80"] ["id:3","name:wangwu","age:18","score:85"] ["id:4","name:zhaoliu","age:18","score:90"]# 正則替換字符串 $ cat data.txt |jq -R 'gsub("name"; "nick")' -r id:1,nick:zhangsan,age:17,score:75 id:2,nick:lisi,age:16,score:80 id:3,nick:wangwu,age:18,score:85 id:4,nick:zhaoliu,age:18,score:90# 正則表達(dá)式捕獲數(shù)據(jù) $ cat data.txt|jq -R 'match("id:(?<id>\\d+),name:(?<name>\\w+),age:\\d+,score:8\\d+")' -cr {"offset":0,"length":30,"string":"id:2,name:lisi,age:16,score:80","captures":[{"offset":3,"length":1,"string":"2","name":"id"},{"offset":10,"length":4,"string":"lisi","name":"name"}]} {"offset":0,"length":32,"string":"id:3,name:wangwu,age:18,score:85","captures":[{"offset":3,"length":1,"string":"3","name":"id"},{"offset":10,"length":6,"string":"wangwu","name":"name"}]}# capture命名捕獲，生成key是捕獲組名稱，value是捕獲值的對(duì)象 $ cat data.txt|jq -R 'capture("id:(?<id>\\d+),name:(?<name>\\w+),age:\\d+,score:8\\d+")' -rc {"id":"2","name":"lisi"} {"id":"3","name":"wangwu"}# 正則掃描輸入字符串 $ cat data.txt|jq -R '[scan("\\w+:\\w+")]' -rc ["id:1","name:zhangsan","age:17","score:75"] ["id:2","name:lisi","age:16","score:80"] ["id:3","name:wangwu","age:18","score:85"] ["id:4","name:zhaoliu","age:18","score:90"]

日期函數(shù)

# 當(dāng)前時(shí)間綴 $ jq -n 'now' 1653820640.939947# 將時(shí)間綴轉(zhuǎn)換為0時(shí)區(qū)的分解時(shí)間(broken down time)，形式為 年 月 日 時(shí) 分 秒 dayOfWeek dayOfYear $ jq -n 'now|gmtime' -c [2022,4,29,10,45,5.466768980026245,0,148]# 將時(shí)間綴轉(zhuǎn)換為本地時(shí)區(qū)的分解時(shí)間(broken down time) $ jq -n 'now|localtime' -c [2022,4,29,18,46,5.386353015899658,0,148]# 分解時(shí)間轉(zhuǎn)換為時(shí)間串 $ jq -n 'now|localtime|strftime("%Y-%m-%dT%H:%M:%S")' -c "2022-05-29T18:50:33"# 與上面等效 $ jq -n 'now|strflocaltime("%Y-%m-%dT%H:%M:%SZ")' "2022-05-29T19:00:40Z"# 時(shí)間串解析為分解時(shí)間 $ date +%FT%T|jq -R 'strptime("%Y-%m-%dT%H:%M:%S")' -c [2022,4,29,18,51,27,0,148]# 分解時(shí)間轉(zhuǎn)換為時(shí)間綴 $ date +%FT%T|jq -R 'strptime("%Y-%m-%dT%H:%M:%S")|mktime' 1653850310

高級(jí)用法

實(shí)際上jq是一門腳本語(yǔ)言，它也支持變量、分支結(jié)構(gòu)、循環(huán)結(jié)構(gòu)與自定義函數(shù)，如下：

$ cat data.json {"id":1,"name":"zhangsan","sex": 0, "age":"17","score":"75"} {"id":2,"name":"lisi","sex": 1, "age":"16","score":"80"} {"id":3,"name":"wangwu","sex": 0, "age":"18","score":"85"} {"id":4,"name":"zhaoliu","sex": 0, "age":"18","score":"90"}# 單變量定義 $ cat data.json| jq '.id as $id|$id' 1 2 3 4# 對(duì)象展開式變量定義 $ cat data.json |jq '. as {id:$id,name:$name}|"id:\($id),name:\($name)"' "id:1,name:zhangsan" "id:2,name:lisi" "id:3,name:wangwu" "id:4,name:zhaoliu"$ cat data.json ["1","zhangsan","17","75"] ["2","lisi","16","80"] ["3","wangwu","18","85"] ["4","zhaoliu","18","90"]# 數(shù)組展開式變量定義 $ cat data.json|jq '. as [$id,$name]|"id:\($id),name:\($name)"' "id:1,name:zhangsan" "id:2,name:lisi" "id:3,name:wangwu" "id:4,name:zhaoliu"# 分支結(jié)構(gòu) $ cat data.json|jq '. as [$id,$name]|if ($id>"1") then "id:\($id),name:\($name)" else empty end' "id:2,name:lisi" "id:3,name:wangwu" "id:4,name:zhaoliu"# 循環(huán)結(jié)構(gòu)，第一個(gè)表達(dá)式條件滿足時(shí)，執(zhí)行只每二個(gè)表達(dá)式 # 循環(huán)結(jié)構(gòu)除了while，還有until、recurse等 $ echo 1|jq 'while(.<100; .*2)' 1 2 4 8 16 32 64# 自定義計(jì)算3次方的函數(shù) $ echo 2|jq 'def cube: .*.*. ; cube' 8

由于這些高級(jí)特性并不常用，這里僅給出了一些簡(jiǎn)單示例，詳細(xì)使用可以man jq查看。

輔助shell編程

熟悉shell腳本編程的同學(xué)都知道，shell本身是沒(méi)有提供Map、List這種數(shù)據(jù)結(jié)構(gòu)的，這導(dǎo)致使用shell實(shí)現(xiàn)某些功能時(shí)，變得很棘手。

但jq本身是處理json的，而json中的對(duì)象就可等同于Map，json中的數(shù)組就可等同于List，如下：

list='[]'; #List添加元素 list=$(echo "$list"|jq '. + [ $val ]' --arg val java); list=$(echo "$list"|jq '. + [ $val ]' --arg val shell); #獲取List大小 echo "$list"|jq '.|length' #獲取List第1個(gè)元素 echo "$list"|jq '.[0]' -r # List是否包含java字符串 echo "$list"|jq 'any(.=="java")' #刪除List第1個(gè)元素 list=$(echo "$list"|jq 'del(.[0])'); # List合并 list=$(echo "$list"|jq '. + $val' --argjson val '["shell","python"]'); # List截取 echo "$list"|jq '.[1:3]' # List遍歷 for o in $(echo "$list" | jq -r '.[]');do echo "$o"; donemap='{}'; #Map添加元素 map=$(echo "$map"|jq '.id=$val' --argjson val 1) map=$(echo "$map"|jq '.courses=$val' --argjson val "$list") #獲取Map大小 echo "$map"|jq '.|length' #獲取Map指定key的值 echo "$map"|jq '.id' -r #判斷Map指定key是否存在 echo "$map" | jq 'has("id")' #刪除Map指定key map=$(echo "$map"|jq 'del(.id)') # Map合并 map=$(echo "$map"|jq '. + $val' --argjson val '{"code":"ID001","name":"hello"}') # Map的KeySet遍歷 for key in $(echo "$map" | jq -r 'keys[]'); do value=$(jq '.[$a]' --arg a "$key" -r <<<"$map"); printf "%s:%s\n" "$key" "$value"; done # Map的entrySet遍歷 while read -r line; do key=$(jq '.key' -r <<<"$line"); value=$(jq '.value' -r <<<"$line"); printf "%s:%s\n" "$key" "$value"; done <<<$(echo "$map" | jq 'to_entries[]' -c)

總結(jié)

可以發(fā)現(xiàn)，jq已經(jīng)實(shí)現(xiàn)了json數(shù)據(jù)處理與分析的方方面面，我個(gè)人最近在工作中，也多次使用jq來(lái)分析調(diào)用日志等，用起來(lái)確實(shí)非常方便。

如果你現(xiàn)在還沒(méi)完全學(xué)會(huì)jq的用法，沒(méi)關(guān)系，建議先收藏起來(lái)，后面一定會(huì)用得到的！

往期內(nèi)容

密碼學(xué)入門
q命令-用SQL分析文本文件
神秘的backlog參數(shù)與TCP連接隊(duì)列
mysql的timestamp會(huì)存在時(shí)區(qū)問(wèn)題？
真正理解可重復(fù)讀事務(wù)隔離級(jí)別
字符編碼解惑

總結(jié)

以上是生活随笔為你收集整理的jq命令用法总结的全部?jī)?nèi)容，希望文章能夠幫你解決所遇到的問(wèn)題。

如果覺(jué)得生活随笔網(wǎng)站內(nèi)容還不錯(cuò)，歡迎將生活随笔推薦給好友。