日韩性视频-久久久蜜桃-www中文字幕-在线中文字幕av-亚洲欧美一区二区三区四区-撸久久-香蕉视频一区-久久无码精品丰满人妻-国产高潮av-激情福利社-日韩av网址大全-国产精品久久999-日本五十路在线-性欧美在线-久久99精品波多结衣一区-男女午夜免费视频-黑人极品ⅴideos精品欧美棵-人人妻人人澡人人爽精品欧美一区-日韩一区在线看-欧美a级在线免费观看

歡迎訪問 生活随笔！

生活随笔

當前位置： 首頁 > 编程资源 > 编程问答 >内容正文

编程问答

大数据 流式计算 apache storm 学习笔记 01 ---汪文君

發(fā)布時間：2023/12/9 编程问答 35 豆豆
生活随笔 收集整理的這篇文章主要介紹了 大数据 流式计算 apache storm 学习笔记 01 ---汪文君 小編覺得挺不錯的,現(xiàn)在分享給大家,幫大家做個參考.

汪文君Apache Storm1.2.2實戰(zhàn)

9個小時
https://www.bilibili.com/video/BV1et41147xq?p=1

P1汪文君Apache Storm1.2.2實戰(zhàn)-01講-Apache Storm的介紹 20:53
P2汪文君Apache Storm1.2.2實戰(zhàn)-02講-Apache Storm關鍵組件的詳細講解 14:42
P3汪文君Apache Storm1.2.2實戰(zhàn)-03講-Apache Storm集群環(huán)境搭建詳解 33:09
P4汪文君Apache Storm1.2.2實戰(zhàn)-04講-第一個Topology程序開發(fā)本地模式 23:53
P5汪文君Apache Storm1.2.2實戰(zhàn)-05講-第一個Topology程序開發(fā)集群模式 23:26
P6汪文君Apache Storm1.2.2實戰(zhàn)-06講-Storm并行度核心概念講解 42:54
P7汪文君Apache Storm1.2.2實戰(zhàn)-07講-通過程序運行分析Topology的并行度-上 29:56
P8汪文君Apache Storm1.2.2實戰(zhàn)-08講-通過程序運行分析Topology的并行度-下 17:39
P9汪文君Apache Storm1.2.2實戰(zhàn)-09講-Storm Topology的Rebalance 10:41
P10汪文君Apache Storm1.2.2實戰(zhàn)-10講-分析Storm Topology并行度內容補充 11:26
P11汪文君Apache Storm1.2.2實戰(zhàn)-11講-shuffle grouping數(shù)據(jù)分組詳解 25:46
P12汪文君Apache Storm1.2.2實戰(zhàn)-12講-shuffle grouping數(shù)據(jù)分組詳解（補充） 05:49
P13汪文君Apache Storm1.2.2實戰(zhàn)-13講-fields grouping數(shù)據(jù)分組詳解（戴耳機聽） 23:29
P14汪文君Apache Storm1.2.2實戰(zhàn)-14講-all grouping講解前的分析過程 20:26
P15汪文君Apache Storm1.2.2實戰(zhàn)-15講-all grouping詳細講解 14:16
P16汪文君Apache Storm1.2.2實戰(zhàn)-16講-global grouping詳細講解 25:51
P17汪文君Apache Storm1.2.2實戰(zhàn)-17講-direct grouping詳細講解 26:17
P18汪文君Apache Storm1.2.2實戰(zhàn)-18講-none grouping& localOrShuffe grouping 04:52
P19汪文君Apache Storm1.2.2實戰(zhàn)-19講-如何自定義storm grouping詳解 33:33
P20汪文君Apache Storm1.2.2實戰(zhàn)-20講-綜合案例之電信號碼主叫被叫實時統(tǒng)計 24:01
P21汪文君Apache Storm1.2.2實戰(zhàn)-21講-綜合案例之Word Count 25:05
P22汪文君Apache Storm1.2.2實戰(zhàn)-22講-Storm擔保數(shù)據(jù)被處理ack,failed,timeout,exeption講解（戴耳機聽） 26:53
P23汪文君Apache Storm1.2.2實戰(zhàn)-23講-Storm Fully擔保數(shù)據(jù)被處理的案例分析 15:38
P24汪文君Apache Storm1.2.2實戰(zhàn)-24講-一些擔保數(shù)據(jù)被處理的方案講解 16:30
P25汪文君Apache Storm1.2.2實戰(zhàn)-25講-高級Tuple的使用（Tick Tuple） 16:42

指令

-----啟動nimbus storm nimbus & --- 修改 日志級別 ------方便查看日志 比如將日志從 info 降級為 warn [root@storm51 log4j2]# vim /usr/local/storm/log4j2/worker.xml ---- 查看日志記錄 [root@storm51 logs]# tail -f 200 /usr/local/storm/logs/nimbus.log --- 查看 topology 的日志 [root@storm53 workers-artifacts]# pwd /usr/local/storm/logs/workers-artifacts [root@storm53 workers-artifacts]# ll total 0 drwxr-xr-x. 3 root root 18 Apr 10 02:13 tp1-1-1618046001 drwxr-xr-x. 3 root root 18 Apr 10 02:51 tp2-2-1618048284 drwxr-xr-x. 3 root root 18 Apr 10 03:51 tp6-7-1618051908

P1汪文君Apache Storm1.2.2實戰(zhàn)-01講-Apache Storm的介紹 20:53

項目Value
1Java8 In Action
1Apache Flume
1PowerMock
1Concordion
1Mockito
1Apache Sqoop
1Java Concurrency
1Google Guava
1Scala In Action
1Apache Kafka 0.11.x
1Metrics
1JMH
1


官網(wǎng)： https://storm.apache.org/

流式計算產(chǎn)品 Streaming Compute System

項目Value
1Apache Spark Streaming
1Apache Flink
1Apache Kafka Streaming
1Apache Storm
1… …

課程大綱

P2汪文君Apache Storm1.2.2實戰(zhàn)-02講-Apache Storm關鍵組件的詳細講解 14:42

A Storm cluster follows a master-slave model

1.Master主節(jié)點-------Nimbus
The Nimbus node is the master in a Storm cluster.
The Nimbus is stateless and stores all of its data in Zookeeper.There is a single Nimbus node in a Storm cluster.

2.Supervisor Nodes 工作節(jié)點

3.Tuples --The Storm data model

4.Storm Topology
5.Spout&Stream&Bolt
5.1 Spout: A spout is the source of tuples in a Storm Topology
5.2 Bolt: A bolt is the processing powerhouse of a Storm topology , and is responsible for transforming a stream
5.3 Stream: The key abstraction in Storm is tha of a stream.

P3汪文君Apache Storm1.2.2實戰(zhàn)-03講-Apache Storm集群環(huán)境搭建詳解 33:09

視頻作者用時30分鐘講完，leo用時4個小時搭建完成（20210405），作者用1.2.2 ，Leo用 1.2.3

linux centos 7.6 安裝 Apache Storm1.2.3
https://blog.csdn.net/wei198621/article/details/115449855

P4汪文君Apache Storm1.2.2實戰(zhàn)-04講-第一個Topology程序開發(fā)本地模式 23:53


spout 生成隨機數(shù) 交個 bolt 顯示 ****outputCollector 用于發(fā)送數(shù)據(jù) step1. RandomStringSpout.java ------------- SpoutOutputCollector.emit(new Tuple(...)); 將數(shù)據(jù)通過collector 以tuple 格式 發(fā)送 ------------- OutputFieldsDeclarer.declare(new Fields("stream")); 命名發(fā)送的tuple 為stream step2. WrapStarBolt.java Tuple.getStringByField("stream"); 獲取spout 發(fā)過來的 名為 stream 的 tuple step3. WrapWellBolt.java Tuple.getStringByField("stream"); 獲取spout 發(fā)過來的 名為 stream 的 tuple step4. RandomStringTopologyLocal.java 4.1 topoBuilder= new TopologyBuilder() 將 1 2 3 步驟的 spout Bolt 賦值給此 TopologyBuilder 4.2 cluster = new LocalCluster() 定義cluster 4.3 conf = new Config() 定義cluster 的配置 4.4 cluster.submitTopology( *, conf, topoBuilder); cluster 提交 TopologyBuilder

P5汪文君Apache Storm1.2.2實戰(zhàn)-05講-第一個Topology程序開發(fā)集群模式 23:26

啟動zookeeper storm 集群

[root@centos7-7 bin]# ll total 24 -rwxr--r--. 1 root root 466 Apr 8 06:06 ctlkafkaauto.sh -rwxr--r--. 1 root root 133 Apr 8 09:23 ctlstorm.sh -rwxr--r--. 1 root root 394 Apr 8 06:06 ctlzookeeperauto.sh lrwxrwxrwx. 1 root root 36 Apr 8 06:09 jps -> /usr/local/java/jdk1.8.0_251/bin/jps -rwxr--r--. 1 root root 154 Apr 8 06:06 xcallkafka.sh -rwxr--r--. 1 root root 153 Apr 8 08:31 xcallstorm.sh -rwxr--r--. 1 root root 145 Apr 8 06:06 xcallzk.sh [root@centos7-7 bin]# ctlzookeeperauto.sh start ... 暫時還不可以批量執(zhí)行 需要手動執(zhí)行 [root@centos7-7 bin]# cat ctlstorm.sh ssh storm51 "storm ui 1>/dev/null 2>&1" ssh storm51 "storm nimbus 1>/dev/null 2>&1" ssh storm52 "storm supervisor 1>/dev/null 2>&1" ssh storm53 "storm supervisor 1>/dev/null 2>&1" [root@centos7-7 bin]# xcallzk.sh jps ============= zk1 jps ============= 9360 QuorumPeerMain 29191 Jps ============= zk2 jps ============= 29269 Jps 9384 QuorumPeerMain ============= zk3 jps ============= 9181 QuorumPeerMain 29085 Jps[root@centos7-7 bin]# xcallstorm.sh jps ============= storm51 jps ============= 12126 core 13054 nimbus 17663 Jps ============= storm52 jps ============= 30615 Supervisor 31870 Jps ============= storm53 jps ============= 30256 worker 27252 Supervisor 31726 Jps

啟動 storm ui
http://192.168.121.51:8080/index.html


啟動腳本

[root@storm51 ~]# storm jar storm-1.0-SNAPSHOT-jar-with-dependencies.jar com.tiza.leo.bigdata.storm.test01Random.RandomStringTopologyRemote 2340 [main] WARN o.a.s.u.Utils - STORM-VERSION new 1.2.3 old 1.2.3 3033 [main] INFO o.a.s.StormSubmitter - Finished submitting topology: RandomStringTopologyRemote [root@storm51 ~]#

查看topology是否啟動腳本

[root@storm51 ~]# storm list Running: .../usr/local/apache-storm-1.2.3/bin org.apache.storm.command.list 5402 [main] INFO o.a.s.u.NimbusClient - Found leader nimbus : master:6627 Topology_name Status Num_tasks Num_workers Uptime_secs ------------------------------------------------------------------- RandomStringTopologyRemote ACTIVE 12 3 98 [root@storm51 ~]#

此時查看jps運行進程

[root@centos7-7 bin]# xcallstorm.sh jps ============= storm51 jps ============= 12126 core 13054 nimbus 17663 Jps ============= storm52 jps ============= 30615 Supervisor 31870 Jps ============= storm53 jps ============= 30256 worker 27252 Supervisor 30214 worker 30215 worker 30185 LogWriter 30186 LogWriter 30221 LogWriter 31726 Jps

敦化 暫停 一個 topology

storm deactivate RandomStringTopologyRemote

P6汪文君Apache Storm1.2.2實戰(zhàn)-06講-Storm并行度核心概念講解 42:54

更改日志節(jié)點為warn ,fang變查看

[root@storm52 log4j2]# vim /usr/local/storm/log4j2/worker.xml [root@storm53 log4j2]# vim /usr/local/storm/log4j2/worker.xml

P7汪文君Apache Storm1.2.2實戰(zhàn)-07講-通過程序運行分析Topology的并行度-上 29:56

到 zhan 筆記本上面 用腳本啟動 zookeerper storm
http://192.168.121.51:8080/index.html 確認一切OK ,
編譯上一步的文件 ，給到 storm 主節(jié)點 storm 51


測試1

[root@storm51 ~]# storm jar storm-1.0-SNAPSHOT-jar-with-dependencies.jar com.tiza.leo.bigdata.storm.test02Parallel.SimpleTopology ------ 報錯，需要參數(shù) ====================================================================================[root@storm51 ~]# storm jar storm-1.0-SNAPSHOT-jar-with-dependencies.jar com.tiza.leo.bigdata.storm.test02Parallel.SimpleTopology tp1 tp1 1 2 1 2 1 {topologyName='tp1', prefix='tp1', workers=1, spoutParallelHint=2, spoutTasks=1, boltParallelHint=2, boltTasks=1}------------------------------------------------------------------------------- 結論 并行度(parallelHint) Num executors = 1worker + 1spoutTask + 1boltTask =3 ====================================================================================

測試2

[root@storm51 ~]# storm jar storm-1.0-SNAPSHOT-jar-with-dependencies.jar com.tiza.leo.bigdata.storm.test02Parallel.SimpleTopology tp2 tp2 2 2 1 2 1 {topologyName='tp2', prefix='tp2', workers=2, spoutParallelHint=2, spoutTasks=1, boltParallelHint=2, boltTasks=1}------------------------------------------------------------------------------- 結論 并行度(parallelHint) Num executors = 2 worker + 1spoutTask + 1boltTask =4

測試3

==================================================================================== [root@storm51 ~]# storm jar storm-1.0-SNAPSHOT-jar-with-dependencies.jar com.tiza.leo.bigdata.storm.test02Parallel.SimpleTopology tp3 tp3 1 2 2 2 2 {topologyName='tp3', prefix='tp3', workers=1, spoutParallelHint=2, spoutTasks=2, boltParallelHint=2, boltTasks=2} ------------------------------------------------------------------------------- 結論 并行度(parallelHint) Num executors = 1 worker + 2 spoutTask + 2 boltTask =5

測試4

==================================================================================== [root@storm51 ~]# storm jar storm-1.0-SNAPSHOT-jar-with-dependencies.jar com.tiza.leo.bigdata.storm.test02Parallel.SimpleTopology tp4 tp4 3 3 1 3 2 {topologyName='tp4', prefix='tp4', workers=3, spoutParallelHint=3, spoutTasks=1, boltParallelHint=3, boltTasks=2} ------------------------------------------------------------------------------- 推測結論 并行度(parallelHint) Num executors = 3 worker + 1 spoutTask + 2 boltTask =6 實際結論 并行度(parallelHint) Num executors = 3 worker + 2 spoutTask + 2 boltTask =7 why ? why ? why ? 2021 10 30 ，原因是 task 數(shù)據(jù)量=（worker+ + boltTask*2 ）

測試5

==================================================================================== [root@storm51 ~]# storm jar storm-1.0-SNAPSHOT-jar-with-dependencies.jar com.tiza.leo.bigdata.storm.test02Parallel.SimpleTopology tp5 tp5 3 2 2 2 2 {topologyName='tp5', prefix='tp5', workers=3, spoutParallelHint=2, spoutTasks=2, boltParallelHint=2, boltTasks=2} ------------------------------------------------------------------------------- 結論 并行度(parallelHint) Num executors = 3 worker + 2 spoutTask + 2 boltTask =7

測試6

==================================================================================== [root@storm51 ~]# storm jar storm-1.0-SNAPSHOT-jar-with-dependencies.jar com.tiza.leo.bigdata.storm.test02Parallel.SimpleTopology tp6 tp6 1 2 4 2 4 {topologyName='tp6', prefix='tp6', workers=1, spoutParallelHint=2, spoutTasks=4, boltParallelHint=2, boltTasks=4} ------------------------------------------------------------------------------- 結論 并行度(parallelHint) Num executors = 1 worker + 4 spoutTask + 4 boltTask =9

測試7

==================================================================================== [root@storm51 ~]# storm jar storm-1.0-SNAPSHOT-jar-with-dependencies.jar com.tiza.leo.bigdata.storm.test02Parallel.SimpleTopology tp7 tp7 1 1 4 1 4 {topologyName='tp7', prefix='tp7', workers=1, spoutParallelHint=1, spoutTasks=4, boltParallelHint=1, boltTasks=4} ------------------------------------------------------------------------------- 結論 并行度(parallelHint) Num executors = 1 worker + 4 spoutTask + 4 boltTask =9

之前的參數(shù)是 -n 1 -e tp7-SimpleSpout=4 -e tp7-SimpleBolt=4 修改為 storm reblance tp7 -n 2 -e tp7-SimpleSpout=8 -e tp7-SimpleBolt=8

----------------------------topo名稱-------------2個;----spout 4個 -----bolt 4 個 ----- bin/storm rebalance SampleStormClusterTopology -n 2 -e SampleSpout=4 -e SapmleBolt=4 [root@storm51 ~]# storm rebalance tp7 -n 2 -e tp7-SimpleSpout=8 -e tp7-SimpleBolt=8

P8汪文君Apache Storm1.2.2實戰(zhàn)-08講-通過程序運行分析Topology的并行度-下 17:39

見上

P9汪文君Apache Storm1.2.2實戰(zhàn)-09講-Storm Topology的Rebalance 10:41

見上

P10汪文君Apache Storm1.2.2實戰(zhàn)-10講-分析Storm Topology并行度內容補充 11:26

P11汪文君Apache Storm1.2.2實戰(zhàn)-11講-shuffle grouping數(shù)據(jù)分組詳解 25:46

P12汪文君Apache Storm1.2.2實戰(zhàn)-12講-shuffle grouping數(shù)據(jù)分組詳解（補充） 05:49

P13汪文君Apache Storm1.2.2實戰(zhàn)-13講-fields grouping數(shù)據(jù)分組詳解（戴耳機聽） 23:29

P14汪文君Apache Storm1.2.2實戰(zhàn)-14講-all grouping講解前的分析過程 20:26

P15汪文君Apache Storm1.2.2實戰(zhàn)-15講-all grouping詳細講解 14:16

P16汪文君Apache Storm1.2.2實戰(zhàn)-16講-global grouping詳細講解 25:51

P17汪文君Apache Storm1.2.2實戰(zhàn)-17講-direct grouping詳細講解 26:17

P18汪文君Apache Storm1.2.2實戰(zhàn)-18講-none grouping& localOrShuffe grouping 04:52

P19汪文君Apache Storm1.2.2實戰(zhàn)-19講-如何自定義storm grouping詳解 33:33

P20汪文君Apache Storm1.2.2實戰(zhàn)-20講-綜合案例之電信號碼主叫被叫實時統(tǒng)計 24:01

P21汪文君Apache Storm1.2.2實戰(zhàn)-21講-綜合案例之Word Count 25:05

P22汪文君Apache Storm1.2.2實戰(zhàn)-22講-Storm擔保數(shù)據(jù)被處理ack,failed,timeout,exeption講解（戴耳機聽） 26:53

P23汪文君Apache Storm1.2.2實戰(zhàn)-23講-Storm Fully擔保數(shù)據(jù)被處理的案例分析 15:38

P24汪文君Apache Storm1.2.2實戰(zhàn)-24講-一些擔保數(shù)據(jù)被處理的方案講解 16:30

P25汪文君Apache Storm1.2.2實戰(zhàn)-25講-高級Tuple的使用（Tick Tuple） 16:42

總結

以上是生活随笔為你收集整理的大数据 流式计算 apache storm 学习笔记 01 ---汪文君的全部內容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網(wǎng)站內容還不錯，歡迎將生活随笔推薦給好友。