SparkSubmit類(伴生對象)

根據提交腳本定位到org.apache.spark.deploy.SparkSubmit

bin/spark-submit \ --class org.apache.spark.examples.SparkPi \ --master yarn \ --deploy-mode cluster \ ./examples/jars/spark-examples_2.12-3.0.0.jar \ 10

伴生對象:

org.apache.spark.deploy.SparkSubmit

main

override def main(args: Array[String]): Unit = {val submit = new SparkSubmit() {self =>override protected def parseArguments(args: Array[String]): SparkSubmitArguments = {new SparkSubmitArguments(args) {override protected def logInfo(msg: => String): Unit = self.logInfo(msg)override protected def logWarning(msg: => String): Unit = self.logWarning(msg)override protected def logError(msg: => String): Unit = self.logError(msg)}}override protected def logInfo(msg: => String): Unit = printMessage(msg)override protected def logWarning(msg: => String): Unit = printMessage(s"Warning: $msg")override protected def logError(msg: => String): Unit = printMessage(s"Error: $msg")override def doSubmit(args: Array[String]): Unit = {try {super.doSubmit(args)} catch {case e: SparkUserAppException =>exitFn(e.exitCode)}}}submit.doSubmit(args)}

doSubmit(args)

第80行：

def doSubmit(args: Array[String]): Unit = {// Initialize logging if it hasn't been done yet. Keep track of whether logging needs to// be reset before the application starts.val uninitLog = initializeLogIfNecessary(true, silent = true)val appArgs = parseArguments(args)if (appArgs.verbose) {logInfo(appArgs.toString)}appArgs.action match {case SparkSubmitAction.SUBMIT => submit(appArgs, uninitLog)case SparkSubmitAction.KILL => kill(appArgs)case SparkSubmitAction.REQUEST_STATUS => requestStatus(appArgs)case SparkSubmitAction.PRINT_VERSION => printVersion()}}

parseArguments(args)

第85行：

protected def parseArguments(args: Array[String]): SparkSubmitArguments = {new SparkSubmitArguments(args)}

new SparkSubmitArguments(args)

108：

// Set parameters from command line argumentsparse(args.asJava)

進入了java代碼：

進入handle：

搜索handle：

查看常量：

查看MASTER常量：

查看CLASS：

搜索action

如果沒有給值默認就是SUBMIT：

submit方法

isStandaloneCluster

doRunMain runMain

準備提交的環境

通過反射獲取類

傳入類名，獲取類的class

是否繼承SparkApplication

new JavaMainApplication(mainClass)

YARN_CLUSTER_SUBMIT_CLASS

看yarn源碼

<!--看yarn源碼--><dependency><groupId>org.apache.spark</groupId><artifactId>spark-yarn_${spark.scala.version}</artifactId><version>${spark.version}</version></dependency>

YarnClusterApplication extends SparkApplication

start方法

總結

以上是生活随笔為你收集整理的SparkSubmit源码解读记录的全部內容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網站內容還不錯，歡迎將生活随笔推薦給好友。