文章目錄

• • 1. 問題描述
  • 2. spark on yarn cluster模式的解決方案
  • 3. 使用maven-shade-plugin

1. 問題描述

??spark項目的包和cdh集群中的沖突了，而且我暫時使用的是client 模式在yarn上部署（如果是cluster模式可以使用spark的config配置來解決）,所以只能使用 maven-shade-plugin的relocation功能來實現。
當前為spark2.3

2. spark on yarn cluster模式的解決方案

將下面兩個參數都設置為true
spark.driver.userClassPathFirst

(Experimental) Whether to give user-added jars precedence over Spark’s own jars when loading classes in the driver. This feature can be used to mitigate conflicts between Spark’s dependencies and user dependencies. It is currently an experimental feature. This is used in cluster mode only.

spark.executor.userClassPathFirst

(Experimental) Same functionality as spark.driver.userClassPathFirst, but applied to executor instances.

參考這里

3. 使用maven-shade-plugin

<plugin><groupId>org.apache.maven.plugins</groupId><artifactId>maven-shade-plugin</artifactId><!--<version>3.2.4</version>--><version>3.0.0</version><executions><!-- Run shade goal on package phase --><execution><phase>package</phase><goals><goal>shade</goal></goals><configuration><filters><filter><!-- Do not copy the signatures in the META-INF folder.Otherwise, this might cause SecurityExceptions when using the JAR. --><artifact>*:*</artifact><excludes><exclude>META-INF/*.SF</exclude><exclude>META-INF/*.DSA</exclude><exclude>META-INF/*.RSA</exclude></excludes></filter></filters><relocations><!-- 這里設置的relocation操作 --><relocation><pattern>org.apache.http</pattern><shadedPattern>org.user_shade.apache.http</shadedPattern></relocation></relocations></configuration></execution></executions></plugin>

執行

mvn clean package -Pprod

報error錯誤，打開調試功能再看一下

mvn -X clean package -Pprod

錯誤信息如下

[INFO] ------------------------------------------------------------------------ [INFO] BUILD FAILURE [INFO] ------------------------------------------------------------------------ [INFO] Total time: 01:02 min [INFO] Finished at: 2020-09-10T10:07:52+08:00 [INFO] Final Memory: 106M/1508M [INFO] ------------------------------------------------------------------------ [ERROR] Failed to execute goal org.apache.maven.plugins:maven-shade-plugin:3.0.0:shade (default) on project user-profile: Error creating shaded jar: null: IllegalArgumentException -> [Help 1] org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.apache.maven.plugins:maven-shade-plugin:3.0.0:shade (default) on project user-profile: Error creating shaded jar: nullat org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:213)at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:154)at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:146)at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117)at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:81)at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:51)at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:128)at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:309)at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:194)at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:107)at org.apache.maven.cli.MavenCli.execute (MavenCli.java:955)at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:290)at org.apache.maven.cli.MavenCli.main (MavenCli.java:194)at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)at sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)at sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)at java.lang.reflect.Method.invoke (Method.java:498)at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:289)at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:229)at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:415)at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:356) Caused by: org.apache.maven.plugin.MojoExecutionException: Error creating shaded jar: nullat org.apache.maven.plugins.shade.mojo.ShadeMojo.execute (ShadeMojo.java:546)at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:134)at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:208)at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:154)at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:146)at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117)at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:81)at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:51)at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:128)at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:309)at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:194)at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:107)at org.apache.maven.cli.MavenCli.execute (MavenCli.java:955)at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:290)at org.apache.maven.cli.MavenCli.main (MavenCli.java:194)at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)at sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)at sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)at java.lang.reflect.Method.invoke (Method.java:498)at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:289)at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:229)at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:415)at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:356) Caused by: java.lang.IllegalArgumentExceptionat org.objectweb.asm.ClassReader.<init> (Unknown Source)at org.objectweb.asm.ClassReader.<init> (Unknown Source)at org.objectweb.asm.ClassReader.<init> (Unknown Source)at org.apache.maven.plugins.shade.DefaultShader.addRemappedClass (DefaultShader.java:418)at org.apache.maven.plugins.shade.DefaultShader.shadeSingleJar (DefaultShader.java:220)at org.apache.maven.plugins.shade.DefaultShader.shadeJars (DefaultShader.java:181)at org.apache.maven.plugins.shade.DefaultShader.shade (DefaultShader.java:104)at org.apache.maven.plugins.shade.mojo.ShadeMojo.execute (ShadeMojo.java:442)at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:134)at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:208)at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:154)at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:146)at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117)at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:81)at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:51)at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:128)at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:309)at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:194)at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:107)at org.apache.maven.cli.MavenCli.execute (MavenCli.java:955)at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:290)at org.apache.maven.cli.MavenCli.main (MavenCli.java:194)at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)at sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)at sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)at java.lang.reflect.Method.invoke (Method.java:498)at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:289)at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:229)at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:415)at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:356) [ERROR] [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException

google了一下，發現可能是這個錯誤,好像是插件依賴的asm等有問題，直接升級plugin版本，

Affects Version/s:
3.1.1
Fix Version/s:
3.2.0

使用

<groupId>org.apache.maven.plugins</groupId> <artifactId>maven-shade-plugin</artifactId> <version>3.2.4</version>

解決問題

超強干貨來襲 云風專訪：近40年碼齡，通宵達旦的技術人生

總結

以上是生活随笔為你收集整理的spark-jar冲突解决方案的全部內容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網站內容還不錯，歡迎將生活随笔推薦給好友。