在dss中运行scala任务报execute code failed! java.lang.NullPointerException: null #2650

subaisubai · 2022-08-09T05:42:22Z

subaisubai
Aug 9, 2022

在dss中运行scala任务报错。请问以下是为什么。此外，sparkSQL和pythonSpark都能正常运行。
版本信息为：dss1.1.0 linkis1.1.1 scala2.12.10 spark3.0.1
脚本内容为：println("123")
日志信息为：

2022-08-09 13:39:27.077 ERROR [Linkis-Default-Scheduler-Thread-8] org.apache.linkis.engineplugin.spark.executor.SparkScalaExecutor 58 error - execute code failed! java.lang.NullPointerException: null
	at scala.tools.nsc.interpreter.ILoop.verbosity(ILoop.scala:753) ~[scala-compiler-2.12.10.jar:?]
	at scala.tools.nsc.interpreter.ILoop.$anonfun$standardCommands$18(ILoop.scala:206) ~[scala-compiler-2.12.10.jar:?]
	at scala.tools.nsc.interpreter.LoopCommands$LoopCommand$.$anonfun$nullary$1(LoopCommands.scala:68) ~[scala-compiler-2.12.10.jar:?]
	at scala.tools.nsc.interpreter.LoopCommands$NullaryCmd.apply(LoopCommands.scala:155) ~[scala-compiler-2.12.10.jar:?]
	at scala.tools.nsc.interpreter.LoopCommands.colonCommand(LoopCommands.scala:121) ~[scala-compiler-2.12.10.jar:?]
	at scala.tools.nsc.interpreter.LoopCommands.colonCommand$(LoopCommands.scala:119) ~[scala-compiler-2.12.10.jar:?]
	at scala.tools.nsc.interpreter.ILoop.colonCommand(ILoop.scala:50) ~[scala-compiler-2.12.10.jar:?]
	at scala.tools.nsc.interpreter.ILoop.command(ILoop.scala:760) ~[scala-compiler-2.12.10.jar:?]
	at org.apache.linkis.engineplugin.spark.executor.SparkScalaExecutor.$anonfun$bindSparkSession$3(SparkScalaExecutor.scala:284) ~[linkis-engineplugin-spark-1.1.1.jar:1.1.1]
	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23) ~[scala-library-2.12.10.jar:?]
	at scala.tools.nsc.interpreter.IMain.beSilentDuring(IMain.scala:220) ~[scala-compiler-2.12.10.jar:?]
	at org.apache.linkis.engineplugin.spark.executor.SparkScalaExecutor.bindSparkSession(SparkScalaExecutor.scala:283) ~[linkis-engineplugin-spark-1.1.1.jar:1.1.1]
	at org.apache.linkis.engineplugin.spark.executor.SparkScalaExecutor.lazyLoadILoop(SparkScalaExecutor.scala:234) ~[linkis-engineplugin-spark-1.1.1.jar:1.1.1]
	at org.apache.linkis.engineplugin.spark.executor.SparkScalaExecutor.runCode(SparkScalaExecutor.scala:124) ~[linkis-engineplugin-spark-1.1.1.jar:1.1.1]
	at org.apache.linkis.engineplugin.spark.executor.SparkEngineConnExecutor.$anonfun$executeLine$8(SparkEngineConnExecutor.scala:87) ~[linkis-engineplugin-spark-1.1.1.jar:1.1.1]
	at org.apache.linkis.common.utils.Utils$.tryFinally(Utils.scala:61) ~[linkis-common-1.1.1.jar:1.1.1]
	at org.apache.linkis.engineplugin.spark.executor.SparkEngineConnExecutor.$anonfun$executeLine$1(SparkEngineConnExecutor.scala:87) ~[linkis-engineplugin-spark-1.1.1.jar:1.1.1]
	at org.apache.linkis.common.utils.Utils$.tryFinally(Utils.scala:61) ~[linkis-common-1.1.1.jar:1.1.1]
	at org.apache.linkis.engineplugin.spark.executor.SparkEngineConnExecutor.executeLine(SparkEngineConnExecutor.scala:96) ~[linkis-engineplugin-spark-1.1.1.jar:1.1.1]
	at org.apache.linkis.engineconn.computation.executor.execute.ComputationExecutor.$anonfun$toExecuteTask$14(ComputationExecutor.scala:181) ~[linkis-computation-engineconn-1.1.1.jar:1.1.1]
	at org.apache.linkis.common.utils.Utils$.tryCatch(Utils.scala:40) ~[linkis-common-1.1.1.jar:1.1.1]
	at org.apache.linkis.engineconn.computation.executor.execute.ComputationExecutor.$anonfun$toExecuteTask$13(ComputationExecutor.scala:183) ~[linkis-computation-engineconn-1.1.1.jar:1.1.1]
	at org.apache.linkis.engineconn.computation.executor.execute.ComputationExecutor.$anonfun$toExecuteTask$13$adapted(ComputationExecutor.scala:176) ~[linkis-computation-engineconn-1.1.1.jar:1.1.1]
	at scala.collection.immutable.Range.foreach(Range.scala:158) ~[scala-library-2.12.10.jar:?]
	at org.apache.linkis.engineconn.computation.executor.execute.ComputationExecutor.$anonfun$toExecuteTask$1(ComputationExecutor.scala:176) ~[linkis-computation-engineconn-1.1.1.jar:1.1.1]
	at org.apache.linkis.common.utils.Utils$.tryFinally(Utils.scala:61) ~[linkis-common-1.1.1.jar:1.1.1]
	at org.apache.linkis.engineconn.computation.executor.execute.ComputationExecutor.toExecuteTask(ComputationExecutor.scala:228) ~[linkis-computation-engineconn-1.1.1.jar:1.1.1]
	at org.apache.linkis.engineconn.computation.executor.execute.ComputationExecutor.$anonfun$execute$2(ComputationExecutor.scala:243) ~[linkis-computation-engineconn-1.1.1.jar:1.1.1]
	at org.apache.linkis.common.utils.Utils$.tryFinally(Utils.scala:61) ~[linkis-common-1.1.1.jar:1.1.1]
	at org.apache.linkis.engineconn.acessible.executor.entity.AccessibleExecutor.ensureIdle(AccessibleExecutor.scala:55) ~[linkis-accessible-executor-1.1.1.jar:1.1.1]
	at org.apache.linkis.engineconn.acessible.executor.entity.AccessibleExecutor.ensureIdle(AccessibleExecutor.scala:49) ~[linkis-accessible-executor-1.1.1.jar:1.1.1]
	at org.apache.linkis.engineconn.computation.executor.execute.ComputationExecutor.ensureOp(ComputationExecutor.scala:135) ~[linkis-computation-engineconn-1.1.1.jar:1.1.1]
	at org.apache.linkis.engineconn.computation.executor.execute.ComputationExecutor.execute(ComputationExecutor.scala:243) ~[linkis-computation-engineconn-1.1.1.jar:1.1.1]
	at org.apache.linkis.engineconn.computation.executor.service.TaskExecutionServiceImpl.org$apache$linkis$engineconn$computation$executor$service$TaskExecutionServiceImpl$$executeTask(TaskExecutionServiceImpl.scala:288) ~[linkis-computation-engineconn-1.1.1.jar:1.1.1]
	at org.apache.linkis.engineconn.computation.executor.service.TaskExecutionServiceImpl$$anon$2.$anonfun$run$3(TaskExecutionServiceImpl.scala:221) ~[linkis-computation-engineconn-1.1.1.jar:1.1.1]
	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23) ~[scala-library-2.12.10.jar:?]
	at org.apache.linkis.common.utils.Utils$.tryCatch(Utils.scala:40) ~[linkis-common-1.1.1.jar:1.1.1]
	at org.apache.linkis.common.utils.Utils$.tryAndWarn(Utils.scala:69) ~[linkis-common-1.1.1.jar:1.1.1]
	at org.apache.linkis.engineconn.computation.executor.service.TaskExecutionServiceImpl$$anon$2.run(TaskExecutionServiceImpl.scala:219) ~[linkis-computation-engineconn-1.1.1.jar:1.1.1]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_202]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_202]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) ~[?:1.8.0_202]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) ~[?:1.8.0_202]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_202]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_202]
	at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_202]
2022-08-09 13:39:27.089 ERROR [Linkis-Default-Scheduler-Thread-8] org.apache.linkis.engineconn.computation.executor.service.TaskExecutionServiceImpl 58 error - NullPointerException: java.lang.NullPointerException: null
	at scala.tools.nsc.interpreter.ILoop.verbosity(ILoop.scala:753) ~[scala-compiler-2.12.10.jar:?]
	at scala.tools.nsc.interpreter.ILoop.$anonfun$standardCommands$18(ILoop.scala:206) ~[scala-compiler-2.12.10.jar:?]
	at scala.tools.nsc.interpreter.LoopCommands$LoopCommand$.$anonfun$nullary$1(LoopCommands.scala:68) ~[scala-compiler-2.12.10.jar:?]
	at scala.tools.nsc.interpreter.LoopCommands$NullaryCmd.apply(LoopCommands.scala:155) ~[scala-compiler-2.12.10.jar:?]
	at scala.tools.nsc.interpreter.LoopCommands.colonCommand(LoopCommands.scala:121) ~[scala-compiler-2.12.10.jar:?]
	at scala.tools.nsc.interpreter.LoopCommands.colonCommand$(LoopCommands.scala:119) ~[scala-compiler-2.12.10.jar:?]
	at scala.tools.nsc.interpreter.ILoop.colonCommand(ILoop.scala:50) ~[scala-compiler-2.12.10.jar:?]
	at scala.tools.nsc.interpreter.ILoop.command(ILoop.scala:760) ~[scala-compiler-2.12.10.jar:?]
	at org.apache.linkis.engineplugin.spark.executor.SparkScalaExecutor.$anonfun$bindSparkSession$3(SparkScalaExecutor.scala:284) ~[linkis-engineplugin-spark-1.1.1.jar:1.1.1]
	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23) ~[scala-library-2.12.10.jar:?]
	at scala.tools.nsc.interpreter.IMain.beSilentDuring(IMain.scala:220) ~[scala-compiler-2.12.10.jar:?]
	at org.apache.linkis.engineplugin.spark.executor.SparkScalaExecutor.bindSparkSession(SparkScalaExecutor.scala:283) ~[linkis-engineplugin-spark-1.1.1.jar:1.1.1]
	at org.apache.linkis.engineplugin.spark.executor.SparkScalaExecutor.lazyLoadILoop(SparkScalaExecutor.scala:234) ~[linkis-engineplugin-spark-1.1.1.jar:1.1.1]
	at org.apache.linkis.engineplugin.spark.executor.SparkScalaExecutor.runCode(SparkScalaExecutor.scala:124) ~[linkis-engineplugin-spark-1.1.1.jar:1.1.1]
	at org.apache.linkis.engineplugin.spark.executor.SparkEngineConnExecutor.$anonfun$executeLine$8(SparkEngineConnExecutor.scala:87) ~[linkis-engineplugin-spark-1.1.1.jar:1.1.1]
	at org.apache.linkis.common.utils.Utils$.tryFinally(Utils.scala:61) ~[linkis-common-1.1.1.jar:1.1.1]
	at org.apache.linkis.engineplugin.spark.executor.SparkEngineConnExecutor.$anonfun$executeLine$1(SparkEngineConnExecutor.scala:87) ~[linkis-engineplugin-spark-1.1.1.jar:1.1.1]
	at org.apache.linkis.common.utils.Utils$.tryFinally(Utils.scala:61) ~[linkis-common-1.1.1.jar:1.1.1]
	at org.apache.linkis.engineplugin.spark.executor.SparkEngineConnExecutor.executeLine(SparkEngineConnExecutor.scala:96) ~[linkis-engineplugin-spark-1.1.1.jar:1.1.1]
	at org.apache.linkis.engineconn.computation.executor.execute.ComputationExecutor.$anonfun$toExecuteTask$14(ComputationExecutor.scala:181) ~[linkis-computation-engineconn-1.1.1.jar:1.1.1]
	at org.apache.linkis.common.utils.Utils$.tryCatch(Utils.scala:40) ~[linkis-common-1.1.1.jar:1.1.1]
	at org.apache.linkis.engineconn.computation.executor.execute.ComputationExecutor.$anonfun$toExecuteTask$13(ComputationExecutor.scala:183) ~[linkis-computation-engineconn-1.1.1.jar:1.1.1]
	at org.apache.linkis.engineconn.computation.executor.execute.ComputationExecutor.$anonfun$toExecuteTask$13$adapted(ComputationExecutor.scala:176) ~[linkis-computation-engineconn-1.1.1.jar:1.1.1]
	at scala.collection.immutable.Range.foreach(Range.scala:158) ~[scala-library-2.12.10.jar:?]
	at org.apache.linkis.engineconn.computation.executor.execute.ComputationExecutor.$anonfun$toExecuteTask$1(ComputationExecutor.scala:176) ~[linkis-computation-engineconn-1.1.1.jar:1.1.1]
	at org.apache.linkis.common.utils.Utils$.tryFinally(Utils.scala:61) ~[linkis-common-1.1.1.jar:1.1.1]
	at org.apache.linkis.engineconn.computation.executor.execute.ComputationExecutor.toExecuteTask(ComputationExecutor.scala:228) ~[linkis-computation-engineconn-1.1.1.jar:1.1.1]
	at org.apache.linkis.engineconn.computation.executor.execute.ComputationExecutor.$anonfun$execute$2(ComputationExecutor.scala:243) ~[linkis-computation-engineconn-1.1.1.jar:1.1.1]
	at org.apache.linkis.common.utils.Utils$.tryFinally(Utils.scala:61) ~[linkis-common-1.1.1.jar:1.1.1]
	at org.apache.linkis.engineconn.acessible.executor.entity.AccessibleExecutor.ensureIdle(AccessibleExecutor.scala:55) ~[linkis-accessible-executor-1.1.1.jar:1.1.1]
	at org.apache.linkis.engineconn.acessible.executor.entity.AccessibleExecutor.ensureIdle(AccessibleExecutor.scala:49) ~[linkis-accessible-executor-1.1.1.jar:1.1.1]
	at org.apache.linkis.engineconn.computation.executor.execute.ComputationExecutor.ensureOp(ComputationExecutor.scala:135) ~[linkis-computation-engineconn-1.1.1.jar:1.1.1]
	at org.apache.linkis.engineconn.computation.executor.execute.ComputationExecutor.execute(ComputationExecutor.scala:243) ~[linkis-computation-engineconn-1.1.1.jar:1.1.1]
	at org.apache.linkis.engineconn.computation.executor.service.TaskExecutionServiceImpl.org$apache$linkis$engineconn$computation$executor$service$TaskExecutionServiceImpl$$executeTask(TaskExecutionServiceImpl.scala:288) ~[linkis-computation-engineconn-1.1.1.jar:1.1.1]
	at org.apache.linkis.engineconn.computation.executor.service.TaskExecutionServiceImpl$$anon$2.$anonfun$run$3(TaskExecutionServiceImpl.scala:221) ~[linkis-computation-engineconn-1.1.1.jar:1.1.1]
	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23) ~[scala-library-2.12.10.jar:?]
	at org.apache.linkis.common.utils.Utils$.tryCatch(Utils.scala:40) ~[linkis-common-1.1.1.jar:1.1.1]
	at org.apache.linkis.common.utils.Utils$.tryAndWarn(Utils.scala:69) ~[linkis-common-1.1.1.jar:1.1.1]
	at org.apache.linkis.engineconn.computation.executor.service.TaskExecutionServiceImpl$$anon$2.run(TaskExecutionServiceImpl.scala:219) ~[linkis-computation-engineconn-1.1.1.jar:1.1.1]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_202]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_202]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) ~[?:1.8.0_202]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) ~[?:1.8.0_202]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_202]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_202]
	at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_202]
2022-08-09 13:39:27.039 ERROR Job with execId-nodeexecution_hadoop_spark_1 + subJobId : 895  execute failed,21304, Task is Failed,errorMsg: NullPointerException:

ruY9527 · 2022-08-30T15:21:03Z

ruY9527
Aug 30, 2022
Collaborator

你好, 可以加下log在SparkScalaExecutor这个类,我看 scala.tools.nsc.interpreter.ILoop.verbosity(ILoop.scala:753 是null , 是不是这里获取 ILoop 出现了 null 的情况?

3 replies

subaisubai Sep 1, 2022
Author

谢谢。我按照上图圈出来的部分，修改了linkis的代码，问题解决了。

ruY9527 Sep 1, 2022
Collaborator

谢谢。我按照上图圈出来的部分，修改了linkis的代码，问题解决了。

方便的话，可以分享下修改后的代码，便于后续有同学有遇到

hehehazj Mar 18, 2023

谢谢了，就是按照这个修改的

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

在dss中运行scala任务报execute code failed! java.lang.NullPointerException: null #2650

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 1 comment 3 replies

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

在dss中运行scala任务报execute code failed! java.lang.NullPointerException: null #2650

subaisubai Aug 9, 2022

Replies: 1 comment · 3 replies

ruY9527 Aug 30, 2022 Collaborator

subaisubai Sep 1, 2022 Author

ruY9527 Sep 1, 2022 Collaborator

hehehazj Mar 18, 2023

subaisubai
Aug 9, 2022

Replies: 1 comment 3 replies

ruY9527
Aug 30, 2022
Collaborator

subaisubai Sep 1, 2022
Author

ruY9527 Sep 1, 2022
Collaborator