MyException - 我的异常网
当前位置:我的异常网» 开源软件 » Spark on YARN - Clinet模式运行spark报错有关问题

Spark on YARN - Clinet模式运行spark报错有关问题

www.MyException.Cn  网友分享于:2018-04-06  浏览:0次
Spark on YARN - Clinet模式运行spark报错问题

应用场景

安装部署完完全分布式的spark后,发现yarn-cluster模式可以运行不报错,但是yarn-client报错,无法进行计算PI的值,导致spark并不能使用,报错信息如下所示,只需要修改yarn的配置即可!

 

操作方案

#./bin/spark-submit--classorg.apache.spark.examples.SparkPi--masteryarn-clientlib/spark-examples-1.6.1-hadoop2.6.0.jar

 

报错信息

[root@hadoop0 spark1.6.1]# ./bin/spark-submit --class org.apache.spark.examples.SparkPi --master yarn-client lib/spark-examples-1.6.1-hadoop2.6.0.jar 17/11/1616:04:59 INFO spark.SparkContext: Running Spark version 1.6.117/11/1616:05:00 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 17/11/1616:05:00 INFO spark.SecurityManager: Changing view acls to: root 17/11/1616:05:00 INFO spark.SecurityManager: Changing modify acls to: root 17/11/1616:05:00 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root); users with modify permissions: Set(root) 17/11/1616:05:00 INFO util.Utils: Successfully started service 'sparkDriver' on port 56204.17/11/1616:05:00 INFO slf4j.Slf4jLogger: Slf4jLogger started 17/11/1616:05:01 INFO Remoting: Starting remoting 17/11/1616:05:01 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriverActorSystem@192.168.210.70:56916]17/11/1616:05:01 INFO util.Utils: Successfully started service 'sparkDriverActorSystem' on port 56916.17/11/1616:05:01 INFO spark.SparkEnv: Registering MapOutputTracker 17/11/1616:05:01 INFO spark.SparkEnv: Registering BlockManagerMaster 17/11/1616:05:01 INFO storage.DiskBlockManager: Created local directory at /tmp/blockmgr-9e904d0f-0d09-4c9a-b523-86dc52613223 17/11/1616:05:01 INFO storage.MemoryStore: MemoryStore started with capacity 511.1 MB 17/11/1616:05:01 INFO spark.SparkEnv: Registering OutputCommitCoordinator 17/11/1616:05:01 INFO server.Server: jetty-8.y.z-SNAPSHOT 17/11/1616:05:01 INFO server.AbstractConnector: Started SelectChannelConnector@0.0.0.0:404017/11/1616:05:01 INFO util.Utils: Successfully started service 'SparkUI' on port 4040.17/11/1616:05:01 INFO ui.SparkUI: Started SparkUI at http://192.168.210.70:404017/11/1616:05:01 INFO spark.HttpFileServer: HTTP File server directory is /tmp/spark-a3aae429-abe4-4bcb-b73e-8fe359aa92d9/httpd-742c21cf-89df-4af2-8b81-431523fe7bfd 17/11/1616:05:01 INFO spark.HttpServer: Starting HTTP Server 17/11/1616:05:01 INFO server.Server: jetty-8.y.z-SNAPSHOT 17/11/1616:05:01 INFO server.AbstractConnector: Started SocketConnector@0.0.0.0:3744017/11/1616:05:01 INFO util.Utils: Successfully started service 'HTTP file server' on port 37440.17/11/1616:05:01 INFO spark.SparkContext: Added JAR file:/opt/spark1.6.1/lib/spark-examples-1.6.1-hadoop2.6.0.jar at http://192.168.210.70:37440/jars/spark-examples-1.6.1-hadoop2.6.0.jar with timestamp 151081950161817/11/1616:05:01 INFO client.RMProxy: Connecting to ResourceManager at hadoop0/192.168.210.70:803217/11/1616:05:01 INFO yarn.Client: Requesting a new application from cluster with3 NodeManagers 17/11/1616:05:01 INFO yarn.Client: Verifying our application has not requested more than the maximum memory capability of the cluster (12288 MB per container) 17/11/1616:05:01 INFO yarn.Client: Will allocate AM container, with896 MB memory including 384 MB overhead 17/11/1616:05:01 INFO yarn.Client: Setting up container launch context for our AM 17/11/1616:05:01 INFO yarn.Client: Setting up the launch environment for our AM container 17/11/1616:05:01 INFO yarn.Client: Preparing resources for our AM container 17/11/1616:05:02 INFO yarn.Client: Uploading resource file:/opt/spark1.6.1/lib/spark-assembly-1.6.1-hadoop2.6.0.jar -> hdfs://hadoop0:9000/user/root/.sparkStaging/application_1510653707211_0005/spark-assembly-1.6.1-hadoop2.6.0.jar17/11/1616:05:04 INFO yarn.Client: Uploading resource file:/tmp/spark-a3aae429-abe4-4bcb-b73e-8fe359aa92d9/__spark_conf__7623958375810260855.zip -> hdfs://hadoop0:9000/user/root/.sparkStaging/application_1510653707211_0005/__spark_conf__7623958375810260855.zip17/11/1616:05:04 INFO spark.SecurityManager: Changing view acls to: root 17/11/1616:05:04 INFO spark.SecurityManager: Changing modify acls to: root 17/11/1616:05:04 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root); users with modify permissions: Set(root) 17/11/1616:05:04 INFO yarn.Client: Submitting application 5to ResourceManager 17/11/1616:05:04 INFO impl.YarnClientImpl: Submitted application application_1510653707211_0005 17/11/1616:05:06 INFO yarn.Client: Application report for application_1510653707211_0005 (state: ACCEPTED) 17/11/1616:05:06 INFO yarn.Client: client token: N/A diagnostics: N/A ApplicationMaster host: N/A ApplicationMaster RPC port: -1 queue: default start time: 1510819504598 final status: UNDEFINED tracking URL: http://hadoop0:8088/proxy/application_1510653707211_0005/ user: root 17/11/1616:05:07 INFO yarn.Client: Application report for application_1510653707211_0005 (state: ACCEPTED) 17/11/1616:05:08 INFO yarn.Client: Application report for application_1510653707211_0005 (state: ACCEPTED) 17/11/1616:05:09 INFO yarn.Client: Application report for application_1510653707211_0005 (state: ACCEPTED) 17/11/1616:05:10 INFO yarn.Client: Application report for application_1510653707211_0005 (state: ACCEPTED) 17/11/1616:05:11 INFO yarn.Client: Application report for application_1510653707211_0005 (state: ACCEPTED) 17/11/1616:05:12 INFO yarn.Client: Application report for application_1510653707211_0005 (state: ACCEPTED) 17/11/1616:05:13 INFO yarn.Client: Application report for application_1510653707211_0005 (state: ACCEPTED) 17/11/1616:05:14 INFO yarn.Client: Application report for application_1510653707211_0005 (state: ACCEPTED) 17/11/1616:05:15 ERROR server.TransportRequestHandler: Error sending result RpcResponse{requestId=89567902 java.nio.channels.ClosedChannelException 17/11/1616:05:15 INFO yarn.Client: Application report for application_1510653707211_0005 (state: ACCEPTED) 17/11/1616:05:16 INFO yarn.Client: Application report for application_1510653707211_0005 (state: ACCEPTED) 17/11/1616:05:17 INFO yarn.Client: Application report for application_1510653707211_0005 (state: ACCEPTED) 17/11/1616:05:18 INFO yarn.Client: Application report for application_1510653707211_0005 (state: ACCEPTED) 17/11/1616:05:18 INFO cluster.YarnSchedulerBackend$YarnSchedulerEndpoint: ApplicationMaster registered as17/11/1616:05:18 INFO cluster.YarnClientSchedulerBackend: Add WebUI Filter. org.apache.hadoop.yarn.server.07211_0005 17/11/1616:05:18 INFO ui.JettyUtils: Adding filter: org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFi 17/11/1616:05:19 INFO yarn.Client: Application report for application_1510653707211_0005 (state: RUNNING) 17/11/1616:05:19 INFO yarn.Client: client token: N/A diagnostics: N/A ApplicationMaster host: 192.168.210.71 ApplicationMaster RPC port: 0 queue: default start time: 1510819504598 final status: UNDEFINED tracking URL: http://hadoop0:8088/proxy/application_1510653707211_0005/ user: root 17/11/1616:05:19 INFO cluster.YarnClientSchedulerBackend: Application application_1510653707211_0005 has s 17/11/1616:05:19 INFO util.Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockT 17/11/1616:05:19 INFO netty.NettyBlockTransferService: Server created on 6093217/11/1616:05:19 INFO storage.BlockManagerMaster: Trying to register BlockManager 17/11/1616:05:19 INFO storage.BlockManagerMasterEndpoint: Registering block manager 192.168.210.70:60932 w 17/11/1616:05:19 INFO storage.BlockManagerMaster: Registered BlockManager 17/11/1616:05:22 INFO cluster.YarnClientSchedulerBackend: Registered executor NettyRpcEndpointRef(null) (h 17/11/1616:05:22 INFO storage.BlockManagerMasterEndpoint: Registering block manager hadoop1:35613with2.717/11/1616:05:22 ERROR cluster.YarnClientSchedulerBackend: Yarn application has already exited with state 17/11/1616:05:22 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/metrics/json,null} 17/11/1616:05:22 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/stage/kill,nul 17/11/1616:05:22 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/api,null} 17/11/1616:05:22 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/,null} 17/11/1616:05:22 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/static,null} 17/11/1616:05:22 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/executors/threadDump/ 17/11/1616:05:22 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/executors/threadDump, 17/11/1616:05:22 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/executors/json,null} 17/11/1616:05:22 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/executors,null} 17/11/1616:05:22 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/environment/json,null17/11/1616:05:22 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/environment,null} 17/11/1616:05:22 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/storage/rdd/json,null17/11/1616:05:22 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/storage/rdd,null} 17/11/1616:05:22 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/storage/json,null} 17/11/1616:05:22 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/storage,null} 17/11/1616:05:22 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/pool/json,null17/11/1616:05:22 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/pool,null} 17/11/1616:05:22 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/stage/json,nul 17/11/1616:05:22 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/stage,null} 17/11/1616:05:22 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/json,null} 17/11/1616:05:22 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages,null} 17/11/1616:05:22 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/jobs/job/json,null} 17/11/1616:05:22 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/jobs/job,null} 17/11/1616:05:22 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/jobs/json,null} 17/11/1616:05:22 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/jobs,null} 17/11/1616:05:22 INFO cluster.YarnClientSchedulerBackend: Registered executor NettyRpcEndpointRef(null) (h 17/11/1616:05:22 INFO ui.SparkUI: Stopped Spark web UI at http://192.168.210.70:404017/11/1616:05:22 INFO cluster.YarnClientSchedulerBackend: Shutting down all executors 17/11/1616:05:22 INFO cluster.YarnClientSchedulerBackend: Asking each executor to shut down 17/11/1616:05:22 INFO cluster.YarnClientSchedulerBackend: Stopped 17/11/1616:05:22 INFO spark.MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped! 17/11/1616:05:22 INFO storage.BlockManagerMasterEndpoint: Registering block manager hadoop2:51640with2.717/11/1616:05:22 INFO storage.MemoryStore: MemoryStore cleared 17/11/1616:05:22 INFO storage.BlockManager: BlockManager stopped 17/11/1616:05:22 ERROR scheduler.LiveListenerBus: SparkListenerBus has already stopped! Dropping event Spa 17/11/1616:05:22 INFO storage.BlockManagerMaster: BlockManagerMaster stopped 17/11/1616:05:22 INFO scheduler.OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoord 17/11/1616:05:22 INFO spark.SparkContext: Successfully stopped SparkContext 17/11/1616:05:22 INFO remote.RemoteActorRefProvider$RemotingTerminator: Shutting down remote daemon. 17/11/1616:05:22 INFO remote.RemoteActorRefProvider$RemotingTerminator: Remote daemon shut down; proceedin 17/11/1616:05:22 INFO cluster.YarnClientSchedulerBackend: SchedulerBackend is ready for scheduling beginni 17/11/1616:05:22 ERROR spark.SparkContext: Error initializing SparkContext. java.lang.NullPointerException at org.apache.spark.SparkContext.<init>(SparkContext.scala:584) at org.apache.spark.examples.SparkPi$.main(SparkPi.scala:29) at org.apache.spark.examples.SparkPi.main(SparkPi.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.sc at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181) at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) 17/11/1616:05:22 INFO spark.SparkContext: SparkContext already stopped. Exception in thread "main" java.lang.NullPointerException at org.apache.spark.SparkContext.<init>(SparkContext.scala:584) at org.apache.spark.examples.SparkPi$.main(SparkPi.scala:29) at org.apache.spark.examples.SparkPi.main(SparkPi.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.sc at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181) at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) 17/11/1616:05:22 INFO util.ShutdownHookManager: Shutdown hook called 17/11/1616:05:22 INFO util.ShutdownHookManager: Deleting directory /tmp/spark-a3aae429-abe4-4bcb-b73e-8fe3 17/11/1616:05:22 INFO remote.RemoteActorRefProvider$RemotingTerminator: Remoting shut down. 17/11/1616:05:22 INFO util.ShutdownHookManager: Deleting directory /tmp/spark-a3aae429-abe4-4bcb-b73e-8fe

 

修改yarn-site.xml配置添加最后两个配置:

 

# vim /opt/hadoop2.6.0/etc/hadoop/yarn-site.xml <property><name>yarn.resourcemanager.hostname</name><value>hadoop0</value><property><description>The http address of the RM web application.</description><name>yarn.resourcemanager.webapp.address</name><value>${yarn.resourcemanager.hostname}:8088</value><name>yarn.resourcemanager.resource-tracker.address</name><value>${yarn.resourcemanager.hostname}:8031</value></property><property><description>The address of the RM admin interface.</description><name>yarn.resourcemanager.admin.address</name><value>${yarn.resourcemanager.hostname}:8033</value></property><property><name>yarn.nodemanager.aux-services</name><value>mapreduce_shuffle</value></property><property><name>yarn.scheduler.maximum-allocation-mb</name><value>8182</value><discription>每个节点可用内存,单位MB,默认8182MB</discription></property><property><name>yarn.nodemanager.pmem-check-enabled</name><value>false</value></property><property><name>yarn.nodemanager.vmem-check-enabled</name><value>false</value></property> # 添加最后两个false的属性即可!

 

 

 

 

 

 

文章评论

鲜为人知的编程真相
鲜为人知的编程真相
“肮脏的”IT工作排行榜
“肮脏的”IT工作排行榜
亲爱的项目经理,我恨你
亲爱的项目经理,我恨你
为什么程序员都是夜猫子
为什么程序员都是夜猫子
如何区分一个程序员是“老手“还是“新手“?
如何区分一个程序员是“老手“还是“新手“?
Web开发人员为什么越来越懒了?
Web开发人员为什么越来越懒了?
做程序猿的老婆应该注意的一些事情
做程序猿的老婆应该注意的一些事情
我的丈夫是个程序员
我的丈夫是个程序员
不懂技术不要对懂技术的人说这很容易实现
不懂技术不要对懂技术的人说这很容易实现
Java程序员必看电影
Java程序员必看电影
Java 与 .NET 的平台发展之争
Java 与 .NET 的平台发展之争
Google伦敦新总部 犹如星级庄园
Google伦敦新总部 犹如星级庄园
程序员周末都喜欢做什么?
程序员周末都喜欢做什么?
60个开发者不容错过的免费资源库
60个开发者不容错过的免费资源库
Web开发者需具备的8个好习惯
Web开发者需具备的8个好习惯
程序员的鄙视链
程序员的鄙视链
老美怎么看待阿里赴美上市
老美怎么看待阿里赴美上市
团队中“技术大拿”并非越多越好
团队中“技术大拿”并非越多越好
老程序员的下场
老程序员的下场
写给自己也写给你 自己到底该何去何从
写给自己也写给你 自己到底该何去何从
旅行,写作,编程
旅行,写作,编程
程序员的一天:一寸光阴一寸金
程序员的一天:一寸光阴一寸金
为啥Android手机总会越用越慢?
为啥Android手机总会越用越慢?
聊聊HTTPS和SSL/TLS协议
聊聊HTTPS和SSL/TLS协议
一个程序员的时间管理
一个程序员的时间管理
程序员和编码员之间的区别
程序员和编码员之间的区别
程序员必看的十大电影
程序员必看的十大电影
5款最佳正则表达式编辑调试器
5款最佳正则表达式编辑调试器
程序员眼里IE浏览器是什么样的
程序员眼里IE浏览器是什么样的
 程序员的样子
程序员的样子
那些性感的让人尖叫的程序员
那些性感的让人尖叫的程序员
看13位CEO、创始人和高管如何提高工作效率
看13位CEO、创始人和高管如何提高工作效率
中美印日四国程序员比较
中美印日四国程序员比较
每天工作4小时的程序员
每天工作4小时的程序员
程序员最害怕的5件事 你中招了吗?
程序员最害怕的5件事 你中招了吗?
科技史上最臭名昭著的13大罪犯
科技史上最臭名昭著的13大罪犯
“懒”出效率是程序员的美德
“懒”出效率是程序员的美德
漫画:程序员的工作
漫画:程序员的工作
代码女神横空出世
代码女神横空出世
我跳槽是因为他们的显示器更大
我跳槽是因为他们的显示器更大
2013年中国软件开发者薪资调查报告
2013年中国软件开发者薪资调查报告
2013年美国开发者薪资调查报告
2013年美国开发者薪资调查报告
程序猿的崛起——Growth Hacker
程序猿的崛起——Growth Hacker
如何成为一名黑客
如何成为一名黑客
10个帮程序员减压放松的网站
10个帮程序员减压放松的网站
我是如何打败拖延症的
我是如何打败拖延症的
当下全球最炙手可热的八位少年创业者
当下全球最炙手可热的八位少年创业者
十大编程算法助程序员走上高手之路
十大编程算法助程序员走上高手之路
什么才是优秀的用户界面设计
什么才是优秀的用户界面设计
软件开发程序错误异常ExceptionCopyright © 2009-2015 MyException 版权所有