Spark部署
2. 部署¶
spark计算会依赖scala所以要先安装scala。
| 序列号 | IP地址 | 主机名 | 角色 | 安装软件 |
|---|---|---|---|---|
| 1 | 192.168.186.10 | master | spark master |
scala，spark |
| 2 | 192.168.186.11 | slave1 | spark work |
scala，spark |
| 3 | 192.168.186.12 | slave2 | spark work |
scala，spark |
2.1 部署scala¶
wget https://downloads.lightbend.com/scala/2.12.4/scala-2.12.4.tgz tar xf scala-2.12.4.tgz ln -sf /usr/local/scala-2.12.4 /usr/local/scala
详细步骤
[root@master src]# wget https://downloads.lightbend.com/scala/2.12.4/scala-2.12.4.tgz --2019-06-27 20:08:57-- https://downloads.lightbend.com/scala/2.12.4/scala-2.12.4.tgz Resolving downloads.lightbend.com (downloads.lightbend.com)... 99.84.239.12, 99.84.239.115, 99.84.239.98, .. .Connecting to downloads.lightbend.com (downloads.lightbend.com)|99.84.239.12|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 19741785 (19M) [application/octet-stream] Saving to: ‘scala-2.12.4.tgz’ 100%[==================================================================>] 19,741,785 1.90MB/s in 18s 2019-06-27 20:09:22 (1.07 MB/s) - ‘scala-2.12.4.tgz’ saved [19741785/19741785] [root@master src]# tar xf scala-2.12.4.tgz [root@master src]# mv scala-2.12.4 /usr/local/ [root@master src]# ln -sf /usr/local/scala-2.12.4 /usr/local/scala
2.2 部署spark¶
wget http://mirror.bit.edu.cn/apache/spark/spark-1.6.3/spark-1.6.3-bin-hadoop2.6.tgz tar xf spark-1.6.3-bin-hadoop2.6.tgz -C /usr/local/ ln -sf /usr/local/spark-1.6.3-bin-hadoop2.6 /usr/local/spark cd /usr/local/spark/conf/ cp spark-env.sh.template spark-env.sh cat>>spark-env.sh<<EOF export SCALA_HOME=/usr/local/scala export JAVA_HOME=/usr/local/jdk1.8.0_172 export HADOOP_HOME=/usr/local/hadoop-2.6.5 export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop SPARK_MASTER_IP=master SPARK_LOCAL_DIRS=/usr/local/spark SPARK_DRIVER_MEMORY=1G EOF egrep -v '#|^$' spark-env.sh cat>slaves<<EOF slave1 slave2 EOF
详细步骤
[root@master src]# tar xf spark-1.6.3-bin-hadoop2.6.tgz -C /usr/local/ [root@master src]# ln -sf /usr/local/spark-1.6.3-bin-hadoop2.6 /usr/local/spark [root@master src]# cd /usr/local/spark/conf/ [root@master conf]# cp spark-env.sh.template spark-env.sh [root@master local]# cat>>spark-env.sh<<EOF > export SCALA_HOME=/usr/local/scala > export JAVA_HOME=/usr/local/jdk1.8.0_172 > export HADOOP_HOME=/usr/local/hadoop-2.6.5 > export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop > SPARK_MASTER_IP=master > SPARK_LOCAL_DIRS=/usr/local/spark > SPARK_DRIVER_MEMORY=1G > EOF [root@master local]# egrep -v '#|^$' spark-env.sh export SCALA_HOME=/usr/local/scala export JAVA_HOME=/usr/local/jdk1.8.0_172 export HADOOP_HOME=/usr/local/hadoop-2.6.5 export HADOOP_CONF_DIR=/usr/local/hadoop-2.6.5/etc/hadoop SPARK_MASTER_IP=master SPARK_LOCAL_DIRS=/usr/local/spark SPARK_DRIVER_MEMORY=1G [root@master conf]# cat>slaves<<EOF slave1 slave2 EOF [root@master conf]# ll total 48 -rw-r--r-- 1 500 500 987 Nov 3 2016 docker.properties.template -rw-r--r-- 1 500 500 1105 Nov 3 2016 fairscheduler.xml.template -rw-r--r-- 1 500 500 1734 Nov 3 2016 log4j.properties.template -rw-r--r-- 1 500 500 6671 Nov 3 2016 metrics.properties.template -rw-r--r-- 1 root root 14 Jun 27 20:32 slaves -rw-r--r-- 1 500 500 865 Nov 3 2016 slaves.template -rw-r--r-- 1 500 500 1292 Nov 3 2016 spark-defaults.conf.template -rwxr-xr-x 1 root root 4209 Jun 27 20:22 spark-env.sh -rwxr-xr-x 1 500 500 4209 Nov 3 2016 spark-env.sh.template
2.3 分发包¶
rsync -az /usr/local/{scala-2.12.4,scala,spark,spark-1.6.3-bin-hadoop2.6} slave1:/usr/local/
rsync -az /usr/local/{scala-2.12.4,scala,spark,spark-1.6.3-bin-hadoop2.6} slave2:/usr/local/
详细步骤
[root@master ~]# rsync -az /usr/local/{scala-2.12.4,scala,spark,spark-1.6.3-bin-hadoop2.6} slave1:/usr/local/
[root@master ~]# rsync -az /usr/local/{scala-2.12.4,scala,spark,spark-1.6.3-bin-hadoop2.6} slave2:/usr/local/
spark只在master上启动和停止
2.4 启停¶
详细步骤
[root@master sbin]# ./start-all.sh starting org.apache.spark.deploy.master.Master, logging to /usr/local/spark/logs/spark-root-org.apache.spark.deploy.master.Master-1-master.out slave2: starting org.apache.spark.deploy.worker.Worker, logging to /usr/local/spark/logs/spark-root-org.apache.spark.deploy.worker.Worker-1-slave2.out slave1: starting org.apache.spark.deploy.worker.Worker, logging to /usr/local/spark/logs/spark-root-org.apache.spark.deploy.worker.Worker-1-slave1.out
启停
cd /usr/local/spark/sbin/ ./start-all.sh # 启动 ./stop-all.sh # 停止
2.5 检查¶
[root@master sbin]# jps 46224 Master 46304 Jps 27970 HQuorumPeer 21699 NameNode 43092 ResourceManager 21882 SecondaryNameNode [root@slave1 conf]# jps 29409 DataNode 117442 NodeManager 15043 Worker 15125 Jps 48264 HQuorumPeer [root@slave2 conf]# jps 17186 HQuorumPeer 36164 Jps 32533 NodeManager 36101 Worker
只要在master上看到Mater进程，slave上看到work进程，就基本判断spark安装OK。
在查看端口
[root@master sbin]# netstat -lnp|grep 8080 tcp6 0 0 :::8080 :::* LISTEN 46224/java
2.6 spark 验证¶
2.6.1 本地验证¶
cd /usr/local/spark ./bin/run-example SparkPi 10 --master local[2]
详细步骤
[root@master ~]# cd /usr/local/spark
[root@master spark]# ./bin/run-example SparkPi 10 --master local[2]
19/06/27 20:50:17 INFO spark.SparkContext: Running Spark version 1.6.3
19/06/27 20:50:17 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
19/06/27 20:50:17 INFO spark.SecurityManager: Changing view acls to: root
19/06/27 20:50:17 INFO spark.SecurityManager: Changing modify acls to: root
19/06/27 20:50:17 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root); users with modify permissions: Set(root)
19/06/27 20:50:18 INFO util.Utils: Successfully started service 'sparkDriver' on port 44014.
19/06/27 20:50:18 INFO slf4j.Slf4jLogger: Slf4jLogger started
19/06/27 20:50:18 INFO Remoting: Starting remoting
19/06/27 20:50:18 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriverActorSystem@192.168.186.10:43051]
19/06/27 20:50:18 INFO util.Utils: Successfully started service 'sparkDriverActorSystem' on port 43051.
19/06/27 20:50:18 INFO spark.SparkEnv: Registering MapOutputTracker
19/06/27 20:50:18 INFO spark.SparkEnv: Registering BlockManagerMaster
19/06/27 20:50:18 INFO storage.DiskBlockManager: Created local directory at /usr/local/spark-1.6.3-bin-hadoop2.6/blockmgr-72a5407a-0b39-418a-b9ec-73ff4d65d20a
19/06/27 20:50:18 INFO storage.MemoryStore: MemoryStore started with capacity 511.1 MB
19/06/27 20:50:18 INFO spark.SparkEnv: Registering OutputCommitCoordinator
19/06/27 20:50:18 INFO server.Server: jetty-8.y.z-SNAPSHOT
19/06/27 20:50:18 INFO server.AbstractConnector: Started SelectChannelConnector@0.0.0.0:4040
19/06/27 20:50:18 INFO util.Utils: Successfully started service 'SparkUI' on port 4040.
19/06/27 20:50:18 INFO ui.SparkUI: Started SparkUI at http://192.168.186.10:4040
19/06/27 20:50:18 INFO spark.HttpFileServer: HTTP File server directory is /usr/local/spark-1.6.3-bin-hadoop2.6/spark-7435e694-5b12-4efa-a784-30a4afcf68b6/httpd-3c04ef20-3378-449f-b4ff-c168
26d5928f19/06/27 20:50:18 INFO spark.HttpServer: Starting HTTP Server
19/06/27 20:50:18 INFO server.Server: jetty-8.y.z-SNAPSHOT
19/06/27 20:50:18 INFO server.AbstractConnector: Started SocketConnector@0.0.0.0:33544
19/06/27 20:50:18 INFO util.Utils: Successfully started service 'HTTP file server' on port 33544.
19/06/27 20:50:19 INFO spark.SparkContext: Added JAR file:/usr/local/spark/lib/spark-examples-1.6.3-hadoop2.6.0.jar at http://192.168.186.10:33544/jars/spark-examples-1.6.3-hadoop2.6.0.jar
with timestamp 156163981947619/06/27 20:50:19 INFO executor.Executor: Starting executor ID driver on host localhost
19/06/27 20:50:19 INFO util.Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 33262.
19/06/27 20:50:19 INFO netty.NettyBlockTransferService: Server created on 33262
19/06/27 20:50:19 INFO storage.BlockManagerMaster: Trying to register BlockManager
19/06/27 20:50:19 INFO storage.BlockManagerMasterEndpoint: Registering block manager localhost:33262 with 511.1 MB RAM, BlockManagerId(driver, localhost, 33262)
19/06/27 20:50:19 INFO storage.BlockManagerMaster: Registered BlockManager
19/06/27 20:50:20 INFO spark.SparkContext: Starting job: reduce at SparkPi.scala:36
19/06/27 20:50:20 INFO scheduler.DAGScheduler: Got job 0 (reduce at SparkPi.scala:36) with 10 output partitions
19/06/27 20:50:20 INFO scheduler.DAGScheduler: Final stage: ResultStage 0 (reduce at SparkPi.scala:36)
19/06/27 20:50:20 INFO scheduler.DAGScheduler: Parents of final stage: List()
19/06/27 20:50:20 INFO scheduler.DAGScheduler: Missing parents: List()
19/06/27 20:50:20 INFO scheduler.DAGScheduler: Submitting ResultStage 0 (MapPartitionsRDD[1] at map at SparkPi.scala:32), which has no missing parents
19/06/27 20:50:20 INFO storage.MemoryStore: Block broadcast_0 stored as values in memory (estimated size 1904.0 B, free 511.1 MB)
19/06/27 20:50:20 INFO storage.MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 1218.0 B, free 511.1 MB)
19/06/27 20:50:20 INFO storage.BlockManagerInfo: Added broadcast_0_piece0 in memory on localhost:33262 (size: 1218.0 B, free: 511.1 MB)
19/06/27 20:50:20 INFO spark.SparkContext: Created broadcast 0 from broadcast at DAGScheduler.scala:1006
19/06/27 20:50:20 INFO scheduler.DAGScheduler: Submitting 10 missing tasks from ResultStage 0 (MapPartitionsRDD[1] at map at SparkPi.scala:32)
19/06/27 20:50:20 INFO scheduler.TaskSchedulerImpl: Adding task set 0.0 with 10 tasks
19/06/27 20:50:21 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0, localhost, partition 0,PROCESS_LOCAL, 2157 bytes)
19/06/27 20:50:21 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 0.0 (TID 1, localhost, partition 1,PROCESS_LOCAL, 2157 bytes)
19/06/27 20:50:21 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 0.0 (TID 2, localhost, partition 2,PROCESS_LOCAL, 2157 bytes)
19/06/27 20:50:21 INFO scheduler.TaskSetManager: Starting task 3.0 in stage 0.0 (TID 3, localhost, partition 3,PROCESS_LOCAL, 2157 bytes)
19/06/27 20:50:21 INFO executor.Executor: Running task 2.0 in stage 0.0 (TID 2)
19/06/27 20:50:21 INFO executor.Executor: Running task 1.0 in stage 0.0 (TID 1)
19/06/27 20:50:21 INFO executor.Executor: Running task 0.0 in stage 0.0 (TID 0)
19/06/27 20:50:21 INFO executor.Executor: Running task 3.0 in stage 0.0 (TID 3)
19/06/27 20:50:21 INFO executor.Executor: Fetching http://192.168.186.10:33544/jars/spark-examples-1.6.3-hadoop2.6.0.jar with timestamp 1561639819476
19/06/27 20:50:21 INFO util.Utils: Fetching http://192.168.186.10:33544/jars/spark-examples-1.6.3-hadoop2.6.0.jar to /usr/local/spark-1.6.3-bin-hadoop2.6/spark-7435e694-5b12-4efa-a784-30a4a
fcf68b6/userFiles-7db6afc8-1a13-46e1-b169-7f61e1776bf4/fetchFileTemp632533051007102802.tmp19/06/27 20:50:21 INFO executor.Executor: Adding file:/usr/local/spark-1.6.3-bin-hadoop2.6/spark-7435e694-5b12-4efa-a784-30a4afcf68b6/userFiles-7db6afc8-1a13-46e1-b169-7f61e1776bf4/spark-ex
amples-1.6.3-hadoop2.6.0.jar to class loader19/06/27 20:50:21 INFO executor.Executor: Finished task 1.0 in stage 0.0 (TID 1). 1031 bytes result sent to driver
19/06/27 20:50:21 INFO executor.Executor: Finished task 2.0 in stage 0.0 (TID 2). 1031 bytes result sent to driver
19/06/27 20:50:21 INFO executor.Executor: Finished task 3.0 in stage 0.0 (TID 3). 1031 bytes result sent to driver
19/06/27 20:50:21 INFO executor.Executor: Finished task 0.0 in stage 0.0 (TID 0). 1031 bytes result sent to driver
19/06/27 20:50:21 INFO scheduler.TaskSetManager: Starting task 4.0 in stage 0.0 (TID 4, localhost, partition 4,PROCESS_LOCAL, 2157 bytes)
19/06/27 20:50:21 INFO executor.Executor: Running task 4.0 in stage 0.0 (TID 4)
19/06/27 20:50:21 INFO scheduler.TaskSetManager: Starting task 5.0 in stage 0.0 (TID 5, localhost, partition 5,PROCESS_LOCAL, 2157 bytes)
19/06/27 20:50:21 INFO executor.Executor: Running task 5.0 in stage 0.0 (TID 5)
19/06/27 20:50:21 INFO scheduler.TaskSetManager: Starting task 6.0 in stage 0.0 (TID 6, localhost, partition 6,PROCESS_LOCAL, 2157 bytes)
19/06/27 20:50:21 INFO executor.Executor: Running task 6.0 in stage 0.0 (TID 6)
19/06/27 20:50:21 INFO scheduler.TaskSetManager: Starting task 7.0 in stage 0.0 (TID 7, localhost, partition 7,PROCESS_LOCAL, 2157 bytes)
19/06/27 20:50:21 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 0.0 (TID 1) in 777 ms on localhost (1/10)
19/06/27 20:50:21 INFO executor.Executor: Running task 7.0 in stage 0.0 (TID 7)
19/06/27 20:50:21 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0) in 809 ms on localhost (2/10)
19/06/27 20:50:21 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 0.0 (TID 2) in 792 ms on localhost (3/10)
19/06/27 20:50:21 INFO scheduler.TaskSetManager: Finished task 3.0 in stage 0.0 (TID 3) in 812 ms on localhost (4/10)
19/06/27 20:50:21 INFO executor.Executor: Finished task 7.0 in stage 0.0 (TID 7). 1031 bytes result sent to driver
19/06/27 20:50:21 INFO scheduler.TaskSetManager: Starting task 8.0 in stage 0.0 (TID 8, localhost, partition 8,PROCESS_LOCAL, 2157 bytes)
19/06/27 20:50:21 INFO scheduler.TaskSetManager: Finished task 7.0 in stage 0.0 (TID 7) in 63 ms on localhost (5/10)
19/06/27 20:50:21 INFO executor.Executor: Finished task 6.0 in stage 0.0 (TID 6). 1031 bytes result sent to driver
19/06/27 20:50:21 INFO executor.Executor: Running task 8.0 in stage 0.0 (TID 8)
19/06/27 20:50:21 INFO scheduler.TaskSetManager: Starting task 9.0 in stage 0.0 (TID 9, localhost, partition 9,PROCESS_LOCAL, 2157 bytes)
19/06/27 20:50:21 INFO executor.Executor: Finished task 5.0 in stage 0.0 (TID 5). 1031 bytes result sent to driver
19/06/27 20:50:21 INFO scheduler.TaskSetManager: Finished task 6.0 in stage 0.0 (TID 6) in 66 ms on localhost (6/10)
19/06/27 20:50:21 INFO executor.Executor: Running task 9.0 in stage 0.0 (TID 9)
19/06/27 20:50:21 INFO executor.Executor: Finished task 4.0 in stage 0.0 (TID 4). 1031 bytes result sent to driver
19/06/27 20:50:21 INFO scheduler.TaskSetManager: Finished task 5.0 in stage 0.0 (TID 5) in 82 ms on localhost (7/10)
19/06/27 20:50:21 INFO scheduler.TaskSetManager: Finished task 4.0 in stage 0.0 (TID 4) in 92 ms on localhost (8/10)
19/06/27 20:50:21 INFO executor.Executor: Finished task 9.0 in stage 0.0 (TID 9). 1031 bytes result sent to driver
19/06/27 20:50:21 INFO scheduler.TaskSetManager: Finished task 9.0 in stage 0.0 (TID 9) in 42 ms on localhost (9/10)
19/06/27 20:50:21 INFO executor.Executor: Finished task 8.0 in stage 0.0 (TID 8). 1031 bytes result sent to driver
19/06/27 20:50:21 INFO scheduler.TaskSetManager: Finished task 8.0 in stage 0.0 (TID 8) in 62 ms on localhost (10/10)
19/06/27 20:50:21 INFO scheduler.DAGScheduler: ResultStage 0 (reduce at SparkPi.scala:36) finished in 0.947 s
19/06/27 20:50:21 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool
19/06/27 20:50:21 INFO scheduler.DAGScheduler: Job 0 finished: reduce at SparkPi.scala:36, took 1.229981 s
Pi is roughly 3.1425271425271424
19/06/27 20:50:21 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/metrics/json,null}
19/06/27 20:50:21 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/stage/kill,null}
19/06/27 20:50:21 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/api,null}
19/06/27 20:50:21 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/,null}
19/06/27 20:50:21 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/static,null}
19/06/27 20:50:21 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/executors/threadDump/json,null}
19/06/27 20:50:21 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/executors/threadDump,null}
19/06/27 20:50:21 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/executors/json,null}
19/06/27 20:50:21 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/executors,null}
19/06/27 20:50:21 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/environment/json,null}
19/06/27 20:50:21 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/environment,null}
19/06/27 20:50:21 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/storage/rdd/json,null}
19/06/27 20:50:21 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/storage/rdd,null}
19/06/27 20:50:21 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/storage/json,null}
19/06/27 20:50:21 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/storage,null}
19/06/27 20:50:21 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/pool/json,null}
19/06/27 20:50:21 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/pool,null}
19/06/27 20:50:21 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/stage/json,null}
19/06/27 20:50:21 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/stage,null}
19/06/27 20:50:21 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/json,null}
19/06/27 20:50:21 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages,null}
19/06/27 20:50:21 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/jobs/job/json,null}
19/06/27 20:50:21 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/jobs/job,null}
19/06/27 20:50:21 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/jobs/json,null}
19/06/27 20:50:21 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/jobs,null}
19/06/27 20:50:22 INFO ui.SparkUI: Stopped Spark web UI at http://192.168.186.10:4040
19/06/27 20:50:22 INFO spark.MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
19/06/27 20:50:22 INFO storage.MemoryStore: MemoryStore cleared
19/06/27 20:50:22 INFO storage.BlockManager: BlockManager stopped
19/06/27 20:50:22 INFO storage.BlockManagerMaster: BlockManagerMaster stopped
19/06/27 20:50:22 INFO scheduler.OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
19/06/27 20:50:22 INFO spark.SparkContext: Successfully stopped SparkContext
19/06/27 20:50:22 INFO remote.RemoteActorRefProvider$RemotingTerminator: Shutting down remote daemon.
19/06/27 20:50:22 INFO remote.RemoteActorRefProvider$RemotingTerminator: Remote daemon shut down; proceeding with flushing remote transports.
19/06/27 20:50:22 INFO util.ShutdownHookManager: Shutdown hook called
19/06/27 20:50:22 INFO util.ShutdownHookManager: Deleting directory /usr/local/spark-1.6.3-bin-hadoop2.6/spark-7435e694-5b12-4efa-a784-30a4afcf68b6/httpd-3c04ef20-3378-449f-b4ff-c16826d5928
f19/06/27 20:50:22 INFO util.ShutdownHookManager: Deleting directory /usr/local/spark-1.6.3-bin-hadoop2.6/spark-7435e694-5b12-4efa-a784-30a4afcf68b6
[root@master spark]#
2.6.2 集群模式 Spark Standalone¶
./bin/spark-submit --class org.apache.spark.examples.SparkPi --master spark://master:7077 lib/spark-examples-1.6.3-hadoop2.6.0.jar 100
详细步骤
[root@master spark]# ./bin/spark-submit --class org.apache.spark.examples.SparkPi --master spark://master:7077 lib/spark-examples-1.6.3-hadoop2.6.0.jar 100
19/06/27 20:53:43 INFO spark.SparkContext: Running Spark version 1.6.3
19/06/27 20:53:44 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
19/06/27 20:53:44 INFO spark.SecurityManager: Changing view acls to: root
19/06/27 20:53:44 INFO spark.SecurityManager: Changing modify acls to: root
19/06/27 20:53:44 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root); users with modify permissions: Set(root)
19/06/27 20:53:44 INFO util.Utils: Successfully started service 'sparkDriver' on port 41697.
19/06/27 20:53:44 INFO slf4j.Slf4jLogger: Slf4jLogger started
19/06/27 20:53:44 INFO Remoting: Starting remoting
19/06/27 20:53:44 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriverActorSystem@192.168.186.10:35129]
19/06/27 20:53:44 INFO util.Utils: Successfully started service 'sparkDriverActorSystem' on port 35129.
19/06/27 20:53:44 INFO spark.SparkEnv: Registering MapOutputTracker
19/06/27 20:53:44 INFO spark.SparkEnv: Registering BlockManagerMaster
19/06/27 20:53:44 INFO storage.DiskBlockManager: Created local directory at /usr/local/spark-1.6.3-bin-hadoop2.6/blockmgr-efde1a8e-585a-46b3-9fa9-2d872944d9c4
19/06/27 20:53:44 INFO storage.MemoryStore: MemoryStore started with capacity 511.1 MB
19/06/27 20:53:45 INFO spark.SparkEnv: Registering OutputCommitCoordinator
19/06/27 20:53:45 INFO server.Server: jetty-8.y.z-SNAPSHOT
19/06/27 20:53:45 INFO server.AbstractConnector: Started SelectChannelConnector@0.0.0.0:4040
19/06/27 20:53:45 INFO util.Utils: Successfully started service 'SparkUI' on port 4040.
19/06/27 20:53:45 INFO ui.SparkUI: Started SparkUI at http://192.168.186.10:4040
19/06/27 20:53:45 INFO spark.HttpFileServer: HTTP File server directory is /usr/local/spark-1.6.3-bin-hadoop2.6/spark-23f8e88f-c8d7-495b-ad4d-3a46e1af260f/httpd-f0176344-5b7c-44af-9bf9-a206
db63add519/06/27 20:53:45 INFO spark.HttpServer: Starting HTTP Server
19/06/27 20:53:45 INFO server.Server: jetty-8.y.z-SNAPSHOT
19/06/27 20:53:45 INFO server.AbstractConnector: Started SocketConnector@0.0.0.0:42961
19/06/27 20:53:45 INFO util.Utils: Successfully started service 'HTTP file server' on port 42961.
19/06/27 20:53:45 INFO spark.SparkContext: Added JAR file:/usr/local/spark-1.6.3-bin-hadoop2.6/lib/spark-examples-1.6.3-hadoop2.6.0.jar at http://192.168.186.10:42961/jars/spark-examples-1.
6.3-hadoop2.6.0.jar with timestamp 156164002584419/06/27 20:53:45 INFO client.AppClient$ClientEndpoint: Connecting to master spark://master:7077...
19/06/27 20:53:46 INFO cluster.SparkDeploySchedulerBackend: Connected to Spark cluster with app ID app-20190627205346-0000
19/06/27 20:53:46 INFO util.Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 43863.
19/06/27 20:53:46 INFO netty.NettyBlockTransferService: Server created on 43863
19/06/27 20:53:46 INFO client.AppClient$ClientEndpoint: Executor added: app-20190627205346-0000/0 on worker-20190627084145-192.168.186.12-35942 (192.168.186.12:35942) with 4 cores
19/06/27 20:53:46 INFO storage.BlockManagerMaster: Trying to register BlockManager
19/06/27 20:53:46 INFO cluster.SparkDeploySchedulerBackend: Granted executor ID app-20190627205346-0000/0 on hostPort 192.168.186.12:35942 with 4 cores, 1024.0 MB RAM
19/06/27 20:53:46 INFO client.AppClient$ClientEndpoint: Executor added: app-20190627205346-0000/1 on worker-20190627084145-192.168.186.11-37378 (192.168.186.11:37378) with 4 cores
19/06/27 20:53:46 INFO cluster.SparkDeploySchedulerBackend: Granted executor ID app-20190627205346-0000/1 on hostPort 192.168.186.11:37378 with 4 cores, 1024.0 MB RAM
19/06/27 20:53:46 INFO storage.BlockManagerMasterEndpoint: Registering block manager 192.168.186.10:43863 with 511.1 MB RAM, BlockManagerId(driver, 192.168.186.10, 43863)
19/06/27 20:53:46 INFO storage.BlockManagerMaster: Registered BlockManager
19/06/27 20:53:47 INFO client.AppClient$ClientEndpoint: Executor updated: app-20190627205346-0000/0 is now RUNNING
19/06/27 20:53:47 INFO client.AppClient$ClientEndpoint: Executor updated: app-20190627205346-0000/1 is now RUNNING
19/06/27 20:53:47 INFO cluster.SparkDeploySchedulerBackend: SchedulerBackend is ready for scheduling beginning after reached minRegisteredResourcesRatio: 0.0
19/06/27 20:53:48 INFO spark.SparkContext: Starting job: reduce at SparkPi.scala:36
19/06/27 20:53:48 INFO scheduler.DAGScheduler: Got job 0 (reduce at SparkPi.scala:36) with 100 output partitions
19/06/27 20:53:48 INFO scheduler.DAGScheduler: Final stage: ResultStage 0 (reduce at SparkPi.scala:36)
19/06/27 20:53:48 INFO scheduler.DAGScheduler: Parents of final stage: List()
19/06/27 20:53:48 INFO scheduler.DAGScheduler: Missing parents: List()
19/06/27 20:53:48 INFO scheduler.DAGScheduler: Submitting ResultStage 0 (MapPartitionsRDD[1] at map at SparkPi.scala:32), which has no missing parents
19/06/27 20:53:48 INFO storage.MemoryStore: Block broadcast_0 stored as values in memory (estimated size 1904.0 B, free 511.1 MB)
19/06/27 20:53:48 INFO storage.MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 1218.0 B, free 511.1 MB)
19/06/27 20:53:48 INFO storage.BlockManagerInfo: Added broadcast_0_piece0 in memory on 192.168.186.10:43863 (size: 1218.0 B, free: 511.1 MB)
19/06/27 20:53:48 INFO spark.SparkContext: Created broadcast 0 from broadcast at DAGScheduler.scala:1006
19/06/27 20:53:48 INFO scheduler.DAGScheduler: Submitting 100 missing tasks from ResultStage 0 (MapPartitionsRDD[1] at map at SparkPi.scala:32)
19/06/27 20:53:48 INFO scheduler.TaskSchedulerImpl: Adding task set 0.0 with 100 tasks
19/06/27 20:53:51 INFO cluster.SparkDeploySchedulerBackend: Registered executor NettyRpcEndpointRef(null) (slave1:58978) with ID 1
19/06/27 20:53:51 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0, slave1, partition 0,PROCESS_LOCAL, 2157 bytes)
19/06/27 20:53:51 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 0.0 (TID 1, slave1, partition 1,PROCESS_LOCAL, 2157 bytes)
19/06/27 20:53:51 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 0.0 (TID 2, slave1, partition 2,PROCESS_LOCAL, 2157 bytes)
19/06/27 20:53:51 INFO scheduler.TaskSetManager: Starting task 3.0 in stage 0.0 (TID 3, slave1, partition 3,PROCESS_LOCAL, 2157 bytes)
19/06/27 20:53:51 INFO storage.BlockManagerMasterEndpoint: Registering block manager slave1:45556 with 511.1 MB RAM, BlockManagerId(1, slave1, 45556)
19/06/27 20:53:53 INFO cluster.SparkDeploySchedulerBackend: Registered executor NettyRpcEndpointRef(null) (slave2:50242) with ID 0
19/06/27 20:53:53 INFO scheduler.TaskSetManager: Starting task 4.0 in stage 0.0 (TID 4, slave2, partition 4,PROCESS_LOCAL, 2157 bytes)
19/06/27 20:53:53 INFO scheduler.TaskSetManager: Starting task 5.0 in stage 0.0 (TID 5, slave2, partition 5,PROCESS_LOCAL, 2157 bytes)
19/06/27 20:53:53 INFO scheduler.TaskSetManager: Starting task 6.0 in stage 0.0 (TID 6, slave2, partition 6,PROCESS_LOCAL, 2157 bytes)
19/06/27 20:53:53 INFO scheduler.TaskSetManager: Starting task 7.0 in stage 0.0 (TID 7, slave2, partition 7,PROCESS_LOCAL, 2157 bytes)
19/06/27 20:53:54 INFO storage.BlockManagerMasterEndpoint: Registering block manager slave2:36781 with 511.1 MB RAM, BlockManagerId(0, slave2, 36781)
19/06/27 20:53:57 INFO storage.BlockManagerInfo: Added broadcast_0_piece0 in memory on slave1:45556 (size: 1218.0 B, free: 511.1 MB)
19/06/27 20:53:57 INFO scheduler.TaskSetManager: Starting task 8.0 in stage 0.0 (TID 8, slave1, partition 8,PROCESS_LOCAL, 2157 bytes)
19/06/27 20:53:57 INFO scheduler.TaskSetManager: Starting task 9.0 in stage 0.0 (TID 9, slave1, partition 9,PROCESS_LOCAL, 2157 bytes)
19/06/27 20:53:57 INFO scheduler.TaskSetManager: Starting task 10.0 in stage 0.0 (TID 10, slave1, partition 10,PROCESS_LOCAL, 2157 bytes)
19/06/27 20:53:57 INFO scheduler.TaskSetManager: Starting task 11.0 in stage 0.0 (TID 11, slave1, partition 11,PROCESS_LOCAL, 2157 bytes)
19/06/27 20:53:57 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 0.0 (TID 2) in 5974 ms on slave1 (1/100)
19/06/27 20:53:57 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 0.0 (TID 1) in 5980 ms on slave1 (2/100)
19/06/27 20:53:57 INFO scheduler.TaskSetManager: Finished task 3.0 in stage 0.0 (TID 3) in 5979 ms on slave1 (3/100)
19/06/27 20:53:57 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0) in 6098 ms on slave1 (4/100)
19/06/27 20:53:57 INFO scheduler.TaskSetManager: Starting task 12.0 in stage 0.0 (TID 12, slave1, partition 12,PROCESS_LOCAL, 2157 bytes)
19/06/27 20:53:57 INFO scheduler.TaskSetManager: Finished task 9.0 in stage 0.0 (TID 9) in 110 ms on slave1 (5/100)
19/06/27 20:53:57 INFO scheduler.TaskSetManager: Starting task 13.0 in stage 0.0 (TID 13, slave1, partition 13,PROCESS_LOCAL, 2157 bytes)
19/06/27 20:53:57 INFO scheduler.TaskSetManager: Finished task 8.0 in stage 0.0 (TID 8) in 147 ms on slave1 (6/100)
19/06/27 20:53:57 INFO scheduler.TaskSetManager: Starting task 14.0 in stage 0.0 (TID 14, slave1, partition 14,PROCESS_LOCAL, 2157 bytes)
19/06/27 20:53:57 INFO scheduler.TaskSetManager: Starting task 15.0 in stage 0.0 (TID 15, slave1, partition 15,PROCESS_LOCAL, 2157 bytes)
19/06/27 20:53:57 INFO scheduler.TaskSetManager: Starting task 16.0 in stage 0.0 (TID 16, slave1, partition 16,PROCESS_LOCAL, 2157 bytes)
19/06/27 20:53:57 INFO scheduler.TaskSetManager: Finished task 10.0 in stage 0.0 (TID 10) in 192 ms on slave1 (7/100)
19/06/27 20:53:57 INFO scheduler.TaskSetManager: Finished task 12.0 in stage 0.0 (TID 12) in 93 ms on slave1 (8/100)
19/06/27 20:53:57 INFO scheduler.TaskSetManager: Finished task 11.0 in stage 0.0 (TID 11) in 186 ms on slave1 (9/100)
19/06/27 20:53:57 INFO storage.BlockManagerInfo: Added broadcast_0_piece0 in memory on slave2:36781 (size: 1218.0 B, free: 511.1 MB)
19/06/27 20:53:57 INFO scheduler.TaskSetManager: Starting task 17.0 in stage 0.0 (TID 17, slave1, partition 17,PROCESS_LOCAL, 2157 bytes)
19/06/27 20:53:57 INFO scheduler.TaskSetManager: Finished task 15.0 in stage 0.0 (TID 15) in 111 ms on slave1 (10/100)
19/06/27 20:53:58 INFO scheduler.TaskSetManager: Starting task 18.0 in stage 0.0 (TID 18, slave1, partition 18,PROCESS_LOCAL, 2157 bytes)
19/06/27 20:53:58 INFO scheduler.TaskSetManager: Finished task 13.0 in stage 0.0 (TID 13) in 202 ms on slave1 (11/100)
19/06/27 20:53:58 INFO scheduler.TaskSetManager: Starting task 19.0 in stage 0.0 (TID 19, slave1, partition 19,PROCESS_LOCAL, 2157 bytes)
19/06/27 20:53:58 INFO scheduler.TaskSetManager: Finished task 16.0 in stage 0.0 (TID 16) in 167 ms on slave1 (12/100)
19/06/27 20:53:58 INFO scheduler.TaskSetManager: Starting task 20.0 in stage 0.0 (TID 20, slave1, partition 20,PROCESS_LOCAL, 2157 bytes)
19/06/27 20:53:58 INFO scheduler.TaskSetManager: Finished task 14.0 in stage 0.0 (TID 14) in 180 ms on slave1 (13/100)
19/06/27 20:53:58 INFO scheduler.TaskSetManager: Starting task 21.0 in stage 0.0 (TID 21, slave1, partition 21,PROCESS_LOCAL, 2157 bytes)
19/06/27 20:53:58 INFO scheduler.TaskSetManager: Finished task 17.0 in stage 0.0 (TID 17) in 103 ms on slave1 (14/100)
19/06/27 20:53:58 INFO scheduler.TaskSetManager: Starting task 22.0 in stage 0.0 (TID 22, slave1, partition 22,PROCESS_LOCAL, 2157 bytes)
19/06/27 20:53:58 INFO scheduler.TaskSetManager: Finished task 18.0 in stage 0.0 (TID 18) in 579 ms on slave1 (15/100)
19/06/27 20:53:58 INFO scheduler.TaskSetManager: Starting task 23.0 in stage 0.0 (TID 23, slave1, partition 23,PROCESS_LOCAL, 2157 bytes)
19/06/27 20:53:58 INFO scheduler.TaskSetManager: Finished task 19.0 in stage 0.0 (TID 19) in 595 ms on slave1 (16/100)
19/06/27 20:53:58 INFO scheduler.TaskSetManager: Starting task 24.0 in stage 0.0 (TID 24, slave2, partition 24,PROCESS_LOCAL, 2157 bytes)
19/06/27 20:53:58 INFO scheduler.TaskSetManager: Finished task 5.0 in stage 0.0 (TID 5) in 4729 ms on slave2 (17/100)
19/06/27 20:53:58 INFO scheduler.TaskSetManager: Starting task 25.0 in stage 0.0 (TID 25, slave1, partition 25,PROCESS_LOCAL, 2157 bytes)
19/06/27 20:53:58 INFO scheduler.TaskSetManager: Starting task 26.0 in stage 0.0 (TID 26, slave1, partition 26,PROCESS_LOCAL, 2157 bytes)
19/06/27 20:53:58 INFO scheduler.TaskSetManager: Finished task 22.0 in stage 0.0 (TID 22) in 81 ms on slave1 (18/100)
19/06/27 20:53:58 INFO scheduler.TaskSetManager: Finished task 20.0 in stage 0.0 (TID 20) in 639 ms on slave1 (19/100)
19/06/27 20:53:58 INFO scheduler.TaskSetManager: Starting task 27.0 in stage 0.0 (TID 27, slave1, partition 27,PROCESS_LOCAL, 2157 bytes)
19/06/27 20:53:58 INFO scheduler.TaskSetManager: Finished task 21.0 in stage 0.0 (TID 21) in 611 ms on slave1 (20/100)
19/06/27 20:53:58 INFO scheduler.TaskSetManager: Starting task 28.0 in stage 0.0 (TID 28, slave2, partition 28,PROCESS_LOCAL, 2157 bytes)
19/06/27 20:53:58 INFO scheduler.TaskSetManager: Finished task 4.0 in stage 0.0 (TID 4) in 4780 ms on slave2 (21/100)
19/06/27 20:53:58 INFO scheduler.TaskSetManager: Starting task 29.0 in stage 0.0 (TID 29, slave2, partition 29,PROCESS_LOCAL, 2157 bytes)
19/06/27 20:53:58 INFO scheduler.TaskSetManager: Finished task 6.0 in stage 0.0 (TID 6) in 4783 ms on slave2 (22/100)
19/06/27 20:53:58 INFO scheduler.TaskSetManager: Starting task 30.0 in stage 0.0 (TID 30, slave2, partition 30,PROCESS_LOCAL, 2157 bytes)
19/06/27 20:53:58 INFO scheduler.TaskSetManager: Finished task 7.0 in stage 0.0 (TID 7) in 4786 ms on slave2 (23/100)
19/06/27 20:53:58 INFO scheduler.TaskSetManager: Starting task 31.0 in stage 0.0 (TID 31, slave1, partition 31,PROCESS_LOCAL, 2157 bytes)
19/06/27 20:53:58 INFO scheduler.TaskSetManager: Finished task 23.0 in stage 0.0 (TID 23) in 105 ms on slave1 (24/100)
19/06/27 20:53:58 INFO scheduler.TaskSetManager: Starting task 32.0 in stage 0.0 (TID 32, slave2, partition 32,PROCESS_LOCAL, 2157 bytes)
19/06/27 20:53:58 INFO scheduler.TaskSetManager: Starting task 33.0 in stage 0.0 (TID 33, slave1, partition 33,PROCESS_LOCAL, 2157 bytes)
19/06/27 20:53:58 INFO scheduler.TaskSetManager: Starting task 34.0 in stage 0.0 (TID 34, slave1, partition 34,PROCESS_LOCAL, 2157 bytes)
19/06/27 20:53:58 INFO scheduler.TaskSetManager: Finished task 24.0 in stage 0.0 (TID 24) in 157 ms on slave2 (25/100)
19/06/27 20:53:58 INFO scheduler.TaskSetManager: Finished task 26.0 in stage 0.0 (TID 26) in 129 ms on slave1 (26/100)
19/06/27 20:53:58 INFO scheduler.TaskSetManager: Finished task 25.0 in stage 0.0 (TID 25) in 142 ms on slave1 (27/100)
19/06/27 20:53:58 INFO scheduler.TaskSetManager: Starting task 35.0 in stage 0.0 (TID 35, slave1, partition 35,PROCESS_LOCAL, 2157 bytes)
19/06/27 20:53:58 INFO scheduler.TaskSetManager: Finished task 27.0 in stage 0.0 (TID 27) in 164 ms on slave1 (28/100)
19/06/27 20:53:58 INFO scheduler.TaskSetManager: Starting task 36.0 in stage 0.0 (TID 36, slave1, partition 36,PROCESS_LOCAL, 2157 bytes)
19/06/27 20:53:58 INFO scheduler.TaskSetManager: Starting task 37.0 in stage 0.0 (TID 37, slave2, partition 37,PROCESS_LOCAL, 2157 bytes)
19/06/27 20:53:58 INFO scheduler.TaskSetManager: Finished task 31.0 in stage 0.0 (TID 31) in 128 ms on slave1 (29/100)
19/06/27 20:53:58 INFO scheduler.TaskSetManager: Finished task 30.0 in stage 0.0 (TID 30) in 158 ms on slave2 (30/100)
19/06/27 20:53:58 INFO scheduler.TaskSetManager: Starting task 38.0 in stage 0.0 (TID 38, slave2, partition 38,PROCESS_LOCAL, 2157 bytes)
19/06/27 20:53:58 INFO scheduler.TaskSetManager: Finished task 28.0 in stage 0.0 (TID 28) in 184 ms on slave2 (31/100)
19/06/27 20:53:58 INFO scheduler.TaskSetManager: Starting task 39.0 in stage 0.0 (TID 39, slave1, partition 39,PROCESS_LOCAL, 2157 bytes)
19/06/27 20:53:58 INFO scheduler.TaskSetManager: Finished task 33.0 in stage 0.0 (TID 33) in 84 ms on slave1 (32/100)
19/06/27 20:53:58 INFO scheduler.TaskSetManager: Starting task 40.0 in stage 0.0 (TID 40, slave2, partition 40,PROCESS_LOCAL, 2157 bytes)
19/06/27 20:53:58 INFO scheduler.TaskSetManager: Finished task 29.0 in stage 0.0 (TID 29) in 229 ms on slave2 (33/100)
19/06/27 20:53:58 INFO scheduler.TaskSetManager: Starting task 41.0 in stage 0.0 (TID 41, slave1, partition 41,PROCESS_LOCAL, 2157 bytes)
19/06/27 20:53:58 INFO scheduler.TaskSetManager: Starting task 42.0 in stage 0.0 (TID 42, slave1, partition 42,PROCESS_LOCAL, 2157 bytes)
19/06/27 20:53:58 INFO scheduler.TaskSetManager: Finished task 34.0 in stage 0.0 (TID 34) in 149 ms on slave1 (34/100)
19/06/27 20:53:58 INFO scheduler.TaskSetManager: Finished task 35.0 in stage 0.0 (TID 35) in 117 ms on slave1 (35/100)
19/06/27 20:53:59 INFO scheduler.TaskSetManager: Starting task 43.0 in stage 0.0 (TID 43, slave2, partition 43,PROCESS_LOCAL, 2157 bytes)
19/06/27 20:53:59 INFO scheduler.TaskSetManager: Finished task 38.0 in stage 0.0 (TID 38) in 144 ms on slave2 (36/100)
19/06/27 20:53:59 INFO scheduler.TaskSetManager: Starting task 44.0 in stage 0.0 (TID 44, slave2, partition 44,PROCESS_LOCAL, 2157 bytes)
19/06/27 20:53:59 INFO scheduler.TaskSetManager: Starting task 45.0 in stage 0.0 (TID 45, slave2, partition 45,PROCESS_LOCAL, 2157 bytes)
19/06/27 20:53:59 INFO scheduler.TaskSetManager: Finished task 37.0 in stage 0.0 (TID 37) in 174 ms on slave2 (37/100)
19/06/27 20:53:59 INFO scheduler.TaskSetManager: Finished task 32.0 in stage 0.0 (TID 32) in 241 ms on slave2 (38/100)
19/06/27 20:53:59 INFO scheduler.TaskSetManager: Starting task 46.0 in stage 0.0 (TID 46, slave2, partition 46,PROCESS_LOCAL, 2157 bytes)
19/06/27 20:53:59 INFO scheduler.TaskSetManager: Finished task 40.0 in stage 0.0 (TID 40) in 178 ms on slave2 (39/100)
19/06/27 20:53:59 INFO scheduler.TaskSetManager: Starting task 47.0 in stage 0.0 (TID 47, slave2, partition 47,PROCESS_LOCAL, 2157 bytes)
19/06/27 20:53:59 INFO scheduler.TaskSetManager: Finished task 43.0 in stage 0.0 (TID 43) in 237 ms on slave2 (40/100)
19/06/27 20:53:59 INFO scheduler.TaskSetManager: Finished task 44.0 in stage 0.0 (TID 44) in 243 ms on slave2 (41/100)
19/06/27 20:53:59 INFO scheduler.TaskSetManager: Starting task 48.0 in stage 0.0 (TID 48, slave2, partition 48,PROCESS_LOCAL, 2157 bytes)
19/06/27 20:53:59 INFO scheduler.TaskSetManager: Starting task 49.0 in stage 0.0 (TID 49, slave2, partition 49,PROCESS_LOCAL, 2157 bytes)
19/06/27 20:53:59 INFO scheduler.TaskSetManager: Finished task 45.0 in stage 0.0 (TID 45) in 259 ms on slave2 (42/100)
19/06/27 20:53:59 INFO scheduler.TaskSetManager: Starting task 50.0 in stage 0.0 (TID 50, slave2, partition 50,PROCESS_LOCAL, 2157 bytes)
19/06/27 20:53:59 INFO scheduler.TaskSetManager: Finished task 46.0 in stage 0.0 (TID 46) in 230 ms on slave2 (43/100)
19/06/27 20:53:59 INFO scheduler.TaskSetManager: Starting task 51.0 in stage 0.0 (TID 51, slave1, partition 51,PROCESS_LOCAL, 2157 bytes)
19/06/27 20:53:59 INFO scheduler.TaskSetManager: Finished task 39.0 in stage 0.0 (TID 39) in 508 ms on slave1 (44/100)
19/06/27 20:53:59 INFO scheduler.TaskSetManager: Starting task 52.0 in stage 0.0 (TID 52, slave1, partition 52,PROCESS_LOCAL, 2157 bytes)
19/06/27 20:53:59 INFO scheduler.TaskSetManager: Finished task 36.0 in stage 0.0 (TID 36) in 539 ms on slave1 (45/100)
19/06/27 20:53:59 INFO scheduler.TaskSetManager: Starting task 53.0 in stage 0.0 (TID 53, slave2, partition 53,PROCESS_LOCAL, 2157 bytes)
19/06/27 20:53:59 INFO scheduler.TaskSetManager: Finished task 47.0 in stage 0.0 (TID 47) in 156 ms on slave2 (46/100)
19/06/27 20:53:59 INFO scheduler.TaskSetManager: Starting task 54.0 in stage 0.0 (TID 54, slave2, partition 54,PROCESS_LOCAL, 2157 bytes)
19/06/27 20:53:59 INFO scheduler.TaskSetManager: Finished task 49.0 in stage 0.0 (TID 49) in 140 ms on slave2 (47/100)
19/06/27 20:53:59 INFO scheduler.TaskSetManager: Starting task 55.0 in stage 0.0 (TID 55, slave2, partition 55,PROCESS_LOCAL, 2157 bytes)
19/06/27 20:53:59 INFO scheduler.TaskSetManager: Finished task 50.0 in stage 0.0 (TID 50) in 141 ms on slave2 (48/100)
19/06/27 20:53:59 INFO scheduler.TaskSetManager: Starting task 56.0 in stage 0.0 (TID 56, slave2, partition 56,PROCESS_LOCAL, 2157 bytes)
19/06/27 20:53:59 INFO scheduler.TaskSetManager: Finished task 48.0 in stage 0.0 (TID 48) in 182 ms on slave2 (49/100)
19/06/27 20:53:59 INFO scheduler.TaskSetManager: Starting task 57.0 in stage 0.0 (TID 57, slave2, partition 57,PROCESS_LOCAL, 2157 bytes)
19/06/27 20:53:59 INFO scheduler.TaskSetManager: Finished task 53.0 in stage 0.0 (TID 53) in 101 ms on slave2 (50/100)
19/06/27 20:53:59 INFO scheduler.TaskSetManager: Starting task 58.0 in stage 0.0 (TID 58, slave1, partition 58,PROCESS_LOCAL, 2157 bytes)
19/06/27 20:53:59 INFO scheduler.TaskSetManager: Finished task 52.0 in stage 0.0 (TID 52) in 129 ms on slave1 (51/100)
19/06/27 20:53:59 INFO scheduler.TaskSetManager: Starting task 59.0 in stage 0.0 (TID 59, slave1, partition 59,PROCESS_LOCAL, 2157 bytes)
19/06/27 20:53:59 INFO scheduler.TaskSetManager: Finished task 41.0 in stage 0.0 (TID 41) in 586 ms on slave1 (52/100)
19/06/27 20:53:59 INFO scheduler.TaskSetManager: Starting task 60.0 in stage 0.0 (TID 60, slave1, partition 60,PROCESS_LOCAL, 2157 bytes)
19/06/27 20:53:59 INFO scheduler.TaskSetManager: Starting task 61.0 in stage 0.0 (TID 61, slave1, partition 61,PROCESS_LOCAL, 2157 bytes)
19/06/27 20:53:59 INFO scheduler.TaskSetManager: Finished task 42.0 in stage 0.0 (TID 42) in 598 ms on slave1 (53/100)
19/06/27 20:53:59 INFO scheduler.TaskSetManager: Finished task 51.0 in stage 0.0 (TID 51) in 160 ms on slave1 (54/100)
19/06/27 20:53:59 INFO scheduler.TaskSetManager: Starting task 62.0 in stage 0.0 (TID 62, slave2, partition 62,PROCESS_LOCAL, 2157 bytes)
19/06/27 20:53:59 INFO scheduler.TaskSetManager: Finished task 54.0 in stage 0.0 (TID 54) in 172 ms on slave2 (55/100)
19/06/27 20:53:59 INFO scheduler.TaskSetManager: Starting task 63.0 in stage 0.0 (TID 63, slave2, partition 63,PROCESS_LOCAL, 2157 bytes)
19/06/27 20:53:59 INFO scheduler.TaskSetManager: Finished task 55.0 in stage 0.0 (TID 55) in 189 ms on slave2 (56/100)
19/06/27 20:53:59 INFO scheduler.TaskSetManager: Starting task 64.0 in stage 0.0 (TID 64, slave1, partition 64,PROCESS_LOCAL, 2157 bytes)
19/06/27 20:53:59 INFO scheduler.TaskSetManager: Finished task 60.0 in stage 0.0 (TID 60) in 123 ms on slave1 (57/100)
19/06/27 20:53:59 INFO scheduler.TaskSetManager: Starting task 65.0 in stage 0.0 (TID 65, slave1, partition 65,PROCESS_LOCAL, 2157 bytes)
19/06/27 20:53:59 INFO scheduler.TaskSetManager: Starting task 66.0 in stage 0.0 (TID 66, slave1, partition 66,PROCESS_LOCAL, 2157 bytes)
19/06/27 20:53:59 INFO scheduler.TaskSetManager: Starting task 67.0 in stage 0.0 (TID 67, slave1, partition 67,PROCESS_LOCAL, 2157 bytes)
19/06/27 20:53:59 INFO scheduler.TaskSetManager: Finished task 58.0 in stage 0.0 (TID 58) in 185 ms on slave1 (58/100)
19/06/27 20:53:59 INFO scheduler.TaskSetManager: Finished task 59.0 in stage 0.0 (TID 59) in 177 ms on slave1 (59/100)
19/06/27 20:53:59 INFO scheduler.TaskSetManager: Finished task 61.0 in stage 0.0 (TID 61) in 191 ms on slave1 (60/100)
19/06/27 20:53:59 INFO scheduler.TaskSetManager: Starting task 68.0 in stage 0.0 (TID 68, slave2, partition 68,PROCESS_LOCAL, 2157 bytes)
19/06/27 20:53:59 INFO scheduler.TaskSetManager: Starting task 69.0 in stage 0.0 (TID 69, slave2, partition 69,PROCESS_LOCAL, 2157 bytes)
19/06/27 20:53:59 INFO scheduler.TaskSetManager: Finished task 56.0 in stage 0.0 (TID 56) in 318 ms on slave2 (61/100)
19/06/27 20:53:59 INFO scheduler.TaskSetManager: Finished task 62.0 in stage 0.0 (TID 62) in 194 ms on slave2 (62/100)
19/06/27 20:53:59 INFO scheduler.TaskSetManager: Starting task 70.0 in stage 0.0 (TID 70, slave2, partition 70,PROCESS_LOCAL, 2157 bytes)
19/06/27 20:53:59 INFO scheduler.TaskSetManager: Finished task 57.0 in stage 0.0 (TID 57) in 306 ms on slave2 (63/100)
19/06/27 20:53:59 INFO scheduler.TaskSetManager: Starting task 71.0 in stage 0.0 (TID 71, slave1, partition 71,PROCESS_LOCAL, 2157 bytes)
19/06/27 20:53:59 INFO scheduler.TaskSetManager: Finished task 64.0 in stage 0.0 (TID 64) in 156 ms on slave1 (64/100)
19/06/27 20:53:59 INFO scheduler.TaskSetManager: Starting task 72.0 in stage 0.0 (TID 72, slave1, partition 72,PROCESS_LOCAL, 2157 bytes)
19/06/27 20:53:59 INFO scheduler.TaskSetManager: Finished task 66.0 in stage 0.0 (TID 66) in 159 ms on slave1 (65/100)
19/06/27 20:53:59 INFO scheduler.TaskSetManager: Starting task 73.0 in stage 0.0 (TID 73, slave1, partition 73,PROCESS_LOCAL, 2157 bytes)
19/06/27 20:53:59 INFO scheduler.TaskSetManager: Starting task 74.0 in stage 0.0 (TID 74, slave1, partition 74,PROCESS_LOCAL, 2157 bytes)
19/06/27 20:53:59 INFO scheduler.TaskSetManager: Starting task 75.0 in stage 0.0 (TID 75, slave2, partition 75,PROCESS_LOCAL, 2157 bytes)
19/06/27 20:53:59 INFO scheduler.TaskSetManager: Finished task 67.0 in stage 0.0 (TID 67) in 217 ms on slave1 (66/100)
19/06/27 20:53:59 INFO scheduler.TaskSetManager: Finished task 65.0 in stage 0.0 (TID 65) in 240 ms on slave1 (67/100)
19/06/27 20:53:59 INFO scheduler.TaskSetManager: Finished task 63.0 in stage 0.0 (TID 63) in 279 ms on slave2 (68/100)
19/06/27 20:53:59 INFO scheduler.TaskSetManager: Starting task 76.0 in stage 0.0 (TID 76, slave1, partition 76,PROCESS_LOCAL, 2157 bytes)
19/06/27 20:53:59 INFO scheduler.TaskSetManager: Finished task 71.0 in stage 0.0 (TID 71) in 111 ms on slave1 (69/100)
19/06/27 20:53:59 INFO scheduler.TaskSetManager: Starting task 77.0 in stage 0.0 (TID 77, slave1, partition 77,PROCESS_LOCAL, 2157 bytes)
19/06/27 20:53:59 INFO scheduler.TaskSetManager: Finished task 72.0 in stage 0.0 (TID 72) in 127 ms on slave1 (70/100)
19/06/27 20:54:00 INFO scheduler.TaskSetManager: Starting task 78.0 in stage 0.0 (TID 78, slave1, partition 78,PROCESS_LOCAL, 2157 bytes)
19/06/27 20:54:00 INFO scheduler.TaskSetManager: Finished task 74.0 in stage 0.0 (TID 74) in 148 ms on slave1 (71/100)
19/06/27 20:54:00 INFO scheduler.TaskSetManager: Starting task 79.0 in stage 0.0 (TID 79, slave1, partition 79,PROCESS_LOCAL, 2157 bytes)
19/06/27 20:54:00 INFO scheduler.TaskSetManager: Finished task 77.0 in stage 0.0 (TID 77) in 108 ms on slave1 (72/100)
19/06/27 20:54:00 INFO scheduler.TaskSetManager: Starting task 80.0 in stage 0.0 (TID 80, slave1, partition 80,PROCESS_LOCAL, 2157 bytes)
19/06/27 20:54:00 INFO scheduler.TaskSetManager: Finished task 73.0 in stage 0.0 (TID 73) in 235 ms on slave1 (73/100)
19/06/27 20:54:00 INFO scheduler.TaskSetManager: Starting task 81.0 in stage 0.0 (TID 81, slave1, partition 81,PROCESS_LOCAL, 2157 bytes)
19/06/27 20:54:00 INFO scheduler.TaskSetManager: Finished task 78.0 in stage 0.0 (TID 78) in 129 ms on slave1 (74/100)
19/06/27 20:54:00 INFO scheduler.TaskSetManager: Starting task 82.0 in stage 0.0 (TID 82, slave1, partition 82,PROCESS_LOCAL, 2157 bytes)
19/06/27 20:54:00 INFO scheduler.TaskSetManager: Finished task 76.0 in stage 0.0 (TID 76) in 307 ms on slave1 (75/100)
19/06/27 20:54:00 INFO scheduler.TaskSetManager: Starting task 83.0 in stage 0.0 (TID 83, slave1, partition 83,PROCESS_LOCAL, 2157 bytes)
19/06/27 20:54:00 INFO scheduler.TaskSetManager: Finished task 79.0 in stage 0.0 (TID 79) in 204 ms on slave1 (76/100)
19/06/27 20:54:00 INFO scheduler.TaskSetManager: Starting task 84.0 in stage 0.0 (TID 84, slave1, partition 84,PROCESS_LOCAL, 2157 bytes)
19/06/27 20:54:00 INFO scheduler.TaskSetManager: Finished task 81.0 in stage 0.0 (TID 81) in 106 ms on slave1 (77/100)
19/06/27 20:54:00 INFO scheduler.TaskSetManager: Starting task 85.0 in stage 0.0 (TID 85, slave1, partition 85,PROCESS_LOCAL, 2157 bytes)
19/06/27 20:54:00 INFO scheduler.TaskSetManager: Finished task 80.0 in stage 0.0 (TID 80) in 217 ms on slave1 (78/100)
19/06/27 20:54:00 INFO scheduler.TaskSetManager: Starting task 86.0 in stage 0.0 (TID 86, slave1, partition 86,PROCESS_LOCAL, 2157 bytes)
19/06/27 20:54:00 INFO scheduler.TaskSetManager: Finished task 83.0 in stage 0.0 (TID 83) in 120 ms on slave1 (79/100)
19/06/27 20:54:00 INFO scheduler.TaskSetManager: Starting task 87.0 in stage 0.0 (TID 87, slave1, partition 87,PROCESS_LOCAL, 2157 bytes)
19/06/27 20:54:00 INFO scheduler.TaskSetManager: Finished task 82.0 in stage 0.0 (TID 82) in 176 ms on slave1 (80/100)
19/06/27 20:54:00 INFO scheduler.TaskSetManager: Starting task 88.0 in stage 0.0 (TID 88, slave1, partition 88,PROCESS_LOCAL, 2157 bytes)
19/06/27 20:54:00 INFO scheduler.TaskSetManager: Finished task 85.0 in stage 0.0 (TID 85) in 81 ms on slave1 (81/100)
19/06/27 20:54:00 INFO scheduler.TaskSetManager: Starting task 89.0 in stage 0.0 (TID 89, slave1, partition 89,PROCESS_LOCAL, 2157 bytes)
19/06/27 20:54:00 INFO scheduler.TaskSetManager: Starting task 90.0 in stage 0.0 (TID 90, slave1, partition 90,PROCESS_LOCAL, 2157 bytes)
19/06/27 20:54:00 INFO scheduler.TaskSetManager: Starting task 91.0 in stage 0.0 (TID 91, slave1, partition 91,PROCESS_LOCAL, 2157 bytes)
19/06/27 20:54:00 INFO scheduler.TaskSetManager: Starting task 92.0 in stage 0.0 (TID 92, slave1, partition 92,PROCESS_LOCAL, 2157 bytes)
19/06/27 20:54:00 INFO scheduler.TaskSetManager: Finished task 87.0 in stage 0.0 (TID 87) in 123 ms on slave1 (82/100)
19/06/27 20:54:00 INFO scheduler.TaskSetManager: Finished task 88.0 in stage 0.0 (TID 88) in 86 ms on slave1 (83/100)
19/06/27 20:54:00 INFO scheduler.TaskSetManager: Finished task 86.0 in stage 0.0 (TID 86) in 150 ms on slave1 (84/100)
19/06/27 20:54:00 INFO scheduler.TaskSetManager: Finished task 84.0 in stage 0.0 (TID 84) in 247 ms on slave1 (85/100)
19/06/27 20:54:00 INFO scheduler.TaskSetManager: Starting task 93.0 in stage 0.0 (TID 93, slave1, partition 93,PROCESS_LOCAL, 2157 bytes)
19/06/27 20:54:00 INFO scheduler.TaskSetManager: Starting task 94.0 in stage 0.0 (TID 94, slave1, partition 94,PROCESS_LOCAL, 2157 bytes)
19/06/27 20:54:00 INFO scheduler.TaskSetManager: Starting task 95.0 in stage 0.0 (TID 95, slave1, partition 95,PROCESS_LOCAL, 2157 bytes)
19/06/27 20:54:00 INFO scheduler.TaskSetManager: Finished task 91.0 in stage 0.0 (TID 91) in 158 ms on slave1 (86/100)
19/06/27 20:54:00 INFO scheduler.TaskSetManager: Finished task 92.0 in stage 0.0 (TID 92) in 159 ms on slave1 (87/100)
19/06/27 20:54:00 INFO scheduler.TaskSetManager: Finished task 90.0 in stage 0.0 (TID 90) in 164 ms on slave1 (88/100)
19/06/27 20:54:00 INFO scheduler.TaskSetManager: Starting task 96.0 in stage 0.0 (TID 96, slave1, partition 96,PROCESS_LOCAL, 2157 bytes)
19/06/27 20:54:00 INFO scheduler.TaskSetManager: Finished task 89.0 in stage 0.0 (TID 89) in 236 ms on slave1 (89/100)
19/06/27 20:54:00 INFO scheduler.TaskSetManager: Starting task 97.0 in stage 0.0 (TID 97, slave1, partition 97,PROCESS_LOCAL, 2157 bytes)
19/06/27 20:54:00 INFO scheduler.TaskSetManager: Starting task 98.0 in stage 0.0 (TID 98, slave1, partition 98,PROCESS_LOCAL, 2157 bytes)
19/06/27 20:54:00 INFO scheduler.TaskSetManager: Starting task 99.0 in stage 0.0 (TID 99, slave1, partition 99,PROCESS_LOCAL, 2157 bytes)
19/06/27 20:54:00 INFO scheduler.TaskSetManager: Finished task 96.0 in stage 0.0 (TID 96) in 151 ms on slave1 (90/100)
19/06/27 20:54:00 INFO scheduler.TaskSetManager: Finished task 98.0 in stage 0.0 (TID 98) in 100 ms on slave1 (91/100)
19/06/27 20:54:00 INFO scheduler.TaskSetManager: Finished task 93.0 in stage 0.0 (TID 93) in 241 ms on slave1 (92/100)
19/06/27 20:54:00 INFO scheduler.TaskSetManager: Finished task 95.0 in stage 0.0 (TID 95) in 241 ms on slave1 (93/100)
19/06/27 20:54:00 INFO scheduler.TaskSetManager: Finished task 94.0 in stage 0.0 (TID 94) in 243 ms on slave1 (94/100)
19/06/27 20:54:00 INFO scheduler.TaskSetManager: Finished task 97.0 in stage 0.0 (TID 97) in 139 ms on slave1 (95/100)
19/06/27 20:54:00 INFO scheduler.TaskSetManager: Finished task 99.0 in stage 0.0 (TID 99) in 124 ms on slave1 (96/100)
19/06/27 20:54:01 INFO scheduler.TaskSetManager: Finished task 69.0 in stage 0.0 (TID 69) in 1931 ms on slave2 (97/100)
19/06/27 20:54:01 INFO scheduler.TaskSetManager: Finished task 70.0 in stage 0.0 (TID 70) in 1923 ms on slave2 (98/100)
19/06/27 20:54:01 INFO scheduler.TaskSetManager: Finished task 68.0 in stage 0.0 (TID 68) in 1973 ms on slave2 (99/100)
19/06/27 20:54:01 INFO scheduler.TaskSetManager: Finished task 75.0 in stage 0.0 (TID 75) in 1849 ms on slave2 (100/100)
19/06/27 20:54:01 INFO scheduler.DAGScheduler: ResultStage 0 (reduce at SparkPi.scala:36) finished in 12.857 s
19/06/27 20:54:01 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool
19/06/27 20:54:01 INFO scheduler.DAGScheduler: Job 0 finished: reduce at SparkPi.scala:36, took 13.542986 s
Pi is roughly 3.1406799140679915
19/06/27 20:54:01 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/metrics/json,null}
19/06/27 20:54:01 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/stage/kill,null}
19/06/27 20:54:01 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/api,null}
19/06/27 20:54:01 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/,null}
19/06/27 20:54:01 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/static,null}
19/06/27 20:54:01 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/executors/threadDump/json,null}
19/06/27 20:54:01 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/executors/threadDump,null}
19/06/27 20:54:01 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/executors/json,null}
19/06/27 20:54:01 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/executors,null}
19/06/27 20:54:01 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/environment/json,null}
19/06/27 20:54:01 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/environment,null}
19/06/27 20:54:01 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/storage/rdd/json,null}
19/06/27 20:54:01 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/storage/rdd,null}
19/06/27 20:54:01 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/storage/json,null}
19/06/27 20:54:01 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/storage,null}
19/06/27 20:54:01 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/pool/json,null}
19/06/27 20:54:01 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/pool,null}
19/06/27 20:54:01 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/stage/json,null}
19/06/27 20:54:01 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/stage,null}
19/06/27 20:54:01 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/json,null}
19/06/27 20:54:01 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages,null}
19/06/27 20:54:01 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/jobs/job/json,null}
19/06/27 20:54:01 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/jobs/job,null}
19/06/27 20:54:01 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/jobs/json,null}
19/06/27 20:54:01 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/jobs,null}
19/06/27 20:54:01 INFO ui.SparkUI: Stopped Spark web UI at http://192.168.186.10:4040
19/06/27 20:54:01 INFO cluster.SparkDeploySchedulerBackend: Shutting down all executors
19/06/27 20:54:01 INFO cluster.SparkDeploySchedulerBackend: Asking each executor to shut down
19/06/27 20:54:01 INFO spark.MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
19/06/27 20:54:02 INFO storage.MemoryStore: MemoryStore cleared
19/06/27 20:54:02 INFO storage.BlockManager: BlockManager stopped
19/06/27 20:54:02 INFO storage.BlockManagerMaster: BlockManagerMaster stopped
19/06/27 20:54:02 INFO scheduler.OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
19/06/27 20:54:02 INFO spark.SparkContext: Successfully stopped SparkContext
19/06/27 20:54:02 INFO util.ShutdownHookManager: Shutdown hook called
19/06/27 20:54:02 INFO util.ShutdownHookManager: Deleting directory /usr/local/spark-1.6.3-bin-hadoop2.6/spark-23f8e88f-c8d7-495b-ad4d-3a46e1af260f
19/06/27 20:54:02 INFO util.ShutdownHookManager: Deleting directory /usr/local/spark-1.6.3-bin-hadoop2.6/spark-23f8e88f-c8d7-495b-ad4d-3a46e1af260f/httpd-f0176344-5b7c-44af-9bf9-a206db63add
519/06/27 20:54:02 INFO remote.RemoteActorRefProvider$RemotingTerminator: Shutting down remote daemon.
2.6.3 集群模式 Spark on Yarn集群上yarn-cluster¶
./bin/spark-submit --class org.apache.spark.examples.SparkPi --master yarn-cluster lib/spark-examples-1.6.3-hadoop2.6.0.jar 10
详细步骤
[root@master spark]# ./bin/spark-submit --class org.apache.spark.examples.SparkPi --master yarn-cluster lib/spark-examples-1.6.3-hadoop2.6.0.jar 10
19/06/27 20:55:01 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
19/06/27 20:55:01 INFO client.RMProxy: Connecting to ResourceManager at master/192.168.186.10:8032
19/06/27 20:55:02 INFO yarn.Client: Requesting a new application from cluster with 2 NodeManagers
19/06/27 20:55:02 INFO yarn.Client: Verifying our application has not requested more than the maximum memory capability of the cluster (8192 MB per container)
19/06/27 20:55:02 INFO yarn.Client: Will allocate AM container, with 1408 MB memory including 384 MB overhead
19/06/27 20:55:02 INFO yarn.Client: Setting up container launch context for our AM
19/06/27 20:55:02 INFO yarn.Client: Setting up the launch environment for our AM container
19/06/27 20:55:02 INFO yarn.Client: Preparing resources for our AM container
19/06/27 20:55:02 INFO yarn.Client: Uploading resource file:/usr/local/spark-1.6.3-bin-hadoop2.6/lib/spark-assembly-1.6.3-hadoop2.6.0.jar -> hdfs://master:9000/user/root/.sparkStaging/appli
cation_1561552001692_0002/spark-assembly-1.6.3-hadoop2.6.0.jar19/06/27 20:55:04 INFO yarn.Client: Uploading resource file:/usr/local/spark-1.6.3-bin-hadoop2.6/lib/spark-examples-1.6.3-hadoop2.6.0.jar -> hdfs://master:9000/user/root/.sparkStaging/appli
cation_1561552001692_0002/spark-examples-1.6.3-hadoop2.6.0.jar19/06/27 20:55:06 INFO yarn.Client: Uploading resource file:/usr/local/spark-1.6.3-bin-hadoop2.6/spark-de2ecced-7f03-44ed-8094-18bdb532451c/__spark_conf__67909345831230438.zip -> hdfs://mas
ter:9000/user/root/.sparkStaging/application_1561552001692_0002/__spark_conf__67909345831230438.zip19/06/27 20:55:06 INFO spark.SecurityManager: Changing view acls to: root
19/06/27 20:55:06 INFO spark.SecurityManager: Changing modify acls to: root
19/06/27 20:55:06 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root); users with modify permissions: Set(root)
19/06/27 20:55:06 INFO yarn.Client: Submitting application 2 to ResourceManager
19/06/27 20:55:06 INFO impl.YarnClientImpl: Submitted application application_1561552001692_0002
19/06/27 20:55:07 INFO yarn.Client: Application report for application_1561552001692_0002 (state: ACCEPTED)
19/06/27 20:55:07 INFO yarn.Client:
client token: N/A
diagnostics: N/A
ApplicationMaster host: N/A
ApplicationMaster RPC port: -1
queue: default
start time: 1561640106294
final status: UNDEFINED
tracking URL: http://master:8088/proxy/application_1561552001692_0002/
user: root
19/06/27 20:55:08 INFO yarn.Client: Application report for application_1561552001692_0002 (state: ACCEPTED)
19/06/27 20:55:09 INFO yarn.Client: Application report for application_1561552001692_0002 (state: ACCEPTED)
19/06/27 20:55:10 INFO yarn.Client: Application report for application_1561552001692_0002 (state: ACCEPTED)
19/06/27 20:55:11 INFO yarn.Client: Application report for application_1561552001692_0002 (state: ACCEPTED)
19/06/27 20:55:12 INFO yarn.Client: Application report for application_1561552001692_0002 (state: RUNNING)
19/06/27 20:55:12 INFO yarn.Client:
client token: N/A
diagnostics: N/A
ApplicationMaster host: 192.168.186.11
ApplicationMaster RPC port: 0
queue: default
start time: 1561640106294
final status: UNDEFINED
tracking URL: http://master:8088/proxy/application_1561552001692_0002/
user: root
19/06/27 20:55:13 INFO yarn.Client: Application report for application_1561552001692_0002 (state: RUNNING)
19/06/27 20:55:14 INFO yarn.Client: Application report for application_1561552001692_0002 (state: RUNNING)
19/06/27 20:55:15 INFO yarn.Client: Application report for application_1561552001692_0002 (state: RUNNING)
19/06/27 20:55:16 INFO yarn.Client: Application report for application_1561552001692_0002 (state: RUNNING)
19/06/27 20:55:17 INFO yarn.Client: Application report for application_1561552001692_0002 (state: RUNNING)
19/06/27 20:55:18 INFO yarn.Client: Application report for application_1561552001692_0002 (state: RUNNING)
19/06/27 20:55:19 INFO yarn.Client: Application report for application_1561552001692_0002 (state: RUNNING)
19/06/27 20:55:20 INFO yarn.Client: Application report for application_1561552001692_0002 (state: RUNNING)
19/06/27 20:55:21 INFO yarn.Client: Application report for application_1561552001692_0002 (state: RUNNING)
19/06/27 20:55:22 INFO yarn.Client: Application report for application_1561552001692_0002 (state: RUNNING)
19/06/27 20:55:23 INFO yarn.Client: Application report for application_1561552001692_0002 (state: RUNNING)
19/06/27 20:55:24 INFO yarn.Client: Application report for application_1561552001692_0002 (state: RUNNING)
19/06/27 20:55:25 INFO yarn.Client: Application report for application_1561552001692_0002 (state: RUNNING)
19/06/27 20:55:26 INFO yarn.Client: Application report for application_1561552001692_0002 (state: FINISHED)
19/06/27 20:55:26 INFO yarn.Client:
client token: N/A
diagnostics: N/A
ApplicationMaster host: 192.168.186.11
ApplicationMaster RPC port: 0
queue: default
start time: 1561640106294
final status: SUCCEEDED
tracking URL: http://master:8088/proxy/application_1561552001692_0002/
user: root
19/06/27 20:55:26 INFO util.ShutdownHookManager: Shutdown hook called
19/06/27 20:55:26 INFO util.ShutdownHookManager: Deleting directory /usr/local/spark-1.6.3-bin-hadoop2.6/spark-de2ecced-7f03-44ed-8094-18bdb532451c
[root@master spark]#