Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-23123

Unable to run Spark Job with Hadoop NameNode Federation using ViewFS

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Not A Problem
    • Affects Version/s: 1.6.3
    • Fix Version/s: None
    • Component/s: Spark Submit
    • Labels:

      Description

      Added following to core-site.xml in order to make use of ViewFS in a NameNode federated cluster.

      <property>
       <name>fs.defaultFS</name>
       <value>viewfs:///</value>
       </property>
      <property>
       <name>fs.viewfs.mounttable.default.link./apps</name>
       <value>hdfs://nameservice1/apps</value>
       </property>
       <property>
       <name>fs.viewfs.mounttable.default.link./app-logs</name>
       <value>hdfs://nameservice2/app-logs</value>
       </property>
       <property>
       <name>fs.viewfs.mounttable.default.link./tmp</name>
       <value>hdfs://nameservice2/tmp</value>
       </property>
       <property>
       <name>fs.viewfs.mounttable.default.link./user</name>
       <value>hdfs://nameservice2/user</value>
       </property>
       <property>
       <name>fs.viewfs.mounttable.default.link./ns1/user</name>
       <value>hdfs://nameservice1/user</value>
       </property>
       <property>
       <name>fs.viewfs.mounttable.default.link./ns2/user</name>
       <value>hdfs://nameservice2/user</value>
       </property>
      

      Got the following error .

      spark-submit --class org.apache.spark.examples.SparkPi --master yarn-client --num-executors 3 --driver-memory 512m --executor-memory 512m --executor-cores 1 ${SPARK_HOME}/lib/spark-examples*.jar 10
      
      
      18/01/17 02:14:45 INFO spark.SparkContext: Added JAR file:/home/nayakxxxx/hdp26_c4000_stg/spark2/lib/spark-examples_2.11-2.1.1.2.6.2.0-205.jar at spark://xxxxx:35633/jars/spark-examples_2.11-2.1.1.2.6.2.0-205.jar with timestamp 1516155285534
      18/01/17 02:14:46 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm2
      18/01/17 02:14:46 INFO yarn.Client: Requesting a new application from cluster with 26 NodeManagers
      18/01/17 02:14:46 INFO yarn.Client: Verifying our application has not requested more than the maximum memory capability of the cluster (13800 MB per container)
      18/01/17 02:14:46 INFO yarn.Client: Will allocate AM container, with 896 MB memory including 384 MB overhead
      18/01/17 02:14:46 INFO yarn.Client: Setting up container launch context for our AM
      18/01/17 02:14:46 INFO yarn.Client: Setting up the launch environment for our AM container
      18/01/17 02:14:46 INFO yarn.Client: Preparing resources for our AM container
      18/01/17 02:14:46 INFO security.HDFSCredentialProvider: getting token for namenode: viewfs:/user/nayakxxxx
      18/01/17 02:14:46 INFO hdfs.DFSClient: Created HDFS_DELEGATION_TOKEN token 22488202 for nayakxxxx on ha-hdfs:nameservice1
      18/01/17 02:14:46 INFO hdfs.DFSClient: Created HDFS_DELEGATION_TOKEN token 50 for nayakxxxx on ha-hdfs:nameservice2
      18/01/17 02:14:47 INFO hive.metastore: Trying to connect to metastore with URI thrift://XXXX:9083
      18/01/17 02:14:47 INFO hive.metastore: Connected to metastore.
      18/01/17 02:14:49 INFO security.HiveCredentialProvider: Get Token from hive metastore: Kind: HIVE_DELEGATION_TOKEN, Service: , Ident: 00 29 6e 61 79 61 6b 6e 69 68 61 72 72 61 30 31 40 53 54 47 32 30 30 30 2e 48 41 44 4f 4f 50 2e 52 41 4b 55 54 45 4e 2e 43 4f 4d 04 68 69 76 65 00 8a 01 61 01 e5 be 03 8a 01 61 25 f2 42 03 8d 02 21 bb 8e 02 b7
      18/01/17 02:14:49 WARN yarn.Client: Neither spark.yarn.jars nor spark.yarn.archive is set, falling back to uploading libraries under SPARK_HOME.
      18/01/17 02:14:50 INFO yarn.Client: Uploading resource file:/tmp/spark-7498ee81-d22b-426e-9466-3a08f7c827b1/__spark_libs__6643608006679813597.zip -> viewfs:/user/nayakxxxx/.sparkStaging/application_1515035441414_275503/__spark_libs__6643608006679813597.zip
      18/01/17 02:14:55 INFO yarn.Client: Uploading resource file:/tmp/spark-7498ee81-d22b-426e-9466-3a08f7c827b1/__spark_conf__405432153902988742.zip -> viewfs:/user/nayakxxxx/.sparkStaging/application_1515035441414_275503/__spark_conf__.zip
      18/01/17 02:14:55 INFO spark.SecurityManager: Changing view acls to: nayakxxxx
      18/01/17 02:14:55 INFO spark.SecurityManager: Changing modify acls to: nayakxxxx
      18/01/17 02:14:55 INFO spark.SecurityManager: Changing view acls groups to:
      18/01/17 02:14:55 INFO spark.SecurityManager: Changing modify acls groups to:
      18/01/17 02:14:55 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(nayakxxxx); groups with view permissions: Set(); users  with modify permissions: Set(nayakxxxx); groups with modify permissions: Set()
      18/01/17 02:14:55 INFO yarn.Client: Submitting application application_1515035441414_275503 to ResourceManager
      18/01/17 02:14:55 INFO impl.YarnClientImpl: Submitted application application_1515035441414_275503
      18/01/17 02:14:55 INFO cluster.SchedulerExtensionServices: Starting Yarn extension services with app application_1515035441414_275503 and attemptId None
      18/01/17 02:14:56 INFO yarn.Client: Application report for application_1515035441414_275503 (state: ACCEPTED)
      18/01/17 02:14:56 INFO yarn.Client:
               client token: Token { kind: YARN_CLIENT_TOKEN, service:  }
               diagnostics: AM container is launched, waiting for AM container to Register with RM
               ApplicationMaster host: N/A
               ApplicationMaster RPC port: -1
               queue: common
               start time: 1516155295100
               final status: UNDEFINED
               tracking URL: http://XXXX:8088/proxy/application_1515035441414_275503/
               user: nayakxxxx
      18/01/17 02:14:57 INFO yarn.Client: Application report for application_1515035441414_275503 (state: ACCEPTED)
      18/01/17 02:14:58 INFO yarn.Client: Application report for application_1515035441414_275503 (state: ACCEPTED)
      18/01/17 02:14:59 INFO yarn.Client: Application report for application_1515035441414_275503 (state: ACCEPTED)
      18/01/17 02:15:00 INFO yarn.Client: Application report for application_1515035441414_275503 (state: ACCEPTED)
      18/01/17 02:15:01 INFO yarn.Client: Application report for application_1515035441414_275503 (state: ACCEPTED)
      18/01/17 02:15:02 INFO yarn.Client: Application report for application_1515035441414_275503 (state: ACCEPTED)
      18/01/17 02:15:03 INFO yarn.Client: Application report for application_1515035441414_275503 (state: ACCEPTED)
      18/01/17 02:15:04 INFO yarn.Client: Application report for application_1515035441414_275503 (state: ACCEPTED)
      18/01/17 02:15:05 INFO yarn.Client: Application report for application_1515035441414_275503 (state: ACCEPTED)
      18/01/17 02:15:06 INFO yarn.Client: Application report for application_1515035441414_275503 (state: ACCEPTED)
      18/01/17 02:15:07 INFO yarn.Client: Application report for application_1515035441414_275503 (state: ACCEPTED)
      18/01/17 02:15:08 INFO yarn.Client: Application report for application_1515035441414_275503 (state: ACCEPTED)
      18/01/17 02:15:09 INFO yarn.Client: Application report for application_1515035441414_275503 (state: ACCEPTED)
      18/01/17 02:15:10 INFO yarn.Client: Application report for application_1515035441414_275503 (state: ACCEPTED)
      18/01/17 02:15:11 INFO yarn.Client: Application report for application_1515035441414_275503 (state: ACCEPTED)
      18/01/17 02:15:12 INFO yarn.Client: Application report for application_1515035441414_275503 (state: ACCEPTED)
      18/01/17 02:15:13 INFO yarn.Client: Application report for application_1515035441414_275503 (state: ACCEPTED)
      18/01/17 02:15:14 INFO yarn.Client: Application report for application_1515035441414_275503 (state: ACCEPTED)
      18/01/17 02:15:15 INFO yarn.Client: Application report for application_1515035441414_275503 (state: ACCEPTED)
      18/01/17 02:15:16 INFO yarn.Client: Application report for application_1515035441414_275503 (state: ACCEPTED)
      18/01/17 02:15:17 INFO yarn.Client: Application report for application_1515035441414_275503 (state: ACCEPTED)
      18/01/17 02:15:18 INFO yarn.Client: Application report for application_1515035441414_275503 (state: ACCEPTED)
      18/01/17 02:15:19 INFO yarn.Client: Application report for application_1515035441414_275503 (state: ACCEPTED)
      18/01/17 02:15:20 INFO yarn.Client: Application report for application_1515035441414_275503 (state: ACCEPTED)
      18/01/17 02:15:21 INFO yarn.Client: Application report for application_1515035441414_275503 (state: ACCEPTED)
      18/01/17 02:15:22 INFO yarn.Client: Application report for application_1515035441414_275503 (state: ACCEPTED)
      18/01/17 02:15:23 INFO yarn.Client: Application report for application_1515035441414_275503 (state: ACCEPTED)
      18/01/17 02:15:24 INFO yarn.Client: Application report for application_1515035441414_275503 (state: ACCEPTED)
      18/01/17 02:15:25 INFO yarn.Client: Application report for application_1515035441414_275503 (state: ACCEPTED)
      18/01/17 02:15:26 INFO yarn.Client: Application report for application_1515035441414_275503 (state: ACCEPTED)
      18/01/17 02:15:27 INFO yarn.Client: Application report for application_1515035441414_275503 (state: ACCEPTED)
      18/01/17 02:15:28 INFO yarn.Client: Application report for application_1515035441414_275503 (state: ACCEPTED)
      18/01/17 02:15:29 INFO yarn.Client: Application report for application_1515035441414_275503 (state: ACCEPTED)
      18/01/17 02:15:30 INFO yarn.Client: Application report for application_1515035441414_275503 (state: ACCEPTED)
      18/01/17 02:15:31 INFO yarn.Client: Application report for application_1515035441414_275503 (state: ACCEPTED)
      18/01/17 02:15:32 INFO yarn.Client: Application report for application_1515035441414_275503 (state: ACCEPTED)
      18/01/17 02:15:33 INFO yarn.Client: Application report for application_1515035441414_275503 (state: ACCEPTED)
      18/01/17 02:15:34 INFO yarn.Client: Application report for application_1515035441414_275503 (state: FAILED)
      18/01/17 02:15:34 INFO yarn.Client:
               client token: N/A
               diagnostics: Application application_1515035441414_275503 failed 10 times due to AM Container for appattempt_1515035441414_275503_000010 exited with  exitCode: -1000
      For more detailed output, check the application tracking page: http://xxxxx:8088/cluster/app/application_1515035441414_275503 Then click on links to logs of each attempt.
      Diagnostics: ViewFs: Cannot initialize: Empty Mount table in config for viewfs://default/
      java.io.IOException: ViewFs: Cannot initialize: Empty Mount table in config for viewfs://default/
              at org.apache.hadoop.fs.viewfs.InodeTree.<init>(InodeTree.java:337)
              at org.apache.hadoop.fs.viewfs.ViewFileSystem$1.<init>(ViewFileSystem.java:168)
              at org.apache.hadoop.fs.viewfs.ViewFileSystem.initialize(ViewFileSystem.java:168)
              at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2795)
              at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:99)
              at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2829)
              at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2811)
              at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:390)
              at org.apache.hadoop.fs.Path.getFileSystem(Path.java:295)
              at org.apache.hadoop.yarn.util.FSDownload.copy(FSDownload.java:251)
              at org.apache.hadoop.yarn.util.FSDownload.access$000(FSDownload.java:63)
              at org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:361)
              at org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:359)
              at java.security.AccessController.doPrivileged(Native Method)
              at javax.security.auth.Subject.doAs(Subject.java:422)
              at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866)
              at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:358)
              at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:62)
              at java.util.concurrent.FutureTask.run(FutureTask.java:266)
              at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
              at java.util.concurrent.FutureTask.run(FutureTask.java:266)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
              at java.lang.Thread.run(Thread.java:748)
      
      Failing this attempt. Failing the application.
               ApplicationMaster host: N/A
               ApplicationMaster RPC port: -1
               queue: common
               start time: 1516155295100
               final status: FAILED
               tracking URL: http://xxxxxx:8088/cluster/app/application_1515035441414_275503
               user: nayakxxxx
      18/01/17 02:15:34 INFO yarn.Client: Deleted staging directory viewfs:/user/nayakxxxx/.sparkStaging/application_1515035441414_275503
      18/01/17 02:15:34 ERROR spark.SparkContext: Error initializing SparkContext.
      

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              niharnayak Nihar Nayak
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: