Uploaded image for project: 'Bigtop'
  1. Bigtop
  2. BIGTOP-2148

generator.SparkDriver outputs empty data in cluster mode

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.1.0
    • Component/s: blueprints
    • Labels:

      Description

      About generator.SparkDriver, bigpetstore-spark/README.md says:

      You will need to change the master if you want to run on a cluster.

      But in reality, if you run it on a cluster, you'll get empty data:

      [sekikn@mobile bigpetstore-spark]$ HADOOP_CONF_DIR=/usr/local/Cellar/hadoop/2.7.1/libexec/etc/hadoop spark-submit --master yarn-cluster --class org.apache.bigtop.bigpetstore.spark.generator.SparkDriver build/libs/bigpetstore-spark-1.1.0-SNAPSHOT-all.jar generated_data 10 1000 365.0 345
      
      (snip)
      
      15/11/18 00:12:30 INFO Client: Application report for application_1447772975157_0003 (state: FINISHED)
      15/11/18 00:12:30 INFO Client: 
      	 client token: N/A
      	 diagnostics: N/A
      	 ApplicationMaster host: 192.168.0.4
      	 ApplicationMaster RPC port: 0
      	 queue: default
      	 start time: 1447773097856
      	 final status: SUCCEEDED
      	 tracking URL: http://mobile.local:8088/proxy/application_1447772975157_0003/
      	 user: sekikn
      15/11/18 00:12:30 INFO ShutdownHookManager: Shutdown hook called
      15/11/18 00:12:30 INFO ShutdownHookManager: Deleting directory /private/var/folders/n2/1bnspz7j4q7100jmh610zd200000gn/T/spark-ccfcde0c-ea95-4361-b2b3-b709a92bee59
      [sekikn@mobile bigpetstore-spark]$ hdfs dfs -ls generated_data/transactions
      15/11/18 00:13:16 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
      Found 3 items
      -rw-r--r--   3 sekikn supergroup          0 2015-11-18 00:12 generated_data/transactions/_SUCCESS
      -rw-r--r--   3 sekikn supergroup          0 2015-11-18 00:12 generated_data/transactions/part-00000
      -rw-r--r--   3 sekikn supergroup          0 2015-11-18 00:12 generated_data/transactions/part-00001
      

      This is because simulationLength is a variable and always -1 in RDD function. It must be a constant or broadcasted.

        Activity

        Hide
        sekikn Kengo Seki added a comment -

        Attaching a patch. After applying this, generator.SparkDriver generates non-empty output as follows:

        [sekikn@mobile bigpetstore-spark]$ hdfs dfs -rm -r generated_data; HADOOP_CONF_DIR=/usr/local/Cellar/hadoop/2.7.1/libexec/etc/hadoop spark-submit --master yarn-cluster --class org.apache.bigtop.bigpetstore.spark.generator.SparkDriver build/libs/bigpetstore-spark-1.1.0-SNAPSHOT-all.jar generated_data 10 1000 365.0 345
        
        (snip)
        
        15/11/18 00:24:36 INFO Client: Application report for application_1447772975157_0005 (state: FINISHED)
        15/11/18 00:24:36 INFO Client: 
        	 client token: N/A
        	 diagnostics: N/A
        	 ApplicationMaster host: 192.168.0.4
        	 ApplicationMaster RPC port: 0
        	 queue: default
        	 start time: 1447773822522
        	 final status: SUCCEEDED
        	 tracking URL: http://mobile.local:8088/proxy/application_1447772975157_0005/
        	 user: sekikn
        15/11/18 00:24:37 INFO ShutdownHookManager: Shutdown hook called
        15/11/18 00:24:37 INFO ShutdownHookManager: Deleting directory /private/var/folders/n2/1bnspz7j4q7100jmh610zd200000gn/T/spark-eaa5069c-4f84-4083-bd1a-ede5dc57dc25
        [sekikn@mobile bigpetstore-spark]$ hdfs dfs -ls generated_data/transactions
        15/11/18 00:24:55 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
        Found 3 items
        -rw-r--r--   3 sekikn supergroup          0 2015-11-18 00:24 generated_data/transactions/_SUCCESS
        -rw-r--r--   3 sekikn supergroup    3430043 2015-11-18 00:24 generated_data/transactions/part-00000
        -rw-r--r--   3 sekikn supergroup    3452131 2015-11-18 00:24 generated_data/transactions/part-00001
        [sekikn@mobile bigpetstore-spark]$ 
        
        Show
        sekikn Kengo Seki added a comment - Attaching a patch. After applying this, generator.SparkDriver generates non-empty output as follows: [sekikn@mobile bigpetstore-spark]$ hdfs dfs -rm -r generated_data; HADOOP_CONF_DIR=/usr/local/Cellar/hadoop/2.7.1/libexec/etc/hadoop spark-submit --master yarn-cluster --class org.apache.bigtop.bigpetstore.spark.generator.SparkDriver build/libs/bigpetstore-spark-1.1.0-SNAPSHOT-all.jar generated_data 10 1000 365.0 345 (snip) 15/11/18 00:24:36 INFO Client: Application report for application_1447772975157_0005 (state: FINISHED) 15/11/18 00:24:36 INFO Client: client token: N/A diagnostics: N/A ApplicationMaster host: 192.168.0.4 ApplicationMaster RPC port: 0 queue: default start time: 1447773822522 final status: SUCCEEDED tracking URL: http: //mobile.local:8088/proxy/application_1447772975157_0005/ user: sekikn 15/11/18 00:24:37 INFO ShutdownHookManager: Shutdown hook called 15/11/18 00:24:37 INFO ShutdownHookManager: Deleting directory / private / var /folders/n2/1bnspz7j4q7100jmh610zd200000gn/T/spark-eaa5069c-4f84-4083-bd1a-ede5dc57dc25 [sekikn@mobile bigpetstore-spark]$ hdfs dfs -ls generated_data/transactions 15/11/18 00:24:55 WARN util.NativeCodeLoader: Unable to load native -hadoop library for your platform... using builtin-java classes where applicable Found 3 items -rw-r--r-- 3 sekikn supergroup 0 2015-11-18 00:24 generated_data/transactions/_SUCCESS -rw-r--r-- 3 sekikn supergroup 3430043 2015-11-18 00:24 generated_data/transactions/part-00000 -rw-r--r-- 3 sekikn supergroup 3452131 2015-11-18 00:24 generated_data/transactions/part-00001 [sekikn@mobile bigpetstore-spark]$
        Hide
        evans_ye Evans Ye added a comment -

        Looks good to me and +1.
        RJ Nowling this is a simple fix would you mind to double confirm?

        Show
        evans_ye Evans Ye added a comment - Looks good to me and +1. RJ Nowling this is a simple fix would you mind to double confirm?
        Hide
        rnowling RJ Nowling added a comment -

        +1 Thank for fixing that Kengo Seki! I committed the patch.

        Evans Ye thanks for reviewing!

        Show
        rnowling RJ Nowling added a comment - +1 Thank for fixing that Kengo Seki ! I committed the patch. Evans Ye thanks for reviewing!

          People

          • Assignee:
            sekikn Kengo Seki
            Reporter:
            sekikn Kengo Seki
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development