Uploaded image for project: 'Pig'
  1. Pig
  2. PIG-4059 Pig on Spark
  3. PIG-4206

e2e tests for Spark can not work in hadoop env

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: spark-branch
    • Component/s: spark
    • Labels:
      None

      Description

      Code:
      Use branch spark 9cb14bc
      9cb14bc - PIG-4174: e2e tests for Spark setup

      Env:
      hadoop 1 + Spark 0.9

      How to build:
      ant jar

      Steps to reproduce:
      1. Initialize Variables
      export OLD_PIG_HOME=/usr/local/Cellar/pig/0.12.0 # Should rather be 14
      export HADOOP_CONF_DIR=/usr/local/Cellar/hadoop/1.0.4/conf
      export HADOOP_BIN=/usr/local/Cellar/hadoop/1.0.4/bin/hadoop
      2. Generate Data
      ant -Dharness.old.pig=$OLD_PIG_HOME -Dharness.cluster.conf=$HADOOP_CONF_DIR -Dharness.cluster.bin=$HADOOP_BIN test-e2e-deploy-local
      (You might want to install necessary cpan modules incase of any dependency errors https://cwiki.apache.org/confluence/display/PIG/HowToTest#HowToTest-End-to-endTesting)
      Copy data to hdfs to use with Spark
      hadoop fs -put test/e2e/pig/testdist/data ./
      3. Run particular test
      ant -Dharness.old.pig=$OLD_PIG_HOME -Dharness.cluster.conf=$HADOOP_CONF_DIR -Dharness.cluster.bin=$HADOOP_BIN -Dtests.to.run="-t Checkin_1" test-e2e-spark

      following error message found in Step 3 "Run particular test"
      [exec] ERROR: driver->run() returned the following error message [./test_harness.pl at 156: Cannot create HDFS directory /user/pig/out/root-1411634072-nightly.conf/: 256 - Illegal seek

        Attachments

        1. test_harnesss_1411881848
          60 kB
          liyunzhang
        2. PIG-4206.patch
          3 kB
          liyunzhang
        3. PIG-4206_1.patch
          0.5 kB
          liyunzhang

          Issue Links

            Activity

              People

              • Assignee:
                kellyzly liyunzhang
                Reporter:
                kellyzly liyunzhang
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: