Uploaded image for project: 'Pig'
  1. Pig
  2. PIG-2791

Pig does not work with ViewFileSystem



    • Type: Bug
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.10.0
    • Fix Version/s: None
    • Component/s: grunt
    • Labels:
    • Environment:

      Pig QE


      The Yahoo Pig QE team ran into a blocking issue when trying to test Client-Side Mount Tables, on a Federated cluster with two NNs, this blocks Pig Testing on Federation.

      Federation relies strongly on the use of CSMT with viewFS, QE found that in this configuration it is not possible to enter grunt shell because Pig makes a call to getDefaultReplication() on the fs, which is ambiguous over viewFS and causes core to throw a org.apache.hadoop.fs.viewfs.NotInMountpointException: "getDefaultReplication on empty path is invalid".

      This in turn cause Pig to exit with an internal error as follows:

      2012-07-06 22:20:25,657 [main] INFO org.apache.pig.Main - Apache Pig version (r1348169) compiled Jun 08 2012, 17:58:42
      2012-07-06 22:20:26,074 [main] WARN org.apache.hadoop.conf.Configuration - mapred.used.genericoptionsparser is deprecated. Instead, use mapreduce.client.genericoptionsparser.used
      2012-07-06 22:20:26,076 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to hadoop file system at: viewfs:///
      2012-07-06 22:20:26,080 [main] WARN org.apache.hadoop.conf.Configuration - fs.default.name is deprecated. Instead, use fs.defaultFS
      2012-07-06 22:20:26,522 [main] ERROR org.apache.pig.Main - ERROR 2999: Unexpected internal error. getDefaultReplication on empty path is invalid
      2012-07-06 22:20:26,522 [main] WARN org.apache.pig.Main - There is no log file to write to.
      2012-07-06 22:20:26,522 [main] ERROR org.apache.pig.Main - org.apache.hadoop.fs.viewfs.NotInMountpointException: getDefaultReplication on empty path is invalid
      at org.apache.hadoop.fs.viewfs.ViewFileSystem.getDefaultReplication(ViewFileSystem.java:482)
      at org.apache.pig.backend.hadoop.datastorage.HDataStorage.init(HDataStorage.java:77)
      at org.apache.pig.backend.hadoop.datastorage.HDataStorage.<init>(HDataStorage.java:58)
      at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:205)
      at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:118)
      at org.apache.pig.impl.PigContext.connect(PigContext.java:208)
      at org.apache.pig.PigServer.<init>(PigServer.java:246)
      at org.apache.pig.PigServer.<init>(PigServer.java:231)
      at org.apache.pig.tools.grunt.Grunt.<init>(Grunt.java:47)
      at org.apache.pig.Main.run(Main.java:487)
      at org.apache.pig.Main.main(Main.java:111)


        1. asf_test_notes.txt
          7 kB
          Araceli Henley
        2. FixMiniCluster-branch10.patch
          2 kB
          Rohini Palaniswamy
        3. FixMiniCluster-branch10-1.patch
          1 kB
          Rohini Palaniswamy
        4. PIG-2791-0.patch
          1 kB
          Daniel Dai
        5. PIG-2791-1.patch
          7 kB
          Rohini Palaniswamy
        6. PIG-2791-2.patch
          7 kB
          Rohini Palaniswamy
        7. PIG-2791-3-branch10.patch
          15 kB
          Rohini Palaniswamy
        8. PIG-2791-3-trunk.patch
          16 kB
          Rohini Palaniswamy
        9. PIG-2791-4-branch10.patch
          15 kB
          Rohini Palaniswamy
        10. PIG-2791-4-trunk.patch
          16 kB
          Rohini Palaniswamy
        11. PIG-2791-5-trunk.patch
          15 kB
          Rohini Palaniswamy

          Issue Links



              • Assignee:
                rohini Rohini Palaniswamy
                patwhitey2007 patrick white
              • Votes:
                0 Vote for this issue
                7 Start watching this issue


                • Created: