Pig
  1. Pig
  2. PIG-2791

Pig does not work with ViewFileSystem

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.10.0
    • Fix Version/s: None
    • Component/s: grunt
    • Labels:
      None
    • Environment:

      Pig QE

      Description

      The Yahoo Pig QE team ran into a blocking issue when trying to test Client-Side Mount Tables, on a Federated cluster with two NNs, this blocks Pig Testing on Federation.

      Federation relies strongly on the use of CSMT with viewFS, QE found that in this configuration it is not possible to enter grunt shell because Pig makes a call to getDefaultReplication() on the fs, which is ambiguous over viewFS and causes core to throw a org.apache.hadoop.fs.viewfs.NotInMountpointException: "getDefaultReplication on empty path is invalid".

      This in turn cause Pig to exit with an internal error as follows:

      2012-07-06 22:20:25,657 [main] INFO org.apache.pig.Main - Apache Pig version 0.10.1.0.1206081058 (r1348169) compiled Jun 08 2012, 17:58:42
      2012-07-06 22:20:26,074 [main] WARN org.apache.hadoop.conf.Configuration - mapred.used.genericoptionsparser is deprecated. Instead, use mapreduce.client.genericoptionsparser.used
      2012-07-06 22:20:26,076 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to hadoop file system at: viewfs:///
      2012-07-06 22:20:26,080 [main] WARN org.apache.hadoop.conf.Configuration - fs.default.name is deprecated. Instead, use fs.defaultFS
      2012-07-06 22:20:26,522 [main] ERROR org.apache.pig.Main - ERROR 2999: Unexpected internal error. getDefaultReplication on empty path is invalid
      2012-07-06 22:20:26,522 [main] WARN org.apache.pig.Main - There is no log file to write to.
      2012-07-06 22:20:26,522 [main] ERROR org.apache.pig.Main - org.apache.hadoop.fs.viewfs.NotInMountpointException: getDefaultReplication on empty path is invalid
      at org.apache.hadoop.fs.viewfs.ViewFileSystem.getDefaultReplication(ViewFileSystem.java:482)
      at org.apache.pig.backend.hadoop.datastorage.HDataStorage.init(HDataStorage.java:77)
      at org.apache.pig.backend.hadoop.datastorage.HDataStorage.<init>(HDataStorage.java:58)
      at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:205)
      at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:118)
      at org.apache.pig.impl.PigContext.connect(PigContext.java:208)
      at org.apache.pig.PigServer.<init>(PigServer.java:246)
      at org.apache.pig.PigServer.<init>(PigServer.java:231)
      at org.apache.pig.tools.grunt.Grunt.<init>(Grunt.java:47)
      at org.apache.pig.Main.run(Main.java:487)
      at org.apache.pig.Main.main(Main.java:111)

      1. PIG-2791-5-trunk.patch
        15 kB
        Rohini Palaniswamy
      2. PIG-2791-4-trunk.patch
        16 kB
        Rohini Palaniswamy
      3. PIG-2791-4-branch10.patch
        15 kB
        Rohini Palaniswamy
      4. PIG-2791-3-trunk.patch
        16 kB
        Rohini Palaniswamy
      5. PIG-2791-3-branch10.patch
        15 kB
        Rohini Palaniswamy
      6. PIG-2791-2.patch
        7 kB
        Rohini Palaniswamy
      7. PIG-2791-1.patch
        7 kB
        Rohini Palaniswamy
      8. PIG-2791-0.patch
        1 kB
        Daniel Dai
      9. FixMiniCluster-branch10-1.patch
        1 kB
        Rohini Palaniswamy
      10. FixMiniCluster-branch10.patch
        2 kB
        Rohini Palaniswamy
      11. asf_test_notes.txt
        7 kB
        Araceli Henley

        Issue Links

          Activity

          patrick white created issue -
          Harsh J made changes -
          Field Original Value New Value
          Link This issue depends on HADOOP-8430 [ HADOOP-8430 ]
          Daniel Dai made changes -
          Attachment PIG-2791-0.patch [ 12535590 ]
          Rohini Palaniswamy made changes -
          Attachment PIG-2791-1.patch [ 12536165 ]
          Rohini Palaniswamy made changes -
          Attachment PIG-2791-2.patch [ 12536166 ]
          Araceli Henley made changes -
          Attachment asf_test_notes.txt [ 12536316 ]
          Rohini Palaniswamy made changes -
          Attachment PIG-2791-3-branch10.patch [ 12537524 ]
          Attachment PIG-2791-3-trunk.patch [ 12537525 ]
          Rohini Palaniswamy made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Assignee Rohini Palaniswamy [ rohini ]
          Rohini Palaniswamy made changes -
          Summary Can not enter grunt shell when using viewFS filesystem with CSMT and Federation Pig does not work with Namenode Federation
          Rohini Palaniswamy made changes -
          Attachment PIG-2791-4-branch10.patch [ 12538193 ]
          Attachment PIG-2791-4-trunk.patch [ 12538194 ]
          Rohini Palaniswamy made changes -
          Attachment PIG-2791-5-trunk.patch [ 12538265 ]
          Rohini Palaniswamy made changes -
          Attachment FixMiniCluster-branch10.patch [ 12538266 ]
          Rohini Palaniswamy made changes -
          Attachment FixMiniCluster-branch10-1.patch [ 12540745 ]
          Daryn Sharp made changes -
          Summary Pig does not work with Namenode Federation Pig does not work with ViewFileSystem
          Daniel Dai made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]
          Cheolsoo Park made changes -
          Link This issue breaks PIG-2930 [ PIG-2930 ]
          Arup Malakar made changes -
          Link This issue is related to HIVE-3645 [ HIVE-3645 ]
          Gavin made changes -
          Link This issue depends on HADOOP-8430 [ HADOOP-8430 ]
          Gavin made changes -
          Link This issue depends upon HADOOP-8430 [ HADOOP-8430 ]

            People

            • Assignee:
              Rohini Palaniswamy
              Reporter:
              patrick white
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development