Pig
  1. Pig
  2. PIG-2860

TestAvroStorageUtils.testGetConcretePathFromGlob fails on some version of hadoop

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.10.0
    • Fix Version/s: 0.11
    • Component/s: piggybank
    • Labels:
      None

      Description

      I found that TestAvroStorageUtils.testGetConcretePathFromGlob fails on some versions of hadoop (not ones that upstream Pig is currently using) with the following error:

      Call From localhost.localdomain/127.0.0.1 to localhost.localdomain:55883 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
      java.net.ConnectException: Call From localhost.localdomain/127.0.0.1 to localhost.localdomain:55883 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
          at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:722)
          at org.apache.hadoop.ipc.Client.call(Client.java:1164)
          at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:195)
          at $Proxy12.getFileInfo(Unknown Source)
          at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:164)
          at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:83)
          at $Proxy12.getFileInfo(Unknown Source)
          at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:613)
          at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1399)
          at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:740)
          at org.apache.hadoop.fs.FileSystem.getFileStatus(FileSystem.java:2083)
          at org.apache.hadoop.fs.FileSystem.globStatusInternal(FileSystem.java:1547)
          at org.apache.hadoop.fs.FileSystem.globStatus(FileSystem.java:1488)
          at org.apache.pig.piggybank.storage.avro.AvroStorageUtils.getConcretePathFromGlob(AvroStorageUtils.java:146)
          at org.apache.pig.piggybank.test.storage.avro.TestAvroStorageUtils.testGetConcretePathFromGlob(TestAvroStorageUtils.java:142)
      

      The fix is to explicitly add the URI scheme "file://" to the path that is used in the test.

      1. PIG-2860.patch
        1 kB
        Cheolsoo Park

        Activity

        Bill Graham made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        Cheolsoo Park made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Resolution Fixed [ 1 ]
        Hide
        Cheolsoo Park added a comment -

        Closing it since the patch is committed.

        Thanks Jonathan for committing my patch!

        Show
        Cheolsoo Park added a comment - Closing it since the patch is committed. Thanks Jonathan for committing my patch!
        Hide
        Jonathan Coveney added a comment -

        +1

        Thanks Chelsoo!

        Show
        Jonathan Coveney added a comment - +1 Thanks Chelsoo!
        Cheolsoo Park made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Hide
        Cheolsoo Park added a comment -
        Show
        Cheolsoo Park added a comment - Review board: https://reviews.apache.org/r/6412/
        Cheolsoo Park made changes -
        Field Original Value New Value
        Attachment PIG-2860.patch [ 12539376 ]
        Cheolsoo Park created issue -

          People

          • Assignee:
            Cheolsoo Park
            Reporter:
            Cheolsoo Park
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development