Uploaded image for project: 'Apache Arrow'
  1. Apache Arrow
  2. ARROW-4058

[C++] arrow-io-hdfs-test fails when run against HDFS cluster from docker-compose

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.12.0
    • Component/s: C++
    • Labels:
      None

      Description

      When running HDFS with docker-compose up hdfs-data-node-1 I get the following error:

      [==========] Running 2 tests from 2 test cases.
      [----------] Global test environment set-up.
      [----------] 1 test from TestHadoopFileSystem/0, where TypeParam = arrow::io::JNIDriver
      [ RUN      ] TestHadoopFileSystem/0.LargeFile
      [       OK ] TestHadoopFileSystem/0.LargeFile (1101 ms)
      [----------] 1 test from TestHadoopFileSystem/0 (1101 ms total)
      
      [----------] 1 test from TestHadoopFileSystem/1, where TypeParam = arrow::io::PivotalDriver
      [ RUN      ] TestHadoopFileSystem/1.LargeFile
      Loading libhdfs3 failed, skipping tests gracefully. IOError: ../src/arrow/io/hdfs-internal.cc:566 code: try_dlopen(libhdfs3_potential_paths, "libhdfs3", shim->handle)
      Unable to load libhdfs3
      Driver not loaded, skipping
      [       OK ] TestHadoopFileSystem/1.LargeFile (0 ms)
      [----------] 1 test from TestHadoopFileSystem/1 (0 ms total)
      
      [----------] Global test environment tear-down
      [==========] 2 tests from 2 test cases ran. (1101 ms total)
      [  PASSED  ] 2 tests.
      (arrow-dev) 21:06 ~/code/arrow/cpp/build  (ARROW-2919)$ ./debug/arrow-io-hdfs-test
      Running main() from gtest_main.cc
      [==========] Running 24 tests from 2 test cases.
      [----------] Global test environment set-up.
      [----------] 12 tests from TestHadoopFileSystem/0, where TypeParam = arrow::io::JNIDriver
      [ RUN      ] TestHadoopFileSystem/0.ConnectsAgain
      [       OK ] TestHadoopFileSystem/0.ConnectsAgain (885 ms)
      [ RUN      ] TestHadoopFileSystem/0.MultipleClients
      [       OK ] TestHadoopFileSystem/0.MultipleClients (194 ms)
      [ RUN      ] TestHadoopFileSystem/0.MakeDirectory
      [       OK ] TestHadoopFileSystem/0.MakeDirectory (139 ms)
      [ RUN      ] TestHadoopFileSystem/0.GetCapacityUsed
      [       OK ] TestHadoopFileSystem/0.GetCapacityUsed (119 ms)
      [ RUN      ] TestHadoopFileSystem/0.GetPathInfo
      [       OK ] TestHadoopFileSystem/0.GetPathInfo (229 ms)
      [ RUN      ] TestHadoopFileSystem/0.AppendToFile
      18/12/17 21:06:38 WARN hdfs.DFSClient: DataStreamer Exception
      java.io.IOException: Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available to try. (Nodes: current=[172.19.0.3:50010], original=[172.19.0.3:50010]). The current failed datanode replacement policy is DEFAULT, and a client may configure this via 'dfs.client.block.write.replace-datanode-on-failure.policy' in its configuration.
      	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:1036)
      	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:1102)
      	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1249)
      	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:593)
      FSDataOutputStream#close error:
      java.io.IOException: Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available to try. (Nodes: current=[172.19.0.3:50010], original=[172.19.0.3:50010]). The current failed datanode replacement policy is DEFAULT, and a client may configure this via 'dfs.client.block.write.replace-datanode-on-failure.policy' in its configuration.
      	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:1036)
      	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:1102)
      	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1249)
      	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:593)
      WARNING: Logging before InitGoogleLogging() is written to STDERR
      F1217 21:06:38.871151 25929 hdfs.cc:275]  Check failed: impl_->Close().ok() 
      *** Check failure stack trace: ***
      Aborted (core dumped)
      

        Attachments

          Activity

            People

            • Assignee:
              wesmckinn Wes McKinney
              Reporter:
              wesmckinn Wes McKinney
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: