Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-10844

test_libhdfs_threaded_hdfs_static and test_libhdfs_zerocopy_hdfs_static are failing

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 3.0.0-alpha1
    • Fix Version/s: 3.0.0-alpha2
    • Component/s: libhdfs
    • Labels:
      None

      Description

      From qbt report:

          Failed CTEST tests :
      
             test_test_libhdfs_threaded_hdfs_static 
             test_test_libhdfs_zerocopy_hdfs_static 
      

      https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/156/artifact/out/patch-hadoop-hdfs-project_hadoop-hdfs-native-client-ctest.txt

        Issue Links

          Activity

          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10408 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10408/)
          HDFS-10844. test_libhdfs_threaded_hdfs_static and (aajisaka: rev 2d1bf53c7e208ad951ebed7ee3f2e44582dfd151)

          • (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs/hdfs.c
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10408 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10408/ ) HDFS-10844 . test_libhdfs_threaded_hdfs_static and (aajisaka: rev 2d1bf53c7e208ad951ebed7ee3f2e44582dfd151) (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs/hdfs.c
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Committed this to trunk. Thanks SammiChen and Kai Zheng for the review.

          Ideally, all of this native code would be hdfs-client. Then when a change is made to to that code, this code will also get tested.

          Agreed. Filed HDFS-10848.

          Show
          ajisakaa Akira Ajisaka added a comment - Committed this to trunk. Thanks SammiChen and Kai Zheng for the review. Ideally, all of this native code would be hdfs-client. Then when a change is made to to that code, this code will also get tested. Agreed. Filed HDFS-10848 .
          Hide
          aw Allen Wittenauer added a comment -

          This is really a symptom of the kind of nonsensical divisions of modules that are in HDFS and MapReduce. Ideally, all of this native code would be hdfs-client. Then when a change is made to to that code, this code will also get tested.

          This does, however, underscore that committers really really really need to pay attention to the qbt report after they do a commit. If tests that weren't failing before are now consistently failing, it's likely because there was a side effect of a recent change.

          Show
          aw Allen Wittenauer added a comment - This is really a symptom of the kind of nonsensical divisions of modules that are in HDFS and MapReduce. Ideally, all of this native code would be hdfs-client. Then when a change is made to to that code, this code will also get tested. This does, however, underscore that committers really really really need to pay attention to the qbt report after they do a commit. If tests that weren't failing before are now consistently failing, it's likely because there was a side effect of a recent change.
          Hide
          drankye Kai Zheng added a comment -

          Thanks Akira for the insights and info, and Sammi for the verifying!

          +1 on the patch.

          Show
          drankye Kai Zheng added a comment - Thanks Akira for the insights and info, and Sammi for the verifying! +1 on the patch.
          Hide
          Sammi SammiChen added a comment -

          Thanks Akira Ajisaka and Allen Wittenauer for all the effort to narrow down the issue and create the patch!
          I reviewed the patch and verified it. The previous failed test cases all passed with the new patch.

          Great Job!

          Show
          Sammi SammiChen added a comment - Thanks Akira Ajisaka and Allen Wittenauer for all the effort to narrow down the issue and create the patch! I reviewed the patch and verified it. The previous failed test cases all passed with the new patch. Great Job!
          Hide
          ajisakaa Akira Ajisaka added a comment -

          How to trigger the ctest invoking the failed test cases?

          $ cd hadoop-hdfs-project/hadoop-hdfs-native-client
          $ mvn test -Pnative
          

          In HDFS-8905, Jenkins did not pick up the tests because the patch did not change hadoop-hdfs-native-client module. Here is the output: https://builds.apache.org/job/PreCommit-HDFS-Build/16657/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-native-client.txt
          All the tests passed.

          Show
          ajisakaa Akira Ajisaka added a comment - How to trigger the ctest invoking the failed test cases? $ cd hadoop-hdfs-project/hadoop-hdfs- native -client $ mvn test -Pnative In HDFS-8905 , Jenkins did not pick up the tests because the patch did not change hadoop-hdfs-native-client module. Here is the output: https://builds.apache.org/job/PreCommit-HDFS-Build/16657/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-native-client.txt All the tests passed.
          Hide
          drankye Kai Zheng added a comment -

          Thanks Akira Ajisaka for the fix! The patch LGTM. A question:

          How to trigger the ctest invoking the failed test cases? I'm wondering if the Jenkins building had verified them or not, by default. It would be great if someone could run the tests and provide the output here additionally.

          Show
          drankye Kai Zheng added a comment - Thanks Akira Ajisaka for the fix! The patch LGTM. A question: How to trigger the ctest invoking the failed test cases? I'm wondering if the Jenkins building had verified them or not, by default. It would be great if someone could run the tests and provide the output here additionally.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 27s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 mvninstall 7m 10s trunk passed
          +1 compile 0m 15s trunk passed
          +1 mvnsite 0m 13s trunk passed
          +1 mvneclipse 0m 12s trunk passed
          +1 mvninstall 0m 9s the patch passed
          +1 compile 0m 12s the patch passed
          +1 cc 0m 12s the patch passed
          +1 javac 0m 12s the patch passed
          +1 mvnsite 0m 11s the patch passed
          +1 mvneclipse 0m 10s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 unit 0m 36s hadoop-hdfs-native-client in the patch passed.
          +1 asflicense 0m 19s The patch does not generate ASF License warnings.
          10m 49s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Issue HDFS-10844
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12827404/HDFS-10844.01.patch
          Optional Tests asflicense compile cc mvnsite javac unit
          uname Linux eacfefd69ef7 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 7fdfcd8
          Default Java 1.8.0_101
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16657/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs-native-client U: hadoop-hdfs-project/hadoop-hdfs-native-client
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16657/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 27s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 7m 10s trunk passed +1 compile 0m 15s trunk passed +1 mvnsite 0m 13s trunk passed +1 mvneclipse 0m 12s trunk passed +1 mvninstall 0m 9s the patch passed +1 compile 0m 12s the patch passed +1 cc 0m 12s the patch passed +1 javac 0m 12s the patch passed +1 mvnsite 0m 11s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 unit 0m 36s hadoop-hdfs-native-client in the patch passed. +1 asflicense 0m 19s The patch does not generate ASF License warnings. 10m 49s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue HDFS-10844 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12827404/HDFS-10844.01.patch Optional Tests asflicense compile cc mvnsite javac unit uname Linux eacfefd69ef7 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 7fdfcd8 Default Java 1.8.0_101 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16657/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs-native-client U: hadoop-hdfs-project/hadoop-hdfs-native-client Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16657/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Thanks Allen Wittenauer for the candidate list.

          Show
          ajisakaa Akira Ajisaka added a comment - Thanks Allen Wittenauer for the candidate list.
          Hide
          ajisakaa Akira Ajisaka added a comment -

          These test were broken by HDFS-8905. HDFS-8905 moved DFSInputStream$ReadStatistics to ReadStatistics but hdfs.c was not updated accordingly.

          Show
          ajisakaa Akira Ajisaka added a comment - These test were broken by HDFS-8905 . HDFS-8905 moved DFSInputStream$ReadStatistics to ReadStatistics but hdfs.c was not updated accordingly.
          Hide
          aw Allen Wittenauer added a comment -

          Going back through the mail archive, they appear to have started failing everyday after this set of patches:

          [Aug 24, 2016 7:47:52 AM] (aajisaka) HADOOP-13538. Deprecate getInstance and initialize methods with Path in
          [Aug 24, 2016 1:46:47 PM] (daryn) HDFS-10762. Pass IIP for file status related methods
          [Aug 24, 2016 1:57:23 PM] (kai.zheng) HDFS-8905. Refactor DFSInputStream#ReaderStrategy. Contributed by Kai
          [Aug 24, 2016 2:17:05 PM] (kai.zheng) MAPREDUCE-6578. Add support for HDFS heterogeneous storage testing to
          [Aug 24, 2016 2:40:51 PM] (jlowe) MAPREDUCE-6761. Regression when handling providers - invalid
          [Aug 24, 2016 5:14:46 PM] (xiao) HADOOP-13396. Allow pluggable audit loggers in KMS. Contributed by Xiao
          [Aug 24, 2016 8:21:08 PM] (kihwal) HDFS-10772. Reduce byte/string conversions for get listing. Contributed
          [Aug 25, 2016 1:55:00 AM] (aajisaka) MAPREDUCE-6767. TestSlive fails after a common change. Contributed by
          [Aug 25, 2016 4:54:57 AM] (aajisaka) HADOOP-13534. Remove unused TrashPolicy#getInstance and initialize code.

          Show
          aw Allen Wittenauer added a comment - Going back through the mail archive, they appear to have started failing everyday after this set of patches: [Aug 24, 2016 7:47:52 AM] (aajisaka) HADOOP-13538 . Deprecate getInstance and initialize methods with Path in [Aug 24, 2016 1:46:47 PM] (daryn) HDFS-10762 . Pass IIP for file status related methods [Aug 24, 2016 1:57:23 PM] (kai.zheng) HDFS-8905 . Refactor DFSInputStream#ReaderStrategy. Contributed by Kai [Aug 24, 2016 2:17:05 PM] (kai.zheng) MAPREDUCE-6578 . Add support for HDFS heterogeneous storage testing to [Aug 24, 2016 2:40:51 PM] (jlowe) MAPREDUCE-6761 . Regression when handling providers - invalid [Aug 24, 2016 5:14:46 PM] (xiao) HADOOP-13396 . Allow pluggable audit loggers in KMS. Contributed by Xiao [Aug 24, 2016 8:21:08 PM] (kihwal) HDFS-10772 . Reduce byte/string conversions for get listing. Contributed [Aug 25, 2016 1:55:00 AM] (aajisaka) MAPREDUCE-6767 . TestSlive fails after a common change. Contributed by [Aug 25, 2016 4:54:57 AM] (aajisaka) HADOOP-13534 . Remove unused TrashPolicy#getInstance and initialize code.
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Attaching the log.

          Show
          ajisakaa Akira Ajisaka added a comment - Attaching the log.

            People

            • Assignee:
              ajisakaa Akira Ajisaka
              Reporter:
              ajisakaa Akira Ajisaka
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development