Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-10798

globStatus() should always return a sorted list of files

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 2.3.0
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: None
    • Labels:
    • Target Version/s:

      Description

      (FileSystem) globStatus() does not return a sorted file list anymore.

      But the API says: " ... Results are sorted by their names."

      Seems to be lost, when the Globber Object was introduced. Can't find a sort in actual code.

      code to check this behavior:

              Configuration conf = new Configuration();
              FileSystem fs = FileSystem.get(conf);
              Path path = new Path("/tmp/" + System.currentTimeMillis());
              fs.mkdirs(path);
              fs.deleteOnExit(path);
              fs.createNewFile(new Path(path, "2"));
              fs.createNewFile(new Path(path, "3"));
              fs.createNewFile(new Path(path, "1"));
      
              FileStatus[] status = fs.globStatus(new Path(path, "*"));
              Collection list = new ArrayList();
              for (FileStatus f: status) {
                  list.add(f.getPath().toString());
                  //System.out.println(f.getPath().toString());
              }
              boolean sorted = Ordering.natural().isOrdered(list);
              Assert.assertTrue(sorted);
      

        Issue Links

          Activity

          Hide
          daryn Daryn Sharp added a comment -

          Which filesystem? Ie. hdfs or local? GlobStatus should return a sorted list because it calls listStatus which returns a sorted list. If the glob results are not sorted, then listStatus is likely the culprit.

          Show
          daryn Daryn Sharp added a comment - Which filesystem? Ie. hdfs or local? GlobStatus should return a sorted list because it calls listStatus which returns a sorted list. If the glob results are not sorted, then listStatus is likely the culprit.
          Hide
          fborchers Felix Borchers added a comment -

          Tested on local filesystem. works on a macosx, but not on linux.

          Show
          fborchers Felix Borchers added a comment - Tested on local filesystem. works on a macosx, but not on linux.
          Hide
          cmccabe Colin P. McCabe added a comment -

          Daryn Sharp: smart. I was wondering why it worked on HDFS but not on localFS.

          I don't know how I feel about this JIRA. The obvious solution is to have localFS sort its output on Linux, but this will cripple performance by forcing us to buffer the whole list of files before we return anything (in addition to the cost of the sort itself, of course).

          It would be a bit easier to do in globStatus, since we always have an array there (unlike in listStatus where we might just have an iterator), but the same issues crop up. We're slowing stuff down, for a feature most don't need.

          Do users really depend on this behavior, or can we just drop this from the spec? I guess the shell probably wants sorted output, to provide a consistent display. But it can sort it itself, of course. Thoughts?

          Show
          cmccabe Colin P. McCabe added a comment - Daryn Sharp : smart. I was wondering why it worked on HDFS but not on localFS. I don't know how I feel about this JIRA. The obvious solution is to have localFS sort its output on Linux, but this will cripple performance by forcing us to buffer the whole list of files before we return anything (in addition to the cost of the sort itself, of course). It would be a bit easier to do in globStatus, since we always have an array there (unlike in listStatus where we might just have an iterator), but the same issues crop up. We're slowing stuff down, for a feature most don't need. Do users really depend on this behavior, or can we just drop this from the spec? I guess the shell probably wants sorted output, to provide a consistent display. But it can sort it itself, of course. Thoughts?
          Hide
          fborchers Felix Borchers added a comment -

          In my oppinion, dropping this is totally ok. I agree, that this sorting feature may not be used often.

          Show
          fborchers Felix Borchers added a comment - In my oppinion, dropping this is totally ok. I agree, that this sorting feature may not be used often.
          Hide
          cmccabe Colin P. McCabe added a comment -

          I thought about this some more, and I think we should just sort the array returned from globStatus. Otherwise, applications will get inconsistent behavior. Applications can always use the raw listStatus if they don't want to pay the cost of a sort. And the cost of a sort is not that high when you already have everything in an array (like we do with globStatus).

          Show
          cmccabe Colin P. McCabe added a comment - I thought about this some more, and I think we should just sort the array returned from globStatus. Otherwise, applications will get inconsistent behavior. Applications can always use the raw listStatus if they don't want to pay the cost of a sort. And the cost of a sort is not that high when you already have everything in an array (like we do with globStatus).
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12656923/HADOOP-10798.001.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          -1 javac. The applied patch generated 1262 javac compiler warnings (more than the trunk's current 1260 warnings).

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs:

          org.apache.hadoop.ipc.TestIPC
          org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/4368//testReport/
          Javac warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/4368//artifact/trunk/patchprocess/diffJavacWarnings.txt
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/4368//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12656923/HADOOP-10798.001.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. -1 javac . The applied patch generated 1262 javac compiler warnings (more than the trunk's current 1260 warnings). +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.ipc.TestIPC org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/4368//testReport/ Javac warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/4368//artifact/trunk/patchprocess/diffJavacWarnings.txt Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/4368//console This message is automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 14m 36s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          -1 javac 7m 29s The applied patch generated 123 additional warning messages.
          +1 javadoc 9m 37s There were no new javadoc warning messages.
          +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 2m 7s The applied patch generated 2 new checkstyle issues (total was 4, now 5).
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 34s mvn install still works.
          +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse.
          +1 findbugs 4m 41s The patch does not introduce any new Findbugs (version 2.0.3) warnings.
          +1 common tests 23m 35s Tests passed in hadoop-common.
          +1 hdfs tests 165m 1s Tests passed in hadoop-hdfs.
              230m 45s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12656923/HADOOP-10798.001.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 6ae2a0d
          javac https://builds.apache.org/job/PreCommit-HADOOP-Build/6435/artifact/patchprocess/diffJavacWarnings.txt
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/6435/artifact/patchprocess/diffcheckstylehadoop-common.txt
          hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/6435/artifact/patchprocess/testrun_hadoop-common.txt
          hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HADOOP-Build/6435/artifact/patchprocess/testrun_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/6435/testReport/
          Java 1.7.0_55
          uname Linux asf901.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6435/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 14m 36s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. -1 javac 7m 29s The applied patch generated 123 additional warning messages. +1 javadoc 9m 37s There were no new javadoc warning messages. +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 2m 7s The applied patch generated 2 new checkstyle issues (total was 4, now 5). +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 34s mvn install still works. +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse. +1 findbugs 4m 41s The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 common tests 23m 35s Tests passed in hadoop-common. +1 hdfs tests 165m 1s Tests passed in hadoop-hdfs.     230m 45s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12656923/HADOOP-10798.001.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 6ae2a0d javac https://builds.apache.org/job/PreCommit-HADOOP-Build/6435/artifact/patchprocess/diffJavacWarnings.txt checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/6435/artifact/patchprocess/diffcheckstylehadoop-common.txt hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/6435/artifact/patchprocess/testrun_hadoop-common.txt hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HADOOP-Build/6435/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/6435/testReport/ Java 1.7.0_55 uname Linux asf901.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6435/console This message was automatically generated.
          Hide
          jghoman Jakob Homan added a comment -

          I would like to drop the requirement from the API. Since no one is reporting their application is breaking, no one appears to be depending on the output being sorted. It's better to let the client pay the cost of sorting, if they need the output sorted, rather than than impose the cost on all callers. This is particularly true for FileSystem implementations that may not by default return sorted listStatus results.

          Show
          jghoman Jakob Homan added a comment - I would like to drop the requirement from the API. Since no one is reporting their application is breaking, no one appears to be depending on the output being sorted. It's better to let the client pay the cost of sorting, if they need the output sorted, rather than than impose the cost on all callers. This is particularly true for FileSystem implementations that may not by default return sorted listStatus results.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 14m 42s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          -1 javac 7m 30s The applied patch generated 2 additional warning messages.
          +1 javadoc 9m 35s There were no new javadoc warning messages.
          +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 2m 14s The applied patch generated 2 new checkstyle issues (total was 4, now 5).
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 35s mvn install still works.
          +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse.
          +1 findbugs 4m 42s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 common tests 22m 58s Tests passed in hadoop-common.
          +1 hdfs tests 163m 57s Tests passed in hadoop-hdfs.
              229m 20s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12656923/HADOOP-10798.001.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / e422e76
          javac https://builds.apache.org/job/PreCommit-HADOOP-Build/6747/artifact/patchprocess/diffJavacWarnings.txt
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/6747/artifact/patchprocess/diffcheckstylehadoop-common.txt
          hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/6747/artifact/patchprocess/testrun_hadoop-common.txt
          hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HADOOP-Build/6747/artifact/patchprocess/testrun_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/6747/testReport/
          Java 1.7.0_55
          uname Linux asf901.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6747/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 14m 42s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. -1 javac 7m 30s The applied patch generated 2 additional warning messages. +1 javadoc 9m 35s There were no new javadoc warning messages. +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 2m 14s The applied patch generated 2 new checkstyle issues (total was 4, now 5). +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 35s mvn install still works. +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse. +1 findbugs 4m 42s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 common tests 22m 58s Tests passed in hadoop-common. +1 hdfs tests 163m 57s Tests passed in hadoop-hdfs.     229m 20s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12656923/HADOOP-10798.001.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / e422e76 javac https://builds.apache.org/job/PreCommit-HADOOP-Build/6747/artifact/patchprocess/diffJavacWarnings.txt checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/6747/artifact/patchprocess/diffcheckstylehadoop-common.txt hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/6747/artifact/patchprocess/testrun_hadoop-common.txt hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HADOOP-Build/6747/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/6747/testReport/ Java 1.7.0_55 uname Linux asf901.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6747/console This message was automatically generated.
          Hide
          lirui Rui Li added a comment -

          Hi guys, does parallel sorting rely on the returned files being sorted? If the sorted results are in multiple files, we need to read the files in specific order to maintain the global sort right?

          Show
          lirui Rui Li added a comment - Hi guys, does parallel sorting rely on the returned files being sorted? If the sorted results are in multiple files, we need to read the files in specific order to maintain the global sort right?
          Hide
          cmccabe Colin P. McCabe added a comment -

          I don't feel strongly about this either way. If you want to implement the no-sort option, please add sorting to the relevant parts of the shell, though, and post a patch modifying the javadoc.

          Show
          cmccabe Colin P. McCabe added a comment - I don't feel strongly about this either way. If you want to implement the no-sort option, please add sorting to the relevant parts of the shell, though, and post a patch modifying the javadoc.
          Hide
          jghoman Jakob Homan added a comment -

          I'd like to go ahead and remove the sorting language from the API. There's no need to do the sort on shell-side since it's not being done already and hence there should be no behavior change. Sorting, particularly for large or already sorted entries can be costly. If the client really needs it sorted either in the code or from the shell, it's easy for them to do so and pay the cost.

          This is a minor change to the API documentation, so it'd be a good newbie task for my colleague, Shradha Revankar and I'll assign it to her if we're cool with this plan.

          Show
          jghoman Jakob Homan added a comment - I'd like to go ahead and remove the sorting language from the API. There's no need to do the sort on shell-side since it's not being done already and hence there should be no behavior change. Sorting, particularly for large or already sorted entries can be costly. If the client really needs it sorted either in the code or from the shell, it's easy for them to do so and pay the cost. This is a minor change to the API documentation, so it'd be a good newbie task for my colleague, Shradha Revankar and I'll assign it to her if we're cool with this plan.
          Hide
          cmccabe Colin P. McCabe added a comment -

          I'd like to go ahead and remove the sorting language from the API.

          OK

          There's no need to do the sort on shell-side since it's not being done already and hence there should be no behavior change.

          Disagree. "ls" on UNIX has always returned entries in sorted order. So does Hadoop's ls, except in the special case where you are using a non-HDFS filesystem. This is not a normal case, so the discrepancy got overlooked. But I think we should fix it now.

          We don't need to be ultra-fast in the shell, so there seems to be no reason why we shouldn't just add a sort. At least that's my thinking right now. What do you think?

          Show
          cmccabe Colin P. McCabe added a comment - I'd like to go ahead and remove the sorting language from the API. OK There's no need to do the sort on shell-side since it's not being done already and hence there should be no behavior change. Disagree. "ls" on UNIX has always returned entries in sorted order. So does Hadoop's ls, except in the special case where you are using a non-HDFS filesystem. This is not a normal case, so the discrepancy got overlooked. But I think we should fix it now. We don't need to be ultra-fast in the shell, so there seems to be no reason why we shouldn't just add a sort. At least that's my thinking right now. What do you think?
          Hide
          jghoman Jakob Homan added a comment -

          Right now this JIRA is limited to correcting an incorrect javadoc. If we add in changing the output of a reasonably distant command (specifically the shell of liststatus, not just the API of FileSystem.listStatus), that's a pretty big change relative to the original JIRA. I'd be fine with having that discussion (to sort the shell output of ls) in another JIRA. Since this additional change would potentially affect current behavior for non-HDFS FileSystems, as you say, it should probably be considered an incompatible change, which again is beyond the scope of the current JIRA.

          Can we just restrict this one to the API correction and open another to discuss adding sorting?

          Show
          jghoman Jakob Homan added a comment - Right now this JIRA is limited to correcting an incorrect javadoc. If we add in changing the output of a reasonably distant command (specifically the shell of liststatus, not just the API of FileSystem.listStatus), that's a pretty big change relative to the original JIRA. I'd be fine with having that discussion (to sort the shell output of ls) in another JIRA. Since this additional change would potentially affect current behavior for non-HDFS FileSystems, as you say, it should probably be considered an incompatible change, which again is beyond the scope of the current JIRA. Can we just restrict this one to the API correction and open another to discuss adding sorting?
          Hide
          cmccabe Colin P. McCabe added a comment -

          OK, I did a little more research into this. The globStatus code back in 2.0.3-alpha does sort the entries it returned. Around Hadoop 2.3, the sort was lost during the globber rewrite. This was a bug, but it was hidden by the fact that HDFS sorts its listStatus entries (this behavior is undocumented, but 100% consistent).

          Since the API documentation says that sorted entries are returned, and since this is the case for the vast majority of use-cases (i.e. when using Hadoop with HDFS), I think changing this behavior in globStatus would be an incompatible change. Any user code relying on the old documented behavior would break. Let's commit the original patch I posted to fix this situation. If we want to have a discussion about changing the API contract we can have that discussion for Hadoop 3.0 only.

          also I feel that the facts that:
          1. globStatus has historically had a sort in it
          2. users who want to optimize by avoiding a sort can use listStatus

          strongly suggest that changing this behavior is not a good idea, even in 3.x.

          Show
          cmccabe Colin P. McCabe added a comment - OK, I did a little more research into this. The globStatus code back in 2.0.3-alpha does sort the entries it returned. Around Hadoop 2.3, the sort was lost during the globber rewrite. This was a bug, but it was hidden by the fact that HDFS sorts its listStatus entries (this behavior is undocumented, but 100% consistent). Since the API documentation says that sorted entries are returned, and since this is the case for the vast majority of use-cases (i.e. when using Hadoop with HDFS), I think changing this behavior in globStatus would be an incompatible change. Any user code relying on the old documented behavior would break. Let's commit the original patch I posted to fix this situation. If we want to have a discussion about changing the API contract we can have that discussion for Hadoop 3.0 only. also I feel that the facts that: 1. globStatus has historically had a sort in it 2. users who want to optimize by avoiding a sort can use listStatus strongly suggest that changing this behavior is not a good idea, even in 3.x.
          Hide
          jghoman Jakob Homan added a comment -

          OK, if this is a regression rather than an incorrect Javadoc, I'm OK with the sorting. +1 on the patch, assuming javadoc and checkstyle warnings are spurious. I'd like it to go back to 2.6 and 2.7 as well, and will backport it if you don't have the time.

          Show
          jghoman Jakob Homan added a comment - OK, if this is a regression rather than an incorrect Javadoc, I'm OK with the sorting. +1 on the patch, assuming javadoc and checkstyle warnings are spurious. I'd like it to go back to 2.6 and 2.7 as well, and will backport it if you don't have the time.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 19m 36s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          -1 javac 7m 34s The applied patch generated 2 additional warning messages.
          +1 javadoc 9m 34s There were no new javadoc warning messages.
          +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 2m 14s The applied patch generated 2 new checkstyle issues (total was 4, now 5).
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 34s mvn install still works.
          +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
          +1 findbugs 4m 59s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 common tests 21m 56s Tests passed in hadoop-common.
          +1 hdfs tests 159m 17s Tests passed in hadoop-hdfs.
              228m 57s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12656923/HADOOP-10798.001.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 5b5bb8d
          javac https://builds.apache.org/job/PreCommit-HADOOP-Build/6991/artifact/patchprocess/diffJavacWarnings.txt
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/6991/artifact/patchprocess/diffcheckstylehadoop-common.txt
          hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/6991/artifact/patchprocess/testrun_hadoop-common.txt
          hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HADOOP-Build/6991/artifact/patchprocess/testrun_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/6991/testReport/
          Java 1.7.0_55
          uname Linux asf907.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6991/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 19m 36s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. -1 javac 7m 34s The applied patch generated 2 additional warning messages. +1 javadoc 9m 34s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 2m 14s The applied patch generated 2 new checkstyle issues (total was 4, now 5). +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 34s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 4m 59s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 common tests 21m 56s Tests passed in hadoop-common. +1 hdfs tests 159m 17s Tests passed in hadoop-hdfs.     228m 57s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12656923/HADOOP-10798.001.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 5b5bb8d javac https://builds.apache.org/job/PreCommit-HADOOP-Build/6991/artifact/patchprocess/diffJavacWarnings.txt checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/6991/artifact/patchprocess/diffcheckstylehadoop-common.txt hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/6991/artifact/patchprocess/testrun_hadoop-common.txt hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HADOOP-Build/6991/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/6991/testReport/ Java 1.7.0_55 uname Linux asf907.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6991/console This message was automatically generated.
          Hide
          cmccabe Colin P. McCabe added a comment -

          Thanks, Jakob Homan. It's good to have this finally fixed. Committed to 2.8

          Show
          cmccabe Colin P. McCabe added a comment - Thanks, Jakob Homan . It's good to have this finally fixed. Committed to 2.8
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #8101 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8101/)
          HADOOP-10798. globStatus() should always return a sorted list of files (cmccabe) (cmccabe: rev 68e588cbee660d55dba518892d064bee3795a002)

          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/Globber.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/TestGlobPaths.java
          • hadoop-common-project/hadoop-common/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #8101 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8101/ ) HADOOP-10798 . globStatus() should always return a sorted list of files (cmccabe) (cmccabe: rev 68e588cbee660d55dba518892d064bee3795a002) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/Globber.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/TestGlobPaths.java hadoop-common-project/hadoop-common/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk #975 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/975/)
          HADOOP-10798. globStatus() should always return a sorted list of files (cmccabe) (cmccabe: rev 68e588cbee660d55dba518892d064bee3795a002)

          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/TestGlobPaths.java
          • hadoop-common-project/hadoop-common/CHANGES.txt
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/Globber.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #975 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/975/ ) HADOOP-10798 . globStatus() should always return a sorted list of files (cmccabe) (cmccabe: rev 68e588cbee660d55dba518892d064bee3795a002) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/TestGlobPaths.java hadoop-common-project/hadoop-common/CHANGES.txt hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/Globber.java
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Yarn-trunk-Java8 #245 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/245/)
          HADOOP-10798. globStatus() should always return a sorted list of files (cmccabe) (cmccabe: rev 68e588cbee660d55dba518892d064bee3795a002)

          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/TestGlobPaths.java
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/Globber.java
          • hadoop-common-project/hadoop-common/CHANGES.txt
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Yarn-trunk-Java8 #245 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/245/ ) HADOOP-10798 . globStatus() should always return a sorted list of files (cmccabe) (cmccabe: rev 68e588cbee660d55dba518892d064bee3795a002) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/TestGlobPaths.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/Globber.java hadoop-common-project/hadoop-common/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk #2172 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2172/)
          HADOOP-10798. globStatus() should always return a sorted list of files (cmccabe) (cmccabe: rev 68e588cbee660d55dba518892d064bee3795a002)

          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/Globber.java
          • hadoop-common-project/hadoop-common/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/TestGlobPaths.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2172 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2172/ ) HADOOP-10798 . globStatus() should always return a sorted list of files (cmccabe) (cmccabe: rev 68e588cbee660d55dba518892d064bee3795a002) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/Globber.java hadoop-common-project/hadoop-common/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/TestGlobPaths.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #233 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/233/)
          HADOOP-10798. globStatus() should always return a sorted list of files (cmccabe) (cmccabe: rev 68e588cbee660d55dba518892d064bee3795a002)

          • hadoop-common-project/hadoop-common/CHANGES.txt
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/Globber.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/TestGlobPaths.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #233 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/233/ ) HADOOP-10798 . globStatus() should always return a sorted list of files (cmccabe) (cmccabe: rev 68e588cbee660d55dba518892d064bee3795a002) hadoop-common-project/hadoop-common/CHANGES.txt hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/Globber.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/TestGlobPaths.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk #2191 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2191/)
          HADOOP-10798. globStatus() should always return a sorted list of files (cmccabe) (cmccabe: rev 68e588cbee660d55dba518892d064bee3795a002)

          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/TestGlobPaths.java
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/Globber.java
          • hadoop-common-project/hadoop-common/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2191 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2191/ ) HADOOP-10798 . globStatus() should always return a sorted list of files (cmccabe) (cmccabe: rev 68e588cbee660d55dba518892d064bee3795a002) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/TestGlobPaths.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/Globber.java hadoop-common-project/hadoop-common/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Mapreduce-trunk-Java8 #243 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/243/)
          HADOOP-10798. globStatus() should always return a sorted list of files (cmccabe) (cmccabe: rev 68e588cbee660d55dba518892d064bee3795a002)

          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/Globber.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/TestGlobPaths.java
          • hadoop-common-project/hadoop-common/CHANGES.txt
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Mapreduce-trunk-Java8 #243 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/243/ ) HADOOP-10798 . globStatus() should always return a sorted list of files (cmccabe) (cmccabe: rev 68e588cbee660d55dba518892d064bee3795a002) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/Globber.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/TestGlobPaths.java hadoop-common-project/hadoop-common/CHANGES.txt

            People

            • Assignee:
              cmccabe Colin P. McCabe
              Reporter:
              fborchers Felix Borchers
            • Votes:
              1 Vote for this issue
              Watchers:
              12 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development