Hadoop Common
  1. Hadoop Common
  2. HADOOP-6467

Performance improvement for liststatus on directories in hadoop archives.

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.21.0
    • Component/s: fs
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      A liststatus call on a directory in hadoop archives leads to ( 2* number of files in directory) open calls to the namenode. This is very sub optimal and needs to be fixed to make it performant enough to be used on a daily basis.

      1. Archives_performance.docx
        111 kB
        Mahadev konar
      2. Archives_performance.docx
        94 kB
        Mahadev konar
      3. HADOOP-6467_v3.patch
        4 kB
        Mahadev konar
      4. HADOOP-6467.patch
        4 kB
        Mahadev konar
      5. HADOOP-6467.patch
        7 kB
        Mahadev konar
      6. HADOOP-6467.patch
        6 kB
        Mahadev konar
      7. HADOOP-6467-v2.patch
        4 kB
        Mahadev konar
      8. HADOOP-6467-y.0.20-branch-v2.patch
        5 kB
        Mahadev konar
      9. HADOOP-6467-y.0.20-branch-v2.patch
        4 kB
        Mahadev konar
      10. HADOOP-6467-y0.20-branch.patch
        4 kB
        Mahadev konar

        Issue Links

          Activity

          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk #258 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk/258/)
          . Improve the performance on HarFileSystem.listStatus(..). Contributed by mahadev

          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk #258 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk/258/ ) . Improve the performance on HarFileSystem.listStatus(..). Contributed by mahadev
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #181 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk-Commit/181/)
          . Improve the performance on HarFileSystem.listStatus(..). Contributed by mahadev

          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #181 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk-Commit/181/ ) . Improve the performance on HarFileSystem.listStatus(..). Contributed by mahadev
          Hide
          Tsz Wo Nicholas Sze added a comment -

          No new tests added. We tested manually and compared the performance as shown before.

          I have committed this. Thanks, Mahadev!

          Show
          Tsz Wo Nicholas Sze added a comment - No new tests added. We tested manually and compared the performance as shown before. I have committed this. Thanks, Mahadev!
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12436656/HADOOP-6467_v3.patch
          against trunk revision 915097.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/369/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/369/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/369/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/369/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12436656/HADOOP-6467_v3.patch against trunk revision 915097. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/369/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/369/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/369/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/369/console This message is automatically generated.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          +1 Got some better numbers. The v3 patch is good to go.

          "-bash-3.1$ date; time $H ${WC_CMD} ${HAR_FULL}/${DIR} ${TT_WC}4
          Tue Feb 23 02:17:18 UTC 2010
          10/02/23 02:17:32 INFO input.FileInputFormat: Total input paths to process : 100000
          10/02/23 02:27:39 INFO mapred.JobClient: Running job: job_201002042035_76681
          10/02/23 02:27:40 INFO mapred.JobClient:  map 0% reduce 0%
          ...
          10/02/23 02:32:35 INFO mapred.JobClient:  map 100% reduce 100%
          10/02/23 02:32:41 INFO mapred.JobClient: Job complete: job_201002042035_76681
          ...
          10/02/23 02:32:42 INFO mapred.JobClient:     Reduce input records=15660
          
          real    15m23.717s
          user    2m22.153s
          sys     0m39.487s
          
          Show
          Tsz Wo Nicholas Sze added a comment - +1 Got some better numbers. The v3 patch is good to go. "-bash-3.1$ date; time $H ${WC_CMD} ${HAR_FULL}/${DIR} ${TT_WC}4 Tue Feb 23 02:17:18 UTC 2010 10/02/23 02:17:32 INFO input.FileInputFormat: Total input paths to process : 100000 10/02/23 02:27:39 INFO mapred.JobClient: Running job: job_201002042035_76681 10/02/23 02:27:40 INFO mapred.JobClient: map 0% reduce 0% ... 10/02/23 02:32:35 INFO mapred.JobClient: map 100% reduce 100% 10/02/23 02:32:41 INFO mapred.JobClient: Job complete: job_201002042035_76681 ... 10/02/23 02:32:42 INFO mapred.JobClient: Reduce input records=15660 real 15m23.717s user 2m22.153s sys 0m39.487s
          Hide
          Mahadev konar added a comment -

          the most recent trunk patch with similar changes mentioned above!

          Show
          Mahadev konar added a comment - the most recent trunk patch with similar changes mentioned above!
          Hide
          Mahadev konar added a comment -

          this patch has minor fixes to it

          • changed Path for every index entry to be only created if the prefix matching happens
          • changed depth of harPath to be calculated once
          • changed formatting to fit 80 characters
          Show
          Mahadev konar added a comment - this patch has minor fixes to it changed Path for every index entry to be only created if the prefix matching happens changed depth of harPath to be calculated once changed formatting to fit 80 characters
          Hide
          Tsz Wo Nicholas Sze added a comment -

          I ran wordcount with the v2 patch twice. Both took 16 mins. Not sure why it ran slower than the previous patch.

          -bash-3.1$ date; time $H ${WC_CMD} ${HAR_FULL}/${DIR} ${TT_WC}3 Mon Feb 22 23:23:03 UTC 2010
          10/02/22 23:23:19 INFO input.FileInputFormat: Total input paths to process : 100000
          10/02/22 23:33:53 INFO mapred.JobClient: Running job: job_201002042035_75937
          10/02/22 23:33:54 INFO mapred.JobClient:  map 0% reduce 0% ...
          10/02/22 23:39:08 INFO mapred.JobClient:  map 100% reduce 100% ...
          10/02/22 23:39:14 INFO mapred.JobClient:     Reduce input records=17729
          
          real    16m11.171s
          user    2m24.217s
          sys     0m39.462s
          

          Anyway, below are some improvement suggestions:

          • use a variable to store harPath.depth() so that the same value won't be computed again and again.
          • Unfortunately, Path is quite expensive (see HADOOP-6532). It is better to check child.startsWith(parentString) before creating thisPath. i.e. replace
            +        String child = lineFeed.substring(0, lineFeed.indexOf(" "));
            +        Path thisPath = new Path(child);
            +        if ((child.startsWith(parentString)) && (thisPath.depth() == harPath.depth() + 1)) {
            +          ...
            

            with

            +        String child = lineFeed.substring(0, lineFeed.indexOf(" "));
            +        if (child.startsWith(parentString)) {
            +          Path thisPath = new Path(child);
            +          if (thisPath.depth() == harPath.depth() + 1) {
            +            ...
            
          • Also, there are lines more than 80 characters.
          Show
          Tsz Wo Nicholas Sze added a comment - I ran wordcount with the v2 patch twice. Both took 16 mins. Not sure why it ran slower than the previous patch. -bash-3.1$ date; time $H ${WC_CMD} ${HAR_FULL}/${DIR} ${TT_WC}3 Mon Feb 22 23:23:03 UTC 2010 10/02/22 23:23:19 INFO input.FileInputFormat: Total input paths to process : 100000 10/02/22 23:33:53 INFO mapred.JobClient: Running job: job_201002042035_75937 10/02/22 23:33:54 INFO mapred.JobClient: map 0% reduce 0% ... 10/02/22 23:39:08 INFO mapred.JobClient: map 100% reduce 100% ... 10/02/22 23:39:14 INFO mapred.JobClient: Reduce input records=17729 real 16m11.171s user 2m24.217s sys 0m39.462s Anyway, below are some improvement suggestions: use a variable to store harPath.depth() so that the same value won't be computed again and again. Unfortunately, Path is quite expensive (see HADOOP-6532 ). It is better to check child.startsWith(parentString) before creating thisPath. i.e. replace + String child = lineFeed.substring(0, lineFeed.indexOf( " " )); + Path thisPath = new Path(child); + if ((child.startsWith(parentString)) && (thisPath.depth() == harPath.depth() + 1)) { + ... with + String child = lineFeed.substring(0, lineFeed.indexOf( " " )); + if (child.startsWith(parentString)) { + Path thisPath = new Path(child); + if (thisPath.depth() == harPath.depth() + 1) { + ... Also, there are lines more than 80 characters.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          Created HADOOP-6591 for the space problem.

          Show
          Tsz Wo Nicholas Sze added a comment - Created HADOOP-6591 for the space problem.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12436637/HADOOP-6467-v2.patch
          against trunk revision 915097.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/367/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/367/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/367/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/367/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12436637/HADOOP-6467-v2.patch against trunk revision 915097. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/367/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/367/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/367/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/367/console This message is automatically generated.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          +1 the v2 patch looks good.

          I will run wordcount on it to check the performance.

          Show
          Tsz Wo Nicholas Sze added a comment - +1 the v2 patch looks good. I will run wordcount on it to check the performance.
          Hide
          Mahadev konar added a comment -

          patch for trunk addressing nicholas's comments.

          Show
          Mahadev konar added a comment - patch for trunk addressing nicholas's comments.
          Hide
          Mahadev konar added a comment -

          this is an updated patch addressing comments by nicholas. This patch is for the 0.20 branch. Ill upload the patch for trunk next.

          Show
          Mahadev konar added a comment - this is an updated patch addressing comments by nicholas. This patch is for the 0.20 branch. Ill upload the patch for trunk next.
          Hide
          Mahadev konar added a comment -

          good points nicholas.

          for the space issue, you are right, ill try and fix that issue in a seperate jira! good catch!

          Show
          Mahadev konar added a comment - good points nicholas. for the space issue, you are right, ill try and fix that issue in a seperate jira! good catch!
          Hide
          Tsz Wo Nicholas Sze added a comment -
          • Should we have a try-finally on aIn/aLin to close the file? Otherwise, the file will remain open in case of an IOException.
          • The codes below are expensive since only parsed[0] is needed. We could avoid regular expression by lineFeed.indexOf(" ") in finding parsed[0].
            +        String[] parsed = lineFeed.split(" ");
            +        Path thisPath = new Path(parsed[0]);
            +        if ((parsed[0].startsWith(parentString)) && (thisPath.depth() == harPath.depth() + 1)) {
            
          • BTW, what if there is a " " in a path? Would har work?? If not, we should fix it or document it.
          Show
          Tsz Wo Nicholas Sze added a comment - Should we have a try-finally on aIn/aLin to close the file? Otherwise, the file will remain open in case of an IOException. The codes below are expensive since only parsed [0] is needed. We could avoid regular expression by lineFeed.indexOf(" ") in finding parsed [0] . + String [] parsed = lineFeed.split( " " ); + Path thisPath = new Path(parsed[0]); + if ((parsed[0].startsWith(parentString)) && (thisPath.depth() == harPath.depth() + 1)) { BTW, what if there is a " " in a path? Would har work?? If not, we should fix it or document it.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12436413/HADOOP-6467.patch
          against trunk revision 912056.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/18/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/18/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/18/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/18/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12436413/HADOOP-6467.patch against trunk revision 912056. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/18/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/18/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/18/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/18/console This message is automatically generated.
          Hide
          Mahadev konar added a comment -

          the patch for trunk.

          Show
          Mahadev konar added a comment - the patch for trunk.
          Hide
          Mahadev konar added a comment -

          renamed the file for clarity.

          Show
          Mahadev konar added a comment - renamed the file for clarity.
          Hide
          Mahadev konar added a comment -

          a clean patch that applies to the 0.20 branch. I am uploading a clean patch for the trunk next.

          Show
          Mahadev konar added a comment - a clean patch that applies to the 0.20 branch. I am uploading a clean patch for the trunk next.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          The new patch is great! It brought down the time to 11mins.

          -bash-3.1$ date; time $H ${WC_CMD} ${HAR_FULL}/${DIR} ${TT_WC}3
          Fri Feb 19 18:12:01 UTC 2010
          10/02/19 18:12:15 INFO input.FileInputFormat: Total input paths to process : 100000
          ...
          10/02/19 18:18:04 INFO mapred.JobClient: Running job: job_201002042035_60236
          10/02/19 18:18:05 INFO mapred.JobClient:  map 0% reduce 0%
          ...
          10/02/19 18:22:53 INFO mapred.JobClient:  map 100% reduce 100%
          ...
          10/02/19 18:23:00 INFO mapred.JobClient:     Reduce input records=19137
          
          real    11m0.423s
          user    1m59.595s
          sys     0m26.681s
          
          Show
          Tsz Wo Nicholas Sze added a comment - The new patch is great! It brought down the time to 11mins. -bash-3.1$ date; time $H ${WC_CMD} ${HAR_FULL}/${DIR} ${TT_WC}3 Fri Feb 19 18:12:01 UTC 2010 10/02/19 18:12:15 INFO input.FileInputFormat: Total input paths to process : 100000 ... 10/02/19 18:18:04 INFO mapred.JobClient: Running job: job_201002042035_60236 10/02/19 18:18:05 INFO mapred.JobClient: map 0% reduce 0% ... 10/02/19 18:22:53 INFO mapred.JobClient: map 100% reduce 100% ... 10/02/19 18:23:00 INFO mapred.JobClient: Reduce input records=19137 real 11m0.423s user 1m59.595s sys 0m26.681s
          Hide
          Mahadev konar added a comment -

          can you try this patch out nicholas? The file split generation was taking a lot of time, i made it to return fake blocks to make it faster. The only implication of this change is that the map reduce tasks might run on non local machines, which I dont think should be a lot of performance impact since it anway accesses multiple files (index and parts) which cannot be on the same host.

          Show
          Mahadev konar added a comment - can you try this patch out nicholas? The file split generation was taking a lot of time, i made it to return fake blocks to make it faster. The only implication of this change is that the map reduce tasks might run on non local machines, which I dont think should be a lot of performance impact since it anway accesses multiple files (index and parts) which cannot be on the same host.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          > nicholas, what is happening between 00:18:22 and 00:46:52. Why is the job launching taking so much time?

          These are non-trivial questions to me. Let me take a look.

          Show
          Tsz Wo Nicholas Sze added a comment - > nicholas, what is happening between 00:18:22 and 00:46:52. Why is the job launching taking so much time? These are non-trivial questions to me. Let me take a look.
          Hide
          Mahadev konar added a comment -
          10/02/14 00:18:22 INFO input.FileInputFormat: Total input paths to process : 100000
          ...
          10/02/14 00:46:52 INFO mapred.JobClient: Running job: job_201002042035_33691
          10/02/14 00:46:53 INFO mapred.JobClient: map 0% reduce 0%
          

          nicholas, what is happening between 00:18:22 and 00:46:52. Why is the job launching taking so much time?

          Show
          Mahadev konar added a comment - 10/02/14 00:18:22 INFO input.FileInputFormat: Total input paths to process : 100000 ... 10/02/14 00:46:52 INFO mapred.JobClient: Running job: job_201002042035_33691 10/02/14 00:46:53 INFO mapred.JobClient: map 0% reduce 0% nicholas, what is happening between 00:18:22 and 00:46:52. Why is the job launching taking so much time?
          Hide
          Tsz Wo Nicholas Sze added a comment -

          The patch improved first part a lot.

          Without the patch, it took ~25 mins to show

          10/02/06 00:07:17 INFO input.FileInputFormat: Total input paths to process : 100000
          

          With the patch, it only took 14 seconds.

          Sun Feb 14 00:18:08 UTC 2010
          10/02/14 00:18:22 INFO input.FileInputFormat: Total input paths to process : 100000
          
          Show
          Tsz Wo Nicholas Sze added a comment - The patch improved first part a lot. Without the patch , it took ~25 mins to show 10/02/06 00:07:17 INFO input.FileInputFormat: Total input paths to process : 100000 With the patch, it only took 14 seconds. Sun Feb 14 00:18:08 UTC 2010 10/02/14 00:18:22 INFO input.FileInputFormat: Total input paths to process : 100000
          Hide
          Mahadev konar added a comment -

          nicholas, just to verify, are you sure you applied the patch? 30 minutes of liststatus should not happen with this patch!

          Show
          Mahadev konar added a comment - nicholas, just to verify, are you sure you applied the patch? 30 minutes of liststatus should not happen with this patch!
          Hide
          Mahadev konar added a comment -

          never mind, i looked at the results you posted, looks like listatus is running for 30 minutes, which seems pretty huge.... let me experiment and see what numbers I am getting.

          Show
          Mahadev konar added a comment - never mind, i looked at the results you posted, looks like listatus is running for 30 minutes, which seems pretty huge.... let me experiment and see what numbers I am getting.
          Hide
          Mahadev konar added a comment -

          nicholas,
          these numbers still seem pretty huge. I was getting subsecond lookups for single files and around max 2-3 seconds for 10K directory file. Is it possible that you are running on a shared cluster and there could be job scheduling delays in the HAR runs and less delays in the non HAR runs? (i mean scheduling of maps could be delayed in case the cluster is shared) ....

          Show
          Mahadev konar added a comment - nicholas, these numbers still seem pretty huge. I was getting subsecond lookups for single files and around max 2-3 seconds for 10K directory file. Is it possible that you are running on a shared cluster and there could be job scheduling delays in the HAR runs and less delays in the non HAR runs? (i mean scheduling of maps could be delayed in case the cluster is shared) ....
          Hide
          Tsz Wo Nicholas Sze added a comment -

          It is better: ~35mins.

          • WordCount on har files with the patch
            -bash-3.1$ date; time $H ${WC_CMD} ${HAR_FULL}/${DIR} ${TT_WC}2
            Sun Feb 14 00:18:08 UTC 2010
            10/02/14 00:18:22 INFO input.FileInputFormat: Total input paths to process : 100000
            ...
            10/02/14 00:46:52 INFO mapred.JobClient: Running job: job_201002042035_33691
            10/02/14 00:46:53 INFO mapred.JobClient: map 0% reduce 0%
            ...
            10/02/14 00:53:22 INFO mapred.JobClient: map 100% reduce 100%
            10/02/14 00:53:28 INFO mapred.JobClient: Job complete: job_201002042035_33691
            ...
            10/02/14 00:53:29 INFO mapred.JobClient: Reduce input records=16665
            
            real 35m21.704s
            user 11m46.324s
            sys 2m34.337s
            
          Show
          Tsz Wo Nicholas Sze added a comment - It is better: ~35mins. WordCount on har files with the patch -bash-3.1$ date; time $H ${WC_CMD} ${HAR_FULL}/${DIR} ${TT_WC}2 Sun Feb 14 00:18:08 UTC 2010 10/02/14 00:18:22 INFO input.FileInputFormat: Total input paths to process : 100000 ... 10/02/14 00:46:52 INFO mapred.JobClient: Running job: job_201002042035_33691 10/02/14 00:46:53 INFO mapred.JobClient: map 0% reduce 0% ... 10/02/14 00:53:22 INFO mapred.JobClient: map 100% reduce 100% 10/02/14 00:53:28 INFO mapred.JobClient: Job complete: job_201002042035_33691 ... 10/02/14 00:53:29 INFO mapred.JobClient: Reduce input records=16665 real 35m21.704s user 11m46.324s sys 2m34.337s
          Hide
          Tsz Wo Nicholas Sze added a comment -
              [javac] ...\src\tools\org\apache\hadoop\tools\HadoopArchives.java:492: cannot find symbol
              [javac] symbol  : class FsPermission
              [javac] location: class org.apache.hadoop.tools.HadoopArchives.HArchivesMapper
              [javac]         partStream = destFs.create(tmpOutput, new FsPermission((short)0700),
          
          Show
          Tsz Wo Nicholas Sze added a comment - [javac] ...\src\tools\org\apache\hadoop\tools\HadoopArchives.java:492: cannot find symbol [javac] symbol : class FsPermission [javac] location: class org.apache.hadoop.tools.HadoopArchives.HArchivesMapper [javac] partStream = destFs.create(tmpOutput, new FsPermission((short)0700),
          Hide
          Mahadev konar added a comment -

          this patch fixes the issue of slow liststatus. I ran it on my machine and it was able to run liststaus of a 10K directory file in 3-4 seconds. This patch does not implement the proposal attached to this jira but does a simple brute force of reading the whole index file to find all the children of a directory. I tried the approach that I had mentioned in the proposal but found that it just complicates the code a little bit (to maintain backwards compatibility), so I tried doing the brute force way, which turns out to be fast enough for daily usage of har filesystem by map reduce jobs.

          nicholas can you try this out and post the numbers with the patch?

          thanks

          Show
          Mahadev konar added a comment - this patch fixes the issue of slow liststatus. I ran it on my machine and it was able to run liststaus of a 10K directory file in 3-4 seconds. This patch does not implement the proposal attached to this jira but does a simple brute force of reading the whole index file to find all the children of a directory. I tried the approach that I had mentioned in the proposal but found that it just complicates the code a little bit (to maintain backwards compatibility), so I tried doing the brute force way, which turns out to be fast enough for daily usage of har filesystem by map reduce jobs. nicholas can you try this out and post the numbers with the patch? thanks
          Hide
          Mahadev konar added a comment -

          thanks on the numbers nicholas. Ill try fixing this soon....

          Show
          Mahadev konar added a comment - thanks on the numbers nicholas. Ill try fixing this soon....
          Hide
          Tsz Wo Nicholas Sze added a comment -

          Ran WordCount on 10^5 files. The one on har took ~1 hour while the one on hdfs took ~5 mins.

          • WordCount on har files
            -bash-3.1$ time hadoop ${WC_CMD} ${HAR_FULL}/${DIR} ${TT_WC}
            10/02/06 00:07:17 INFO input.FileInputFormat: Total input paths to process : 100000
            10/02/06 00:07:17 INFO lzo.GPLNativeCodeLoader: Loaded native gpl library
            10/02/06 00:07:17 INFO lzo.LzoCodec: Successfully loaded & initialized native-lzo library
            10/02/06 00:35:28 INFO mapred.JobClient: Running job: job_201002042035_5217
            10/02/06 00:35:29 INFO mapred.JobClient:  map 0% reduce 0%
            ...
            10/02/06 00:39:26 INFO mapred.JobClient:  map 99% reduce 31%
            ...
            10/02/06 00:39:49 INFO mapred.JobClient:     Reduce input records=17193
            
            real    57m26.291s
            user    21m49.516s
            sys     4m58.084s
            
          • WordCount on hdfs files
            -bash-3.1$ time hadoop ${WC_CMD} ${DIR} ${WC_DIR}
            10/02/06 01:03:44 INFO input.FileInputFormat: Total input paths to process : 100000
            10/02/06 01:03:44 INFO lzo.GPLNativeCodeLoader: Loaded native gpl library
            10/02/06 01:03:44 INFO lzo.LzoCodec: Successfully loaded & initialized native-lzo library
            10/02/06 01:04:53 INFO mapred.JobClient: Running job: job_201002042035_5628
            10/02/06 01:04:54 INFO mapred.JobClient:  map 0% reduce 0%
            ...
            10/02/06 01:08:26 INFO mapred.JobClient:  map 100% reduce 100%
            ...
            
            10/02/06 01:08:30 INFO mapred.JobClient:     Reduce input records=21949
            
            real    4m54.531s
            user    0m35.346s
            sys     0m4.876s
            
          Show
          Tsz Wo Nicholas Sze added a comment - Ran WordCount on 10^5 files. The one on har took ~1 hour while the one on hdfs took ~5 mins. WordCount on har files -bash-3.1$ time hadoop ${WC_CMD} ${HAR_FULL}/${DIR} ${TT_WC} 10/02/06 00:07:17 INFO input.FileInputFormat: Total input paths to process : 100000 10/02/06 00:07:17 INFO lzo.GPLNativeCodeLoader: Loaded native gpl library 10/02/06 00:07:17 INFO lzo.LzoCodec: Successfully loaded & initialized native-lzo library 10/02/06 00:35:28 INFO mapred.JobClient: Running job: job_201002042035_5217 10/02/06 00:35:29 INFO mapred.JobClient: map 0% reduce 0% ... 10/02/06 00:39:26 INFO mapred.JobClient: map 99% reduce 31% ... 10/02/06 00:39:49 INFO mapred.JobClient: Reduce input records=17193 real 57m26.291s user 21m49.516s sys 4m58.084s WordCount on hdfs files -bash-3.1$ time hadoop ${WC_CMD} ${DIR} ${WC_DIR} 10/02/06 01:03:44 INFO input.FileInputFormat: Total input paths to process : 100000 10/02/06 01:03:44 INFO lzo.GPLNativeCodeLoader: Loaded native gpl library 10/02/06 01:03:44 INFO lzo.LzoCodec: Successfully loaded & initialized native-lzo library 10/02/06 01:04:53 INFO mapred.JobClient: Running job: job_201002042035_5628 10/02/06 01:04:54 INFO mapred.JobClient: map 0% reduce 0% ... 10/02/06 01:08:26 INFO mapred.JobClient: map 100% reduce 100% ... 10/02/06 01:08:30 INFO mapred.JobClient: Reduce input records=21949 real 4m54.531s user 0m35.346s sys 0m4.876s
          Hide
          Mahadev konar added a comment -

          An updated doc that does a much better job of explaining the current architecture of hadoop archives and also points out the performance drawback and the solution for which this jira has been created. comments are welcome.

          Show
          Mahadev konar added a comment - An updated doc that does a much better job of explaining the current architecture of hadoop archives and also points out the performance drawback and the solution for which this jira has been created. comments are welcome.
          Hide
          Mahadev konar added a comment -

          doug,
          most of the metadata that is used in archives is plain text. For later uses when we have more advanced archives avro would surely be suitable. For this jira, the goal is to make archives performant and be usable on a regular basis by users.

          Show
          Mahadev konar added a comment - doug, most of the metadata that is used in archives is plain text. For later uses when we have more advanced archives avro would surely be suitable. For this jira, the goal is to make archives performant and be usable on a regular basis by users.
          Hide
          Doug Cutting added a comment -

          Might we use Avro for these files? That way we'll be more easily able to write non-Java archive readers.

          Show
          Doug Cutting added a comment - Might we use Avro for these files? That way we'll be more easily able to write non-Java archive readers.
          Hide
          Hong Tang added a comment -

          Some general comments on the document:

          • Should mention that the outputDirectory must end with ".har".
          • Can we do away with the file names for a directory in the index file now that all files under the same directory should be clustered in a contiguous region in the index file?
          • How is FileStatus serialized? Assuming you use FileStatus.Writable, then we need to make sure that FileStatus serialization format being kept stable.
          • Should explicitly mention that forward compatibility is not maintained.
          Show
          Hong Tang added a comment - Some general comments on the document: Should mention that the outputDirectory must end with ".har". Can we do away with the file names for a directory in the index file now that all files under the same directory should be clustered in a contiguous region in the index file? How is FileStatus serialized? Assuming you use FileStatus.Writable, then we need to make sure that FileStatus serialization format being kept stable. Should explicitly mention that forward compatibility is not maintained.
          Hide
          Mahadev konar added a comment -

          attached document on how archives is implemented currently and what improvements need to be made. please take a look. Feedback is welcome.

          Show
          Mahadev konar added a comment - attached document on how archives is implemented currently and what improvements need to be made. please take a look. Feedback is welcome.

            People

            • Assignee:
              Mahadev konar
              Reporter:
              Mahadev konar
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development