Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.21.0
    • Component/s: harchive
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      -bash-3.1$ hadoop  archive -archiveName t4.har -p . t4 .
      Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
              at java.util.regex.Pattern.compile(Pattern.java:1432)
              at java.util.regex.Pattern.<init>(Pattern.java:1133)
              at java.util.regex.Pattern.compile(Pattern.java:847)
              at java.lang.String.replace(String.java:2208)
              at org.apache.hadoop.fs.Path.normalizePath(Path.java:146)
              at org.apache.hadoop.fs.Path.initialize(Path.java:137)
              at org.apache.hadoop.fs.Path.<init>(Path.java:126)
              at org.apache.hadoop.fs.Path.makeQualified(Path.java:296)
              at org.apache.hadoop.hdfs.DistributedFileSystem.makeQualified(DistributedFileSystem.java:244)
              at org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:256)
              at org.apache.hadoop.tools.HadoopArchives.archive(HadoopArchives.java:393)
              at org.apache.hadoop.tools.HadoopArchives.run(HadoopArchives.java:736)
              at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
              at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
              at org.apache.hadoop.tools.HadoopArchives.main(HadoopArchives.java:751)
      
      1. m1425_20100129TextFileGenerator.patch
        10 kB
        Tsz Wo Nicholas Sze
      2. har.sh
        2 kB
        Tsz Wo Nicholas Sze
      3. MAPREDUCE-1425.patch
        2 kB
        Mahadev konar
      4. MAPREDUCE-1425.patch
        5 kB
        Mahadev konar
      5. MAPREDUCE-1425_y_0.20.patch
        4 kB
        Mahadev konar
      6. MAPREDUCE-1425.patch
        4 kB
        Mahadev konar

        Activity

        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #232 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/232/)

        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #232 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/232/ )
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk-Commit #233 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/233/)

        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #233 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/233/ )
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12435033/MAPREDUCE-1425.patch
        against trunk revision 907967.

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed core unit tests.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/439/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/439/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/439/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/439/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12435033/MAPREDUCE-1425.patch against trunk revision 907967. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/439/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/439/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/439/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/439/console This message is automatically generated.
        Hide
        Tsz Wo Nicholas Sze added a comment -

        I have committed this. Thanks, Mahadev!

        Show
        Tsz Wo Nicholas Sze added a comment - I have committed this. Thanks, Mahadev!
        Hide
        Tsz Wo Nicholas Sze added a comment -

        The manual test is simple: run archive on 10^5 files and jmap to read the memory usages as shown previously.

        Show
        Tsz Wo Nicholas Sze added a comment - The manual test is simple: run archive on 10^5 files and jmap to read the memory usages as shown previously.
        Hide
        Tsz Wo Nicholas Sze added a comment -

        All tests passed except TestChainErrors, which still failed after the patch had been reverted.

        Show
        Tsz Wo Nicholas Sze added a comment - All tests passed except TestChainErrors, which still failed after the patch had been reverted.
        Hide
        Tsz Wo Nicholas Sze added a comment -

        Hudson does not seem working. Ran test-patch locally.

             [exec] -1 overall.  
             [exec] 
             [exec]     +1 @author.  The patch does not contain any @author tags.
             [exec] 
             [exec]     -1 tests included.  The patch doesn't appear to include any new or modified tests.
             [exec]                         Please justify why no new tests are needed for this patch.
             [exec]                         Also please list what manual steps were performed to verify this patch.
             [exec] 
             [exec]     +1 javadoc.  The javadoc tool did not generate any warning messages.
             [exec] 
             [exec]     +1 javac.  The applied patch does not increase the total number of javac compiler warnings.
             [exec] 
             [exec]     +1 findbugs.  The patch does not introduce any new Findbugs warnings.
             [exec] 
             [exec]     +1 release audit.  The applied patch does not increase the total number of release audit warnings.
             [exec]
        
        Show
        Tsz Wo Nicholas Sze added a comment - Hudson does not seem working. Ran test-patch locally. [exec] -1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] -1 tests included. The patch doesn't appear to include any new or modified tests. [exec] Please justify why no new tests are needed for this patch. [exec] Also please list what manual steps were performed to verify this patch. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings. [exec] [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings. [exec]
        Hide
        Tsz Wo Nicholas Sze added a comment -

        Hudson is not picking up, re-submitting ...

        Show
        Tsz Wo Nicholas Sze added a comment - Hudson is not picking up, re-submitting ...
        Hide
        Tsz Wo Nicholas Sze added a comment -

        +1 patch looks good.

        Show
        Tsz Wo Nicholas Sze added a comment - +1 patch looks good.
        Hide
        Tsz Wo Nicholas Sze added a comment -

        After the latest patch, archive uses the same amount of memory as ls. Great job, Mahadev!

        • archive
           num     #instances         #bytes  class name
          ----------------------------------------------
             1:        304340       21085440  [C
             2:        804442       19306608  java.lang.String
             3:        100014        8001120  java.net.URI
             4:        100001        6400064  org.apache.hadoop.fs.FileStatus
             5:        100001        2400024  org.apache.hadoop.fs.permission.FsPermission
          
        Show
        Tsz Wo Nicholas Sze added a comment - After the latest patch, archive uses the same amount of memory as ls. Great job, Mahadev! archive num #instances #bytes class name ---------------------------------------------- 1: 304340 21085440 [C 2: 804442 19306608 java.lang.String 3: 100014 8001120 java.net.URI 4: 100001 6400064 org.apache.hadoop.fs.FileStatus 5: 100001 2400024 org.apache.hadoop.fs.permission.FsPermission
        Hide
        Mahadev konar added a comment -

        I dont have a test case here, since we did manual testing to see the memory improvements made from the patch.

        Show
        Mahadev konar added a comment - I dont have a test case here, since we did manual testing to see the memory improvements made from the patch.
        Hide
        Mahadev konar added a comment -

        a patch for trunk.

        Show
        Mahadev konar added a comment - a patch for trunk.
        Hide
        Tsz Wo Nicholas Sze added a comment -

        Yes, the patch works for 10^5 files. The har job took 41mins 21sec.

        Show
        Tsz Wo Nicholas Sze added a comment - Yes, the patch works for 10^5 files. The har job took 41mins 21sec.
        Hide
        Mahadev konar added a comment -

        here is a clean patch for y-0.20 branch. I tested the performance issues, and it fixes them. I will upload a patch for the trunk next.

        Show
        Mahadev konar added a comment - here is a clean patch for y-0.20 branch. I tested the performance issues, and it fixes them. I will upload a patch for the trunk next.
        Hide
        Mahadev konar added a comment -

        sorry the name of the file should have been mapreduce and not zookeeper patch!! Looks like I am not good at multi tasking!

        Show
        Mahadev konar added a comment - sorry the name of the file should have been mapreduce and not zookeeper patch!! Looks like I am not good at multi tasking!
        Hide
        Mahadev konar added a comment -

        can you try this patch nicholas? It fixes the issue for me.. can you also try it out?

        Show
        Mahadev konar added a comment - can you try this patch nicholas? It fixes the issue for me.. can you also try it out?
        Hide
        Tsz Wo Nicholas Sze added a comment -

        > With the patch + MAPREDUCE-1399 ...

        The patch should be applied to y0.20 but not apache 0.20. My fault!

        Show
        Tsz Wo Nicholas Sze added a comment - > With the patch + MAPREDUCE-1399 ... The patch should be applied to y0.20 but not apache 0.20. My fault!
        Hide
        Tsz Wo Nicholas Sze added a comment -

        After the patch, there are some improvement but archive still uses double memory of ls.

        • archive
           num     #instances         #bytes  class name
          ----------------------------------------------
             1:        658875       42680832  [C
             2:       1434792       34435008  java.lang.String
             3:        255174       20413920  java.net.URI
             4:        255163       16330432  org.apache.hadoop.fs.FileStatus
             5:        200001        4800024  org.apache.hadoop.fs.permission.FsPermission
             6:        255172        4082752  org.apache.hadoop.fs.Path
          
        • ls
           num     #instances         #bytes  class name
          ----------------------------------------------
             1:        304186       21086344  [C
             2:        804264       19302336  java.lang.String
             3:        100009        8000720  java.net.URI
             4:        100001        6400064  org.apache.hadoop.fs.FileStatus
             5:        100002        2400048  org.apache.hadoop.fs.permission.FsPermission
             6:        100008        1600128  org.apache.hadoop.fs.Path
          
        Show
        Tsz Wo Nicholas Sze added a comment - After the patch, there are some improvement but archive still uses double memory of ls. archive num #instances #bytes class name ---------------------------------------------- 1: 658875 42680832 [C 2: 1434792 34435008 java.lang.String 3: 255174 20413920 java.net.URI 4: 255163 16330432 org.apache.hadoop.fs.FileStatus 5: 200001 4800024 org.apache.hadoop.fs.permission.FsPermission 6: 255172 4082752 org.apache.hadoop.fs.Path ls num #instances #bytes class name ---------------------------------------------- 1: 304186 21086344 [C 2: 804264 19302336 java.lang.String 3: 100009 8000720 java.net.URI 4: 100001 6400064 org.apache.hadoop.fs.FileStatus 5: 100002 2400048 org.apache.hadoop.fs.permission.FsPermission 6: 100008 1600128 org.apache.hadoop.fs.Path
        Hide
        Tsz Wo Nicholas Sze added a comment -

        With the patch + MAPREDUCE-1399

        $hadoop archive -Dmapred.job.queue.name=q -archiveName t300.har -p /user/tsz t300 /user/tsz
        Exception in archives
        java.lang.NullPointerException
                at org.apache.hadoop.tools.HadoopArchives.run(HadoopArchives.java:662)
                at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
                at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
                at org.apache.hadoop.tools.HadoopArchives.main(HadoopArchives.java:681)
        
        Show
        Tsz Wo Nicholas Sze added a comment - With the patch + MAPREDUCE-1399 $hadoop archive -Dmapred.job.queue.name=q -archiveName t300.har -p /user/tsz t300 /user/tsz Exception in archives java.lang.NullPointerException at org.apache.hadoop.tools.HadoopArchives.run(HadoopArchives.java:662) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79) at org.apache.hadoop.tools.HadoopArchives.main(HadoopArchives.java:681)
        Hide
        Tsz Wo Nicholas Sze added a comment -

        Tried the patch. Got

        Exception in archives
        null
        
        Show
        Tsz Wo Nicholas Sze added a comment - Tried the patch. Got Exception in archives null
        Hide
        Mahadev konar added a comment -

        the patch is against 0.20 branch of hadoop.

        Show
        Mahadev konar added a comment - the patch is against 0.20 branch of hadoop.
        Hide
        Mahadev konar added a comment -

        this patch fixes the problem of too many filestatus objects while creating archvies. Nicholas, can you test this out?

        Show
        Mahadev konar added a comment - this patch fixes the problem of too many filestatus objects while creating archvies. Nicholas, can you test this out?
        Hide
        Tsz Wo Nicholas Sze added a comment -

        The archive command is roughly double the memory usage compared with ls. Below are the jmap outputs on 10^5 files

        • hadoop archive
          num     #instances         #bytes  class name
          ----------------------------------------------
             1:        665091       43671104  [C
             2:       1471573       35317752  java.lang.String
             3:        261286       20902880  java.net.URI
             4:        261276       16721664  org.apache.hadoop.fs.FileStatus
             5:        200001        4800024  org.apache.hadoop.fs.permission.FsPermission
             6:        261285        4180560  org.apache.hadoop.fs.Path
          
        • hadoop fs -ls
           num     #instances         #bytes  class name
          ----------------------------------------------
             1:        304173       21091592  [C
             2:        804254       19302096  java.lang.String
             3:        100009        8000720  java.net.URI
             4:        100001        6400064  org.apache.hadoop.fs.FileStatus
             5:        100002        2400048  org.apache.hadoop.fs.permission.FsPermission
             6:        100008        1600128  org.apache.hadoop.fs.Path
          
        Show
        Tsz Wo Nicholas Sze added a comment - The archive command is roughly double the memory usage compared with ls. Below are the jmap outputs on 10^5 files hadoop archive num #instances #bytes class name ---------------------------------------------- 1: 665091 43671104 [C 2: 1471573 35317752 java.lang.String 3: 261286 20902880 java.net.URI 4: 261276 16721664 org.apache.hadoop.fs.FileStatus 5: 200001 4800024 org.apache.hadoop.fs.permission.FsPermission 6: 261285 4180560 org.apache.hadoop.fs.Path hadoop fs -ls num #instances #bytes class name ---------------------------------------------- 1: 304173 21091592 [C 2: 804254 19302096 java.lang.String 3: 100009 8000720 java.net.URI 4: 100001 6400064 org.apache.hadoop.fs.FileStatus 5: 100002 2400048 org.apache.hadoop.fs.permission.FsPermission 6: 100008 1600128 org.apache.hadoop.fs.Path
        Hide
        Tsz Wo Nicholas Sze added a comment -

        m1425_20100129TextFileGenerator.patch: a program to generate text files.

        har.sh: useful commands for testing archive.

        Show
        Tsz Wo Nicholas Sze added a comment - m1425_20100129TextFileGenerator.patch: a program to generate text files. har.sh: useful commands for testing archive.
        Hide
        Tsz Wo Nicholas Sze added a comment -

        Got NPE this time.

        -bash-3.1$ hadoop  archive  -archiveName t4.har -p  /user/tsz  t4 .
        Exception in thread "LeaseChecker" 10/01/28 22:33:17 WARN hdfs.DFSClient: DataStreamer Exception: java.lang.NullPointerException
                at org.apache.hadoop.ipc.Client$Connection.sendParam(Client.java:467)
                at org.apache.hadoop.ipc.Client.call(Client.java:720)
                at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
                at $Proxy0.addBlock(Unknown Source)
                at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
                at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                at java.lang.reflect.Method.invoke(Method.java:597)
                at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
                at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
                at $Proxy0.addBlock(Unknown Source)
                at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2904)
                at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2786)
                at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:2076)
                at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2262)
        
        10/01/28 22:33:17 WARN hdfs.DFSClient: Error Recovery for block null bad datanode[0] nodes == null
        10/01/28 22:33:17 WARN hdfs.DFSClient: Could not get block locations. Source file "/mapredsystem/hadoop/mapredsystem/har_cezmmo/_har_src_files" - Aborting...
        Could not get block locations. Source file "/mapredsystem/hadoop/mapredsystem/har_cezmmo/_har_src_files" - Aborting...
        
        Show
        Tsz Wo Nicholas Sze added a comment - Got NPE this time. -bash-3.1$ hadoop archive -archiveName t4.har -p /user/tsz t4 . Exception in thread "LeaseChecker" 10/01/28 22:33:17 WARN hdfs.DFSClient: DataStreamer Exception: java.lang.NullPointerException at org.apache.hadoop.ipc.Client$Connection.sendParam(Client.java:467) at org.apache.hadoop.ipc.Client.call(Client.java:720) at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220) at $Proxy0.addBlock(Unknown Source) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59) at $Proxy0.addBlock(Unknown Source) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2904) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2786) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:2076) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2262) 10/01/28 22:33:17 WARN hdfs.DFSClient: Error Recovery for block null bad datanode[0] nodes == null 10/01/28 22:33:17 WARN hdfs.DFSClient: Could not get block locations. Source file "/mapredsystem/hadoop/mapredsystem/har_cezmmo/_har_src_files" - Aborting... Could not get block locations. Source file "/mapredsystem/hadoop/mapredsystem/har_cezmmo/_har_src_files" - Aborting...
        Hide
        Tsz Wo Nicholas Sze added a comment -

        Got OutOfMemoryError again after replaced . with full path.

        -bash-3.1$ hadoop  archive  -archiveName t4.har -p  /user/tsz  t4 .
        Exception in thread "LeaseChecker" java.lang.OutOfMemoryError: Java heap space
                at java.io.BufferedOutputStream.<init>(BufferedOutputStream.java:59)
                at java.io.BufferedOutputStream.<init>(BufferedOutputStream.java:42)
                at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:318)
                at org.apache.hadoop.ipc.Client$Connection.access$1700(Client.java:176)
                at org.apache.hadoop.ipc.Client.getConnection(Client.java:859)
                at org.apache.hadoop.ipc.Client.call(Client.java:719)
                at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
                at $Proxy0.renewLease(Unknown Source)
                at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
                at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                at java.lang.reflect.Method.invoke(Method.java:597)
                at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
                at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
                at $Proxy0.renewLease(Unknown Source)
                at org.apache.hadoop.hdfs.DFSClient$LeaseChecker.renew(DFSClient.java:1046)
                at org.apache.hadoop.hdfs.DFSClient$LeaseChecker.run(DFSClient.java:1058)
                at java.lang.Thread.run(Thread.java:619)
        10/01/28 22:27:57 WARN hdfs.DFSClient: DataStreamer Exception: java.lang.NullPointerException
                at org.apache.hadoop.ipc.Client$Connection.sendParam(Client.java:467)
                at org.apache.hadoop.ipc.Client.call(Client.java:720)
                at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
                at $Proxy0.addBlock(Unknown Source)
                at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
                at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                at java.lang.reflect.Method.invoke(Method.java:597)
                at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
                at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
                at $Proxy0.addBlock(Unknown Source)
                at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2904)
                at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2786)
                at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:2076)
                at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2262)
        
        10/01/28 22:27:57 WARN hdfs.DFSClient: Error Recovery for block null bad datanode[0] nodes == null
        10/01/28 22:27:57 WARN hdfs.DFSClient: Could not get block locations. Source file "/mapredsystem/hadoop/mapredsystem/har_ljz1el/_har_src_files" - Aborting...
        Could not get block locations. Source file "/mapredsystem/hadoop/mapredsystem/har_ljz1el/_har_src_files" - Aborting...
        
        Show
        Tsz Wo Nicholas Sze added a comment - Got OutOfMemoryError again after replaced . with full path. -bash-3.1$ hadoop archive -archiveName t4.har -p /user/tsz t4 . Exception in thread "LeaseChecker" java.lang.OutOfMemoryError: Java heap space at java.io.BufferedOutputStream.<init>(BufferedOutputStream.java:59) at java.io.BufferedOutputStream.<init>(BufferedOutputStream.java:42) at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:318) at org.apache.hadoop.ipc.Client$Connection.access$1700(Client.java:176) at org.apache.hadoop.ipc.Client.getConnection(Client.java:859) at org.apache.hadoop.ipc.Client.call(Client.java:719) at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220) at $Proxy0.renewLease(Unknown Source) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59) at $Proxy0.renewLease(Unknown Source) at org.apache.hadoop.hdfs.DFSClient$LeaseChecker.renew(DFSClient.java:1046) at org.apache.hadoop.hdfs.DFSClient$LeaseChecker.run(DFSClient.java:1058) at java.lang.Thread.run(Thread.java:619) 10/01/28 22:27:57 WARN hdfs.DFSClient: DataStreamer Exception: java.lang.NullPointerException at org.apache.hadoop.ipc.Client$Connection.sendParam(Client.java:467) at org.apache.hadoop.ipc.Client.call(Client.java:720) at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220) at $Proxy0.addBlock(Unknown Source) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59) at $Proxy0.addBlock(Unknown Source) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2904) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2786) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:2076) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2262) 10/01/28 22:27:57 WARN hdfs.DFSClient: Error Recovery for block null bad datanode[0] nodes == null 10/01/28 22:27:57 WARN hdfs.DFSClient: Could not get block locations. Source file "/mapredsystem/hadoop/mapredsystem/har_ljz1el/_har_src_files" - Aborting... Could not get block locations. Source file "/mapredsystem/hadoop/mapredsystem/har_ljz1el/_har_src_files" - Aborting...

          People

          • Assignee:
            Mahadev konar
            Reporter:
            Tsz Wo Nicholas Sze
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development