Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-4940

namenode OOMs under Bigtop's TestCLI

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Cannot Reproduce
    • Affects Version/s: 2.1.0-beta
    • Fix Version/s: 2.3.0
    • Component/s: None
    • Labels:
      None

      Description

      Bigtop's TestCLI when executed against Hadoop 2.1.0 seems to make it OOM quite reliably regardless of the heap size settings. I'm attaching a heap dump URL. Alliteratively anybody can just take Bigtop's tests, compiled them against Hadoop 2.1.0 bits and try to reproduce it.

        Issue Links

          Activity

          Hide
          Roman Shaposhnik added a comment -
          Show
          Roman Shaposhnik added a comment - Here's where to get the heap dump: http://people.apache.org/~rvs/jira/apache/HDFS-4940/java_pid29413.hprof.bz2
          Hide
          Colin Patrick McCabe added a comment -

          You've got 1 GB of data being held in the 'data' element of a org.apache.hadoop.ipc.Server$Connection object. It looks like it's in the buffer inside the connection.

          Show
          Colin Patrick McCabe added a comment - You've got 1 GB of data being held in the 'data' element of a org.apache.hadoop.ipc.Server$Connection object. It looks like it's in the buffer inside the connection.
          Hide
          Colin Patrick McCabe added a comment -

          so I think the issue here is that the RPC layer reads 4 bytes from the client, and allocates a buffer of that size, no matter how big it is.

          Code here:

                    dataLength = dataLengthBuffer.getInt();
                    if ((dataLength == Client.PING_CALL_ID) && (!useWrap)) {
                      // covers the !useSasl too
                      dataLengthBuffer.clear();
                      return 0; // ping message
                    }
                    
                    if (dataLength < 0) {
                      LOG.warn("Unexpected data length " + dataLength + "!! from " + 
                          getHostAddress());
                    }
                    data = ByteBuffer.allocate(dataLength);
          

          It seems silly to allocate such large RPC buffers. It allows clients to bring down the NN with just one or two RPCs.

          Why don't we make the maximum RPC size configurable, and perhaps default to 64 MB or something? Even a block report of a few million blocks should fit in that size.

          Show
          Colin Patrick McCabe added a comment - so I think the issue here is that the RPC layer reads 4 bytes from the client, and allocates a buffer of that size, no matter how big it is. Code here: dataLength = dataLengthBuffer.getInt(); if ((dataLength == Client.PING_CALL_ID) && (!useWrap)) { // covers the !useSasl too dataLengthBuffer.clear(); return 0; // ping message } if (dataLength < 0) { LOG.warn( "Unexpected data length " + dataLength + "!! from " + getHostAddress()); } data = ByteBuffer.allocate(dataLength); It seems silly to allocate such large RPC buffers. It allows clients to bring down the NN with just one or two RPCs. Why don't we make the maximum RPC size configurable, and perhaps default to 64 MB or something? Even a block report of a few million blocks should fit in that size.
          Hide
          Suresh Srinivas added a comment -

          Roman Shaposhnik, we had exchange few emails about this. With Xmx == Xms, do you still see namenode running OOME? Also I had suggested setting bigger PermSize on the client side, did you get a chance to test that?

          so I think the issue here is that the RPC layer reads 4 bytes from the client, and allocates a buffer of that size, no matter how big it is.

          While it is possible, the likelihood of some request sending incorrect size in these tests is probably low. I am not sure if this is the cause.

          Show
          Suresh Srinivas added a comment - Roman Shaposhnik , we had exchange few emails about this. With Xmx == Xms, do you still see namenode running OOME? Also I had suggested setting bigger PermSize on the client side, did you get a chance to test that? so I think the issue here is that the RPC layer reads 4 bytes from the client, and allocates a buffer of that size, no matter how big it is. While it is possible, the likelihood of some request sending incorrect size in these tests is probably low. I am not sure if this is the cause.
          Hide
          Roman Shaposhnik added a comment -

          Suresh Srinivas yes NN still OOMs – the heapdump I provided is from the NN, not from the client. Client is now fine.

          I'm actually rebuilding the 2.1 RC with the patch from HADOOP-9676 to try and see what leads to the gigantic RPC call.

          Show
          Roman Shaposhnik added a comment - Suresh Srinivas yes NN still OOMs – the heapdump I provided is from the NN, not from the client. Client is now fine. I'm actually rebuilding the 2.1 RC with the patch from HADOOP-9676 to try and see what leads to the gigantic RPC call.
          Hide
          Colin Patrick McCabe added a comment -

          Have you checked out the heap dump? I looked at it in Eclipse Memory Analyzer and that's how I found the giant buffer inside org.apache.hadoop.ipc.Server$Connection. Unless the heap dump is wrong, it really does look like this is where the memory is going.

          I'm still not sure how the test is causing this problem-- hopefully the patch from HADOOP-9676 will make this more reproducible (apparently it formerly didn't reproduce all the time)

          Show
          Colin Patrick McCabe added a comment - Have you checked out the heap dump? I looked at it in Eclipse Memory Analyzer and that's how I found the giant buffer inside org.apache.hadoop.ipc.Server$Connection . Unless the heap dump is wrong, it really does look like this is where the memory is going. I'm still not sure how the test is causing this problem-- hopefully the patch from HADOOP-9676 will make this more reproducible (apparently it formerly didn't reproduce all the time)
          Hide
          Suresh Srinivas added a comment -

          Have you checked out the heap dump?

          yes. I agree with your assessment based on that alone.

          I'm still not sure how the test is causing this problem

          It would be good to get bigtop results for this to understand what causes this.

          Show
          Suresh Srinivas added a comment - Have you checked out the heap dump? yes. I agree with your assessment based on that alone. I'm still not sure how the test is causing this problem It would be good to get bigtop results for this to understand what causes this.
          Hide
          Roman Shaposhnik added a comment -

          HADOOP-9676 now allows to isolated this down to a few tests. I'll keep you guys posted.

          Show
          Roman Shaposhnik added a comment - HADOOP-9676 now allows to isolated this down to a few tests. I'll keep you guys posted.
          Hide
          Suresh Srinivas added a comment -

          Roman Shaposhnik Thanks for the update. I am really interested in seeing what is causing the memory growth. Namenode logs with HADOOP-9676 will help understand the issue.

          Show
          Suresh Srinivas added a comment - Roman Shaposhnik Thanks for the update. I am really interested in seeing what is causing the memory growth. Namenode logs with HADOOP-9676 will help understand the issue.
          Hide
          Suresh Srinivas added a comment -

          Roman Shaposhnik Can you please post the logs so that we can debug what causes buffer growth?

          Show
          Suresh Srinivas added a comment - Roman Shaposhnik Can you please post the logs so that we can debug what causes buffer growth?
          Hide
          Roman Shaposhnik added a comment -

          Suresh Srinivas Suresh, sorry for the belated reply – I'm actually on vacation till last week of July. While I'm away you can just run a Bigtop's TestCLI against the latest 2.1-based bits. Drop an email on Bigtop's ML and I'm sure folks will be more than happy to help with the details. I'll totally pick this JIRA back up once I get back.

          Show
          Roman Shaposhnik added a comment - Suresh Srinivas Suresh, sorry for the belated reply – I'm actually on vacation till last week of July. While I'm away you can just run a Bigtop's TestCLI against the latest 2.1-based bits. Drop an email on Bigtop's ML and I'm sure folks will be more than happy to help with the details. I'll totally pick this JIRA back up once I get back.
          Hide
          Arun C Murthy added a comment -

          HADOOP-9676 is a long standing bug - how is that Bigtop never hit it before? What am I missing here?

          Show
          Arun C Murthy added a comment - HADOOP-9676 is a long standing bug - how is that Bigtop never hit it before? What am I missing here?
          Hide
          Colin Patrick McCabe added a comment -

          HADOOP-9676 was not the root cause. It was just a secondary issue that was making debugging more difficult. We are still waiting on more data (such as logs) to help understand why BigTop's TestCLI is failing.

          Roman will be back from vacation next week and I believe he said "I'll totally pick this JIRA back up once I get back." So I will assign this to him.

          Show
          Colin Patrick McCabe added a comment - HADOOP-9676 was not the root cause. It was just a secondary issue that was making debugging more difficult. We are still waiting on more data (such as logs) to help understand why BigTop's TestCLI is failing. Roman will be back from vacation next week and I believe he said "I'll totally pick this JIRA back up once I get back." So I will assign this to him.
          Hide
          Arun C Murthy added a comment -

          Roman Shaposhnik Did you get a chance to look at this? I'm inclined to push this to a 2.1.1-beta to unblock 2.1.0-beta since it'll be good to get f/b on APIs etc. while we investigate this further. Makes sense? Thanks.

          Show
          Arun C Murthy added a comment - Roman Shaposhnik Did you get a chance to look at this? I'm inclined to push this to a 2.1.1-beta to unblock 2.1.0-beta since it'll be good to get f/b on APIs etc. while we investigate this further. Makes sense? Thanks.
          Hide
          Roman Shaposhnik added a comment -

          Sorry just got back from vacation – I'm currently building/running tests on RC1 in Bigtop – will update you on Wed.

          Show
          Roman Shaposhnik added a comment - Sorry just got back from vacation – I'm currently building/running tests on RC1 in Bigtop – will update you on Wed.
          Hide
          Roman Shaposhnik added a comment -

          I wasn't able to reproduce it in my 2.1.0-beta RC testing. I suggest we close this one as non-reproducible. Objections?

          Show
          Roman Shaposhnik added a comment - I wasn't able to reproduce it in my 2.1.0-beta RC testing. I suggest we close this one as non-reproducible. Objections?
          Hide
          Roman Shaposhnik added a comment -

          @arun any reason not to close this? I don't think I can repro it anymore.

          Show
          Roman Shaposhnik added a comment - @arun any reason not to close this? I don't think I can repro it anymore.
          Hide
          Suresh Srinivas added a comment -

          Roman Shaposhnik, I have closed this. We can reopen this, if this problem occurs again.

          Show
          Suresh Srinivas added a comment - Roman Shaposhnik , I have closed this. We can reopen this, if this problem occurs again.

            People

            • Assignee:
              Roman Shaposhnik
              Reporter:
              Roman Shaposhnik
            • Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development