HBase
  1. HBase
  2. HBASE-5035

Runtime exceptions during meta scan

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Incomplete
    • Affects Version/s: 0.90.3
    • Fix Version/s: None
    • Component/s: Client
    • Labels:
      None

      Description

      Version: 0.90.3 + patches back ported

      The other day our client started spitting these two runtime exceptions. Not all clients connected to the cluster were under impact. Only 4 of them. While 3 of them were throwing NPE, one of them was throwing ArrayIndexOutOfBoundsException. The errors are :

      1. http://pastie.org/2987926
      2. http://pastie.org/2987927

      Clients did not recover from this and I had to restart them.

      Motive of this jira is to identify and put null checks at appropriate places. Also with the given stack trace I can not tell which line caused NPE of AIOBE, hence additional motive is to make the trace more helpful.

        Activity

        Hide
        Shrijeet Paliwal added a comment -

        Amm you might be right.

        final String serverAddress = Bytes.toString(value);
        
        // instantiate the location
        HRegionLocation loc = new HRegionLocation(regionInfo,
                        new HServerAddress(serverAddress));
        

        The Bytes.toString call, in theory, may return both an empty string or a null string.
        In the case when it returns a null (see below), it tries to log an error which I didn't see in my log file.
        So I am not still 100% sure this is out guy.

         try {
              return new String(b, off, len, HConstants.UTF8_ENCODING);
            } catch (UnsupportedEncodingException e) {
              LOG.error("UTF-8 not supported?", e);
              return null;
            }
        

        Nonetheless it will be good to put a check against serverAddress variable for emptiness as well nullness since HServerAddress construtor may throw runtime error otherwise. Interesting point is - it can throw both ArrayIndexOutOfBoundsException and NPE and I saw both cases.

        /**
           * @param hostAndPort Hostname and port formatted as <code>&lt;hostname> ':' &lt;port></code>
           */
          public HServerAddress(String hostAndPort) {
            int colonIndex = hostAndPort.lastIndexOf(':');
        

        I will open a subtask to make the trace more helpful.

        Show
        Shrijeet Paliwal added a comment - Amm you might be right. final String serverAddress = Bytes.toString(value); // instantiate the location HRegionLocation loc = new HRegionLocation(regionInfo, new HServerAddress(serverAddress)); The Bytes.toString call, in theory, may return both an empty string or a null string. In the case when it returns a null (see below), it tries to log an error which I didn't see in my log file. So I am not still 100% sure this is out guy. try { return new String(b, off, len, HConstants.UTF8_ENCODING); } catch (UnsupportedEncodingException e) { LOG.error("UTF-8 not supported?", e); return null; } Nonetheless it will be good to put a check against serverAddress variable for emptiness as well nullness since HServerAddress construtor may throw runtime error otherwise. Interesting point is - it can throw both ArrayIndexOutOfBoundsException and NPE and I saw both cases. /** * @param hostAndPort Hostname and port formatted as <code>&lt;hostname> ':' &lt;port></code> */ public HServerAddress(String hostAndPort) { int colonIndex = hostAndPort.lastIndexOf(':'); I will open a subtask to make the trace more helpful.
        Hide
        Ted Yu added a comment -

        I checked the code again and think regionInfo is unlikely to be null.
        Remaining possibilities are regionInfo.getTableDesc() and somewhere in the HSA ctor:

          public HServerAddress(String hostAndPort) {
        

        Making the trace more helpful should be the first action.

        Show
        Ted Yu added a comment - I checked the code again and think regionInfo is unlikely to be null. Remaining possibilities are regionInfo.getTableDesc() and somewhere in the HSA ctor: public HServerAddress( String hostAndPort) { Making the trace more helpful should be the first action.
        Hide
        Shrijeet Paliwal added a comment -

        Ted, you had mentioned following in the email thread:

        "Null check for regionInfo should be added"

        I could not gather why regionInfo could possibly be null. The call 'Writables.getHRegionInfo(value);' does not seem to return null ever. Could you please tell me your reasoning.

        Meanwhile I am still reading code and trying to find the place where NPE might occur.

        Show
        Shrijeet Paliwal added a comment - Ted, you had mentioned following in the email thread: "Null check for regionInfo should be added" I could not gather why regionInfo could possibly be null. The call 'Writables.getHRegionInfo(value);' does not seem to return null ever. Could you please tell me your reasoning. Meanwhile I am still reading code and trying to find the place where NPE might occur.
        Hide
        Shrijeet Paliwal added a comment -

        Here is the patched HCM https://gist.github.com/1478070 , can be used to match line numbers.

        Show
        Shrijeet Paliwal added a comment - Here is the patched HCM https://gist.github.com/1478070 , can be used to match line numbers.

          People

          • Assignee:
            Unassigned
            Reporter:
            Shrijeet Paliwal
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development