Hadoop Common
  1. Hadoop Common
  2. HADOOP-9115

Deadlock in configuration when writing configuration to hdfs

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 1.1.1
    • Fix Version/s: 1.1.2
    • Component/s: None
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed
    • Release Note:
      This fixes a bug where Hive could trigger a deadlock condition in the Hadoop configuration management code.

      Description

      This was noticed when using hive with hadoop-1.1.1 and running

      select count(*) from tbl;
      

      This would cause a deadlock configuration.

      1. HADOOP-7082.b1.002.patch
        6 kB
        Jing Zhao
      2. HADOOP-7082.b1.patch
        6 kB
        Jing Zhao
      3. hive-jstack.log
        8 kB
        Arpit Gupta

        Issue Links

          Activity

          Hide
          Arpit Gupta added a comment -

          Thread 1640 and 1577 show the deadlock.

          Show
          Arpit Gupta added a comment - Thread 1640 and 1577 show the deadlock.
          Hide
          Jing Zhao added a comment -

          The deadlock seems to be the same issue with HDFS-1542 and HADOOP-7082. Thus we need to backport these two jiras to branch-1. Upload the backport patch first.

          Show
          Jing Zhao added a comment - The deadlock seems to be the same issue with HDFS-1542 and HADOOP-7082 . Thus we need to backport these two jiras to branch-1. Upload the backport patch first.
          Hide
          Jing Zhao added a comment -

          The backport patch passed all the unit tests and the test-patch result is:

          -1 overall.  
              +1 @author.  The patch does not contain any @author tags.
              +1 tests included.  The patch appears to include 3 new or modified tests.
              +1 javadoc.  The javadoc tool did not generate any warning messages.
              +1 javac.  The applied patch does not increase the total number of javac compiler warnings.
              -1 findbugs.  The patch appears to introduce 225 new Findbugs (version 2.0.1) warnings.
          
          Show
          Jing Zhao added a comment - The backport patch passed all the unit tests and the test-patch result is: -1 overall. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 225 new Findbugs (version 2.0.1) warnings.
          Hide
          Arpit Gupta added a comment -

          Built the jar using the patch attached and the hive query ran through without any issues.

          Show
          Arpit Gupta added a comment - Built the jar using the patch attached and the hive query ran through without any issues.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          Comparing the patch with trunk:

          • The line "Properties properties = getProps();" is in asXmlDocument() for the patch but it is in writeXml(Writer out) in trunk. Is it intended?
          • In trunk, writeXml(Writer out) is declared before asXmlDocument() but the patch put them the other way. Could you move the code to make it the same as trunk?
          Show
          Tsz Wo Nicholas Sze added a comment - Comparing the patch with trunk: The line "Properties properties = getProps();" is in asXmlDocument() for the patch but it is in writeXml(Writer out) in trunk. Is it intended? In trunk, writeXml(Writer out) is declared before asXmlDocument() but the patch put them the other way. Could you move the code to make it the same as trunk?
          Hide
          Jing Zhao added a comment -

          Thanks for the comments Nicholas. The line "Properties properties = getProps();" is moved there to make sure properties is not null (otherwise if we directly call writeXml after creating a configuration the properties can be null). The new patch addressed the function sequence comment.

          Show
          Jing Zhao added a comment - Thanks for the comments Nicholas. The line "Properties properties = getProps();" is moved there to make sure properties is not null (otherwise if we directly call writeXml after creating a configuration the properties can be null). The new patch addressed the function sequence comment.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          +1 patch looks good.

          Show
          Tsz Wo Nicholas Sze added a comment - +1 patch looks good.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          I have committed this. Thanks, Jing!

          Show
          Tsz Wo Nicholas Sze added a comment - I have committed this. Thanks, Jing!
          Hide
          Matt Foley added a comment -

          Closed upon successful release of 1.1.2.

          Show
          Matt Foley added a comment - Closed upon successful release of 1.1.2.

            People

            • Assignee:
              Jing Zhao
              Reporter:
              Arpit Gupta
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development