Hadoop Common
  1. Hadoop Common
  2. HADOOP-10612

NFS failed to refresh the user group id mapping table

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 2.4.0
    • Fix Version/s: 2.4.1
    • Component/s: nfs
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Found by Preetham Kukillaya. The user/group id mapping table is not updated periodically.

      1. HADOOP-10612.003.patch
        1 kB
        Brandon Li
      2. HADOOP-10612.002.patch
        2 kB
        Brandon Li
      3. HADOOP-10612.patch
        2 kB
        Brandon Li

        Issue Links

          Activity

          Brandon Li created issue -
          Hide
          Brandon Li added a comment -

          In IdUserGroup.java:

            synchronized private boolean isExpired() {
              return lastUpdateTime - System.currentTimeMillis() > timeout;
            }
          

          should be :

            synchronized private boolean isExpired() {
              return  System.currentTimeMillis() - lastUpdateTime > timeout;
            }
          
          Show
          Brandon Li added a comment - In IdUserGroup.java: synchronized private boolean isExpired() { return lastUpdateTime - System.currentTimeMillis() > timeout; } should be : synchronized private boolean isExpired() { return System.currentTimeMillis() - lastUpdateTime > timeout; }
          Brandon Li made changes -
          Field Original Value New Value
          Attachment HADOOP-10612.patch [ 12645118 ]
          Brandon Li made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Brandon Li made changes -
          Description Found by Preetham Kukillaya. The user/group id mapping table is not update periodically. Found by Preetham Kukillaya. The user/group id mapping table is not updated periodically.
          Hide
          Chris Nauroth added a comment -

          Hi, Brandon. The change looks good. Two questions:

          1. Is changing this config property name backwards-incompatible with existing configs that are already deployed?
          2. Do you think it's worthwhile to switch to Time#monotonicNow so that this isn't subject to system clock bugs? (i.e. Someone resets the clock to a time in the past, and then updates don't happen for a long time.)
          Show
          Chris Nauroth added a comment - Hi, Brandon. The change looks good. Two questions: Is changing this config property name backwards-incompatible with existing configs that are already deployed? Do you think it's worthwhile to switch to Time#monotonicNow so that this isn't subject to system clock bugs? (i.e. Someone resets the clock to a time in the past, and then updates don't happen for a long time.)
          Brandon Li made changes -
          Attachment HADOOP-10612.002.patch [ 12645132 ]
          Brandon Li made changes -
          Hadoop Flags Incompatible change [ 10342 ]
          Hide
          Brandon Li added a comment -

          Thank you, Chris.
          You are right that it's a incompatible change. I've updated the JIRA. This property is a hidden property for development/test and not visible in user guide or configuration files.

          I've switched to Time#monotonicNow in the new patch and will also file a JIRA to update the related code in HDFS project. Thanks for the suggestion.

          Show
          Brandon Li added a comment - Thank you, Chris. You are right that it's a incompatible change. I've updated the JIRA. This property is a hidden property for development/test and not visible in user guide or configuration files. I've switched to Time#monotonicNow in the new patch and will also file a JIRA to update the related code in HDFS project. Thanks for the suggestion.
          Brandon Li made changes -
          Link This issue relates to HDFS-6044 [ HDFS-6044 ]
          Hide
          Aaron T. Myers added a comment -

          Hey folks, why not just add a config deprecation so that the change can be done compatibly?

          Show
          Aaron T. Myers added a comment - Hey folks, why not just add a config deprecation so that the change can be done compatibly?
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12645132/HADOOP-10612.002.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          -1 javac. The patch appears to cause the build to fail.

          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/3942//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12645132/HADOOP-10612.002.patch against trunk revision . +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 javac . The patch appears to cause the build to fail. Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/3942//console This message is automatically generated.
          Hide
          Brandon Li added a comment -

          How about using this JIRA to track the bug fix, and doing the configuration change as part of the fix to HDFS-6056?

          Show
          Brandon Li added a comment - How about using this JIRA to track the bug fix, and doing the configuration change as part of the fix to HDFS-6056 ?
          Hide
          Chris Nauroth added a comment -

          How about using this JIRA to track the bug fix, and doing the configuration change as part of the fix to HDFS-6056?

          That sounds like a good plan to me. I'll be +1 for the patch after removal of the configuration property name change.

          Show
          Chris Nauroth added a comment - How about using this JIRA to track the bug fix, and doing the configuration change as part of the fix to HDFS-6056 ? That sounds like a good plan to me. I'll be +1 for the patch after removal of the configuration property name change.
          Brandon Li made changes -
          Attachment HADOOP-10612.003.patch [ 12645268 ]
          Brandon Li made changes -
          Hadoop Flags Incompatible change [ 10342 ]
          Hide
          Brandon Li added a comment -

          Uploaded a new patch to address the comments.

          Show
          Brandon Li added a comment - Uploaded a new patch to address the comments.
          Hide
          Chris Nauroth added a comment -

          +1 for patch v3 pending Jenkins. Thank you, Brandon!

          Show
          Chris Nauroth added a comment - +1 for patch v3 pending Jenkins. Thank you, Brandon!
          Chris Nauroth made changes -
          Hadoop Flags Reviewed [ 10343 ]
          Hide
          Aaron T. Myers added a comment -

          Sure, that sounds fine.

          Thanks guys.

          Show
          Aaron T. Myers added a comment - Sure, that sounds fine. Thanks guys.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12645268/HADOOP-10612.003.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-nfs.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/3946//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/3946//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12645268/HADOOP-10612.003.patch against trunk revision . +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-common-project/hadoop-nfs. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/3946//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/3946//console This message is automatically generated.
          Hide
          Brandon Li added a comment -

          It's not straightforward to add a meaningful unit test. The existing test TestIdUserGroup#testUserUpdateSetting() can cover the configuration setting and I've manually tested the table update on a local machine.

          Show
          Brandon Li added a comment - It's not straightforward to add a meaningful unit test. The existing test TestIdUserGroup#testUserUpdateSetting() can cover the configuration setting and I've manually tested the table update on a local machine.
          Hide
          Brandon Li added a comment -

          Thank you, Chris Nauroth and Aaron T. Myers. I've committed the patch.

          Show
          Brandon Li added a comment - Thank you, Chris Nauroth and Aaron T. Myers . I've committed the patch.
          Brandon Li made changes -
          Fix Version/s 2.4.1 [ 12326695 ]
          Brandon Li made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]
          Hide
          Hudson added a comment -

          SUCCESS: Integrated in Hadoop-trunk-Commit #5606 (See https://builds.apache.org/job/Hadoop-trunk-Commit/5606/)
          HADOOP-10612. NFS failed to refresh the user group id mapping table. Contributed by Brandon Li (brandonli: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1596075)

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/IdUserGroup.java
          Show
          Hudson added a comment - SUCCESS: Integrated in Hadoop-trunk-Commit #5606 (See https://builds.apache.org/job/Hadoop-trunk-Commit/5606/ ) HADOOP-10612 . NFS failed to refresh the user group id mapping table. Contributed by Brandon Li (brandonli: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1596075 ) /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/IdUserGroup.java
          Hide
          Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk #562 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/562/)
          HADOOP-10612. NFS failed to refresh the user group id mapping table. Contributed by Brandon Li (brandonli: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1596075)

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/IdUserGroup.java
          Show
          Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #562 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/562/ ) HADOOP-10612 . NFS failed to refresh the user group id mapping table. Contributed by Brandon Li (brandonli: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1596075 ) /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/IdUserGroup.java
          Hide
          Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk #1754 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1754/)
          HADOOP-10612. NFS failed to refresh the user group id mapping table. Contributed by Brandon Li (brandonli: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1596075)

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/IdUserGroup.java
          Show
          Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #1754 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1754/ ) HADOOP-10612 . NFS failed to refresh the user group id mapping table. Contributed by Brandon Li (brandonli: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1596075 ) /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/IdUserGroup.java
          Hide
          Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk #1780 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1780/)
          HADOOP-10612. NFS failed to refresh the user group id mapping table. Contributed by Brandon Li (brandonli: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1596075)

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/IdUserGroup.java
          Show
          Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #1780 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1780/ ) HADOOP-10612 . NFS failed to refresh the user group id mapping table. Contributed by Brandon Li (brandonli: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1596075 ) /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/IdUserGroup.java
          Brandon Li made changes -
          Link This issue is related to HDFS-6416 [ HDFS-6416 ]
          Arun C Murthy made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Transition Time In Source Status Execution Times Last Executer Last Execution Date
          Open Open Patch Available Patch Available
          8m 20s 1 Brandon Li 16/May/14 00:04
          Patch Available Patch Available Resolved Resolved
          3d 23h 18m 1 Brandon Li 19/May/14 23:23
          Resolved Resolved Closed Closed
          41d 9h 54m 1 Arun C Murthy 30/Jun/14 09:18

            People

            • Assignee:
              Brandon Li
              Reporter:
              Brandon Li
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development