Hadoop Common
  1. Hadoop Common
  2. HADOOP-7708

config generator does not update the properties file if on exists already

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: 0.20.205.0
    • Fix Version/s: 0.20.205.0, 0.23.0
    • Component/s: conf
    • Labels:
      None
    • Release Note:
      Fixed hadoop-setup-conf.sh to handle config file consistently. (Eric Yang)

      Description

      We are copying configs to the hadoop conf dir but are not using the -f option. This leads to conf file not getting replaced in case the file exists and thus the user never gets the new conf.

      1. HADOOP-7708-trunk-1.patch
        6 kB
        Eric Yang
      2. HADOOP-7708-trunk.patch
        2 kB
        Eric Yang
      3. HADOOP-7708-1.patch
        6 kB
        Eric Yang
      4. HADOOP-7708.patch
        1 kB
        Eric Yang

        Activity

        Hide
        Eric Yang added a comment -

        hadoop-metrics2.propreties and log4j.properties don't get replaced if those configuration files already exist in the HADOOP_CONF_DIR directory. The intended fix is to always replace existing configuration files when hadoop-setup-conf.sh is called.

        Show
        Eric Yang added a comment - hadoop-metrics2.propreties and log4j.properties don't get replaced if those configuration files already exist in the HADOOP_CONF_DIR directory. The intended fix is to always replace existing configuration files when hadoop-setup-conf.sh is called.
        Hide
        Eric Yang added a comment -

        Replace log4j.properties and hadoop-metrics2.properties if they already existed.

        Show
        Eric Yang added a comment - Replace log4j.properties and hadoop-metrics2.properties if they already existed.
        Hide
        Arpit Gupta added a comment -

        Does it make sense to make the option to replace an existing file or not behave the same for all files not just some specific files?

        May be we can take an input from the user whether he wants to replace the configs or not? That way he only gets the new config files?

        Show
        Arpit Gupta added a comment - Does it make sense to make the option to replace an existing file or not behave the same for all files not just some specific files? May be we can take an input from the user whether he wants to replace the configs or not? That way he only gets the new config files?
        Hide
        Steve Loughran added a comment -

        This may make sense to be optional, similar as in RPMs you would flag these as %config files =you don't always want your custom settings to get stamped on. Maybe the tactic is to warn that the copies don't get copied. the Gnu cp command supports the -b option to do a backup too.

        Show
        Steve Loughran added a comment - This may make sense to be optional, similar as in RPMs you would flag these as %config files =you don't always want your custom settings to get stamped on. Maybe the tactic is to warn that the copies don't get copied. the Gnu cp command supports the -b option to do a backup too.
        Hide
        Steve Loughran added a comment -

        oh, and the problem with asking the user (e.g. cp --interactive) is that it doesn't work on automated rollouts. You really want the action option (overwrite/preserve) to be passed down

        Show
        Steve Loughran added a comment - oh, and the problem with asking the user (e.g. cp --interactive) is that it doesn't work on automated rollouts. You really want the action option (overwrite/preserve) to be passed down
        Hide
        Eric Yang added a comment -

        We should keep the operation idempotent. Therefore, the result of the hadoop-setup-conf script should be consistent when same parameters are passed to the script. The optional parameter to diverge the configuration state, seems dangerous for upgrade case.

        Show
        Eric Yang added a comment - We should keep the operation idempotent. Therefore, the result of the hadoop-setup-conf script should be consistent when same parameters are passed to the script. The optional parameter to diverge the configuration state, seems dangerous for upgrade case.
        Hide
        Eric Yang added a comment -

        Same patch for trunk.

        Show
        Eric Yang added a comment - Same patch for trunk.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12497551/HADOOP-7708-trunk.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in .

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/259//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/259//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12497551/HADOOP-7708-trunk.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/259//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/259//console This message is automatically generated.
        Hide
        Arpit Gupta added a comment -

        Eric this block also needs to be added to the block when its not automated. Other than that everything looks good.

        Show
        Arpit Gupta added a comment - Eric this block also needs to be added to the block when its not automated. Other than that everything looks good.
        Hide
        Eric Yang added a comment -
        • Revised template generation code to be consistent for all config files for 0.20 security and 0.20.205 branch
        Show
        Eric Yang added a comment - Revised template generation code to be consistent for all config files for 0.20 security and 0.20.205 branch
        Hide
        Eric Yang added a comment -

        Same patch for trunk.

        Show
        Eric Yang added a comment - Same patch for trunk.
        Hide
        Arpit Gupta added a comment -

        +1 tested the new script on a 205 cluster and it replaced the log4j.properties with a new one and moved the old one to .bak

        Show
        Arpit Gupta added a comment - +1 tested the new script on a 205 cluster and it replaced the log4j.properties with a new one and moved the old one to .bak
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12497704/HADOOP-7708-trunk-1.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in .

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/265//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/265//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12497704/HADOOP-7708-trunk-1.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/265//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/265//console This message is automatically generated.
        Hide
        Eric Yang added a comment -

        I just committed this to 0.20.205, 0.20 security, 0.23 and trunk. Thanks Arpit.

        Show
        Eric Yang added a comment - I just committed this to 0.20.205, 0.20 security, 0.23 and trunk. Thanks Arpit.
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-trunk-Commit #1013 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1013/)
        HADOOP-7708. Fixed hadoop-setup-conf.sh to handle config file
        consistently. (Eric Yang)
        HADOOP-7708. Fixed hadoop-setup-conf.sh to handle config files
        consistently. (Eric Yang)

        eyang : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1178969
        Files :

        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/hadoop-setup-conf.sh

        eyang : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1178965
        Files :

        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/hadoop-setup-conf.sh
        Show
        Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #1013 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1013/ ) HADOOP-7708 . Fixed hadoop-setup-conf.sh to handle config file consistently. (Eric Yang) HADOOP-7708 . Fixed hadoop-setup-conf.sh to handle config files consistently. (Eric Yang) eyang : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1178969 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/hadoop-setup-conf.sh eyang : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1178965 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/hadoop-setup-conf.sh
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk-Commit #1091 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1091/)
        HADOOP-7708. Fixed hadoop-setup-conf.sh to handle config file
        consistently. (Eric Yang)
        HADOOP-7708. Fixed hadoop-setup-conf.sh to handle config files
        consistently. (Eric Yang)

        eyang : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1178969
        Files :

        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/hadoop-setup-conf.sh

        eyang : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1178965
        Files :

        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/hadoop-setup-conf.sh
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #1091 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1091/ ) HADOOP-7708 . Fixed hadoop-setup-conf.sh to handle config file consistently. (Eric Yang) HADOOP-7708 . Fixed hadoop-setup-conf.sh to handle config files consistently. (Eric Yang) eyang : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1178969 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/hadoop-setup-conf.sh eyang : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1178965 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/hadoop-setup-conf.sh
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk-Commit #1031 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1031/)
        HADOOP-7708. Fixed hadoop-setup-conf.sh to handle config file
        consistently. (Eric Yang)
        HADOOP-7708. Fixed hadoop-setup-conf.sh to handle config files
        consistently. (Eric Yang)

        eyang : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1178969
        Files :

        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/hadoop-setup-conf.sh

        eyang : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1178965
        Files :

        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/hadoop-setup-conf.sh
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #1031 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1031/ ) HADOOP-7708 . Fixed hadoop-setup-conf.sh to handle config file consistently. (Eric Yang) HADOOP-7708 . Fixed hadoop-setup-conf.sh to handle config files consistently. (Eric Yang) eyang : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1178969 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/hadoop-setup-conf.sh eyang : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1178965 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/hadoop-setup-conf.sh
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk #821 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/821/)
        HADOOP-7708. Fixed hadoop-setup-conf.sh to handle config file
        consistently. (Eric Yang)
        HADOOP-7708. Fixed hadoop-setup-conf.sh to handle config files
        consistently. (Eric Yang)

        eyang : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1178969
        Files :

        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/hadoop-setup-conf.sh

        eyang : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1178965
        Files :

        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/hadoop-setup-conf.sh
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #821 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/821/ ) HADOOP-7708 . Fixed hadoop-setup-conf.sh to handle config file consistently. (Eric Yang) HADOOP-7708 . Fixed hadoop-setup-conf.sh to handle config files consistently. (Eric Yang) eyang : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1178969 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/hadoop-setup-conf.sh eyang : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1178965 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/hadoop-setup-conf.sh
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-0.23-Build #30 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/30/)
        HADOOP-7708. Fixed hadoop-setup-conf.sh to handle config file
        consistently. (Eric Yang)
        HADOOP-7708. Fixed hadoop-setup-conf.sh to handle config file
        consistently. (Eric Yang)

        eyang : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1178967
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/packages/hadoop-setup-conf.sh

        eyang : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1178966
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/packages/hadoop-setup-conf.sh
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Build #30 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/30/ ) HADOOP-7708 . Fixed hadoop-setup-conf.sh to handle config file consistently. (Eric Yang) HADOOP-7708 . Fixed hadoop-setup-conf.sh to handle config file consistently. (Eric Yang) eyang : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1178967 Files : /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/packages/hadoop-setup-conf.sh eyang : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1178966 Files : /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/packages/hadoop-setup-conf.sh
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-0.23-Build #37 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/37/)
        HADOOP-7708. Fixed hadoop-setup-conf.sh to handle config file
        consistently. (Eric Yang)
        HADOOP-7708. Fixed hadoop-setup-conf.sh to handle config file
        consistently. (Eric Yang)

        eyang : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1178967
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/packages/hadoop-setup-conf.sh

        eyang : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1178966
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/packages/hadoop-setup-conf.sh
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-0.23-Build #37 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/37/ ) HADOOP-7708 . Fixed hadoop-setup-conf.sh to handle config file consistently. (Eric Yang) HADOOP-7708 . Fixed hadoop-setup-conf.sh to handle config file consistently. (Eric Yang) eyang : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1178967 Files : /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/packages/hadoop-setup-conf.sh eyang : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1178966 Files : /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/packages/hadoop-setup-conf.sh
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #851 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/851/)
        HADOOP-7708. Fixed hadoop-setup-conf.sh to handle config file
        consistently. (Eric Yang)
        HADOOP-7708. Fixed hadoop-setup-conf.sh to handle config files
        consistently. (Eric Yang)

        eyang : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1178969
        Files :

        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/hadoop-setup-conf.sh

        eyang : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1178965
        Files :

        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/hadoop-setup-conf.sh
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #851 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/851/ ) HADOOP-7708 . Fixed hadoop-setup-conf.sh to handle config file consistently. (Eric Yang) HADOOP-7708 . Fixed hadoop-setup-conf.sh to handle config files consistently. (Eric Yang) eyang : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1178969 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/hadoop-setup-conf.sh eyang : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1178965 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/hadoop-setup-conf.sh
        Hide
        Matt Foley added a comment -

        Closed upon release of 0.20.205.0

        Show
        Matt Foley added a comment - Closed upon release of 0.20.205.0

          People

          • Assignee:
            Eric Yang
            Reporter:
            Arpit Gupta
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development