HBase
  1. HBase
  2. HBASE-5697

Audit HBase for usage of deprecated hadoop 0.20.x property names.

    Details

    • Type: Task Task
    • Status: Patch Available
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:

      Description

      Many xml config properties in Hadoop have changed in 0.23. We should audit hbase to insulate it from hadoop property name changes.

      Here is a list of the hadoop property name changes:
      http://hadoop.apache.org/common/docs/r0.23.1/hadoop-project-dist/hadoop-common/DeprecatedProperties.html

      1. deprecated_properties
        1 kB
        Srikanth Srungarapu
      2. HBASE-5697.patch
        49 kB
        Srikanth Srungarapu

        Issue Links

          Activity

          Jonathan Hsieh created issue -
          Jonathan Hsieh made changes -
          Field Original Value New Value
          Link This issue relates to HBASE-4924 [ HBASE-4924 ]
          Hide
          stack added a comment -

          ooo... nelly. Thats a long list Jon.

          Show
          stack added a comment - ooo... nelly. Thats a long list Jon.
          Hide
          Jonathan Hsieh added a comment -

          Hopefully that is a comprehensive list. My guess is that only a handful are relevant. I started testing on hadoop 23 and there are definitely some new deprecation warnings that show up in logs/console. Also, some have been bugs in previous versions - I've gotten snagged on this one before:

          fs.default.name -> fs.defaultFS

          Show
          Jonathan Hsieh added a comment - Hopefully that is a comprehensive list. My guess is that only a handful are relevant. I started testing on hadoop 23 and there are definitely some new deprecation warnings that show up in logs/console. Also, some have been bugs in previous versions - I've gotten snagged on this one before: fs.default.name -> fs.defaultFS
          Jonathan Hsieh made changes -
          Link This issue is related to HBASE-5738 [ HBASE-5738 ]
          Jonathan Hsieh made changes -
          Labels noob
          Hide
          Srikanth Srungarapu added a comment -

          As mentioned in https://hbase.apache.org/book/configuration.html that 0.98 version does support only versions starting from Hadoop-2.2.0, all the references in the code base to deprecated property names are changed as per https://hadoop.apache.org/docs/r2.2.0/hadoop-project-dist/hadoop-common/DeprecatedProperties.html. Also attaching all the deprecated property name changes in this patch just for the sake of completion.

          Show
          Srikanth Srungarapu added a comment - As mentioned in https://hbase.apache.org/book/configuration.html that 0.98 version does support only versions starting from Hadoop-2.2.0, all the references in the code base to deprecated property names are changed as per https://hadoop.apache.org/docs/r2.2.0/hadoop-project-dist/hadoop-common/DeprecatedProperties.html . Also attaching all the deprecated property name changes in this patch just for the sake of completion.
          Srikanth Srungarapu made changes -
          Attachment HBASE-5697.patch [ 12640356 ]
          Attachment deprecated_properties [ 12640357 ]
          Ted Yu made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Ted Yu made changes -
          Assignee Srikanth Srungarapu [ srikanth235 ]
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12640356/HBASE-5697.patch
          against trunk revision .
          ATTACHMENT ID: 12640356

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 99 new or modified tests.

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/9299//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12640356/HBASE-5697.patch against trunk revision . ATTACHMENT ID: 12640356 +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 99 new or modified tests. -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/9299//console This message is automatically generated.
          Hide
          Enis Soztutar added a comment -

          I like this. Jon, how did you compile this list? Manually?

          Show
          Enis Soztutar added a comment - I like this. Jon, how did you compile this list? Manually?
          Hide
          Srikanth Srungarapu added a comment -

          This patch is developed for 0.98 branch as it doesn't support versions before Hadoop-2.2.0 and trying to apply to trunk will result in failure( hence the Hadoop QA failure mentioned above). Enis Soztutar, I need to confess that this list was compiled manually.

          Show
          Srikanth Srungarapu added a comment - This patch is developed for 0.98 branch as it doesn't support versions before Hadoop-2.2.0 and trying to apply to trunk will result in failure( hence the Hadoop QA failure mentioned above). Enis Soztutar, I need to confess that this list was compiled manually.
          Hide
          Jonathan Hsieh added a comment - - edited

          Here's an updated link to where deprecated hadoop configs are. http://hadoop.apache.org/docs/r2.3.0/hadoop-project-dist/hadoop-common/DeprecatedProperties.html

          Will double check on review.

          Show
          Jonathan Hsieh added a comment - - edited Here's an updated link to where deprecated hadoop configs are. http://hadoop.apache.org/docs/r2.3.0/hadoop-project-dist/hadoop-common/DeprecatedProperties.html Will double check on review.
          Hide
          Jonathan Hsieh added a comment -

          Looks good. From the list i think there might be a bug – can you double check?

          mapred.map.max.attempts -> mapreduce.map.max.attempts

          According to the link I provided, this should be mapreduce.map.maxattempts (no '.' between max.attempts.)

          diff --git a/hbase-server/src/test/java/org/apache/hadoop/hbase/snapshot/TestExportSnapshot.java b/hbase-server/src/test/java/org/apache/hadoop/hbase/snapshot/TestExportSnapshot.java
          index 9b21f36..1957c4d 100644
          --- a/hbase-server/src/test/java/org/apache/hadoop/hbase/snapshot/TestExportSnapshot.java
          +++ b/hbase-server/src/test/java/org/apache/hadoop/hbase/snapshot/TestExportSnapshot.java
          @@ -82,7 +82,6 @@ public class TestExportSnapshot {
               conf.setInt(HConstants.HBASE_CLIENT_RETRIES_NUMBER, 6);
               conf.setBoolean("hbase.master.enabletable.roundrobin", true);
               conf.setInt("mapreduce.map.max.attempts", 10);
          -    conf.setInt("mapred.map.max.attempts", 10);
             }
          

          Did you grep for all the ones on the page? I'm fine with committing this but file a follow on to check the rest.

          Show
          Jonathan Hsieh added a comment - Looks good. From the list i think there might be a bug – can you double check? mapred.map.max.attempts -> mapreduce.map.max.attempts According to the link I provided, this should be mapreduce.map.maxattempts (no '.' between max.attempts.) diff --git a/hbase-server/src/test/java/org/apache/hadoop/hbase/snapshot/TestExportSnapshot.java b/hbase-server/src/test/java/org/apache/hadoop/hbase/snapshot/TestExportSnapshot.java index 9b21f36..1957c4d 100644 --- a/hbase-server/src/test/java/org/apache/hadoop/hbase/snapshot/TestExportSnapshot.java +++ b/hbase-server/src/test/java/org/apache/hadoop/hbase/snapshot/TestExportSnapshot.java @@ -82,7 +82,6 @@ public class TestExportSnapshot { conf.setInt(HConstants.HBASE_CLIENT_RETRIES_NUMBER, 6); conf.setBoolean( "hbase.master.enabletable.roundrobin" , true ); conf.setInt( "mapreduce.map.max.attempts" , 10); - conf.setInt( "mapred.map.max.attempts" , 10); } Did you grep for all the ones on the page? I'm fine with committing this but file a follow on to check the rest.

            People

            • Assignee:
              Srikanth Srungarapu
              Reporter:
              Jonathan Hsieh
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:

                Development