Hive
  1. Hive
  2. HIVE-5342

Remove pre hadoop-0.20.0 related codes

    Details

    • Type: Task Task
    • Status: Closed
    • Priority: Trivial Trivial
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.14.0
    • Component/s: None
    • Labels:
      None

      Description

      Recently, we discussed not supporting hadoop-0.20.0. If it would be done like that or not, 0.17 related codes would be removed before that.

      1. HIVE-5342.2.patch
        8 kB
        Jason Dere
      2. HIVE-5342.2.patch
        8 kB
        Jason Dere
      3. HIVE-5342.2.patch
        8 kB
        Ashutosh Chauhan
      4. HIVE-5342.1.patch
        7 kB
        Jason Dere
      5. D13047.1.patch
        320 kB
        Phabricator

        Issue Links

          Activity

          Hide
          Navis added a comment -

          Removed dummy counters in Operator.

          Show
          Navis added a comment - Removed dummy counters in Operator.
          Hide
          Phabricator added a comment -

          navis requested code review of "HIVE-5342 [jira] Remove pre hadoop-0.20.0 related codes".

          Reviewers: JIRA

          HIVE-5342 Remove pre hadoop-0.20.0 related codes

          Recently, we discussed not supporting hadoop-0.20.0. If it would be done like that or not, 0.17 related codes would be removed before that.

          TEST PLAN
          EMPTY

          REVISION DETAIL
          https://reviews.facebook.net/D13047

          AFFECTED FILES
          ql/src/java/org/apache/hadoop/hive/ql/exec/CommonJoinOperator.java
          ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java
          ql/src/java/org/apache/hadoop/hive/ql/exec/GroupByOperator.java
          ql/src/java/org/apache/hadoop/hive/ql/exec/ListSinkOperator.java
          ql/src/java/org/apache/hadoop/hive/ql/exec/Operator.java
          ql/src/java/org/apache/hadoop/hive/ql/exec/OperatorFactory.java
          ql/src/java/org/apache/hadoop/hive/ql/exec/ReduceSinkOperator.java
          ql/src/java/org/apache/hadoop/hive/ql/io/SymlinkTextInputFormat.java
          ql/src/java/org/apache/hadoop/hive/ql/parse/MapReduceCompiler.java
          ql/src/test/results/compiler/plan/case_sensitivity.q.xml
          ql/src/test/results/compiler/plan/cast1.q.xml
          ql/src/test/results/compiler/plan/groupby1.q.xml
          ql/src/test/results/compiler/plan/groupby2.q.xml
          ql/src/test/results/compiler/plan/groupby3.q.xml
          ql/src/test/results/compiler/plan/groupby4.q.xml
          ql/src/test/results/compiler/plan/groupby5.q.xml
          ql/src/test/results/compiler/plan/groupby6.q.xml
          ql/src/test/results/compiler/plan/input1.q.xml
          ql/src/test/results/compiler/plan/input2.q.xml
          ql/src/test/results/compiler/plan/input20.q.xml
          ql/src/test/results/compiler/plan/input3.q.xml
          ql/src/test/results/compiler/plan/input4.q.xml
          ql/src/test/results/compiler/plan/input5.q.xml
          ql/src/test/results/compiler/plan/input6.q.xml
          ql/src/test/results/compiler/plan/input7.q.xml
          ql/src/test/results/compiler/plan/input8.q.xml
          ql/src/test/results/compiler/plan/input9.q.xml
          ql/src/test/results/compiler/plan/input_part1.q.xml
          ql/src/test/results/compiler/plan/input_testsequencefile.q.xml
          ql/src/test/results/compiler/plan/input_testxpath.q.xml
          ql/src/test/results/compiler/plan/input_testxpath2.q.xml
          ql/src/test/results/compiler/plan/join1.q.xml
          ql/src/test/results/compiler/plan/join2.q.xml
          ql/src/test/results/compiler/plan/join3.q.xml
          ql/src/test/results/compiler/plan/join4.q.xml
          ql/src/test/results/compiler/plan/join5.q.xml
          ql/src/test/results/compiler/plan/join6.q.xml
          ql/src/test/results/compiler/plan/join7.q.xml
          ql/src/test/results/compiler/plan/join8.q.xml
          ql/src/test/results/compiler/plan/sample1.q.xml
          ql/src/test/results/compiler/plan/sample2.q.xml
          ql/src/test/results/compiler/plan/sample3.q.xml
          ql/src/test/results/compiler/plan/sample4.q.xml
          ql/src/test/results/compiler/plan/sample5.q.xml
          ql/src/test/results/compiler/plan/sample6.q.xml
          ql/src/test/results/compiler/plan/sample7.q.xml
          ql/src/test/results/compiler/plan/subq.q.xml
          ql/src/test/results/compiler/plan/udf1.q.xml
          ql/src/test/results/compiler/plan/udf4.q.xml
          ql/src/test/results/compiler/plan/udf6.q.xml
          ql/src/test/results/compiler/plan/udf_case.q.xml
          ql/src/test/results/compiler/plan/udf_when.q.xml
          ql/src/test/results/compiler/plan/union.q.xml

          MANAGE HERALD RULES
          https://reviews.facebook.net/herald/view/differential/

          WHY DID I GET THIS EMAIL?
          https://reviews.facebook.net/herald/transcript/37887/

          To: JIRA, navis

          Show
          Phabricator added a comment - navis requested code review of " HIVE-5342 [jira] Remove pre hadoop-0.20.0 related codes". Reviewers: JIRA HIVE-5342 Remove pre hadoop-0.20.0 related codes Recently, we discussed not supporting hadoop-0.20.0. If it would be done like that or not, 0.17 related codes would be removed before that. TEST PLAN EMPTY REVISION DETAIL https://reviews.facebook.net/D13047 AFFECTED FILES ql/src/java/org/apache/hadoop/hive/ql/exec/CommonJoinOperator.java ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java ql/src/java/org/apache/hadoop/hive/ql/exec/GroupByOperator.java ql/src/java/org/apache/hadoop/hive/ql/exec/ListSinkOperator.java ql/src/java/org/apache/hadoop/hive/ql/exec/Operator.java ql/src/java/org/apache/hadoop/hive/ql/exec/OperatorFactory.java ql/src/java/org/apache/hadoop/hive/ql/exec/ReduceSinkOperator.java ql/src/java/org/apache/hadoop/hive/ql/io/SymlinkTextInputFormat.java ql/src/java/org/apache/hadoop/hive/ql/parse/MapReduceCompiler.java ql/src/test/results/compiler/plan/case_sensitivity.q.xml ql/src/test/results/compiler/plan/cast1.q.xml ql/src/test/results/compiler/plan/groupby1.q.xml ql/src/test/results/compiler/plan/groupby2.q.xml ql/src/test/results/compiler/plan/groupby3.q.xml ql/src/test/results/compiler/plan/groupby4.q.xml ql/src/test/results/compiler/plan/groupby5.q.xml ql/src/test/results/compiler/plan/groupby6.q.xml ql/src/test/results/compiler/plan/input1.q.xml ql/src/test/results/compiler/plan/input2.q.xml ql/src/test/results/compiler/plan/input20.q.xml ql/src/test/results/compiler/plan/input3.q.xml ql/src/test/results/compiler/plan/input4.q.xml ql/src/test/results/compiler/plan/input5.q.xml ql/src/test/results/compiler/plan/input6.q.xml ql/src/test/results/compiler/plan/input7.q.xml ql/src/test/results/compiler/plan/input8.q.xml ql/src/test/results/compiler/plan/input9.q.xml ql/src/test/results/compiler/plan/input_part1.q.xml ql/src/test/results/compiler/plan/input_testsequencefile.q.xml ql/src/test/results/compiler/plan/input_testxpath.q.xml ql/src/test/results/compiler/plan/input_testxpath2.q.xml ql/src/test/results/compiler/plan/join1.q.xml ql/src/test/results/compiler/plan/join2.q.xml ql/src/test/results/compiler/plan/join3.q.xml ql/src/test/results/compiler/plan/join4.q.xml ql/src/test/results/compiler/plan/join5.q.xml ql/src/test/results/compiler/plan/join6.q.xml ql/src/test/results/compiler/plan/join7.q.xml ql/src/test/results/compiler/plan/join8.q.xml ql/src/test/results/compiler/plan/sample1.q.xml ql/src/test/results/compiler/plan/sample2.q.xml ql/src/test/results/compiler/plan/sample3.q.xml ql/src/test/results/compiler/plan/sample4.q.xml ql/src/test/results/compiler/plan/sample5.q.xml ql/src/test/results/compiler/plan/sample6.q.xml ql/src/test/results/compiler/plan/sample7.q.xml ql/src/test/results/compiler/plan/subq.q.xml ql/src/test/results/compiler/plan/udf1.q.xml ql/src/test/results/compiler/plan/udf4.q.xml ql/src/test/results/compiler/plan/udf6.q.xml ql/src/test/results/compiler/plan/udf_case.q.xml ql/src/test/results/compiler/plan/udf_when.q.xml ql/src/test/results/compiler/plan/union.q.xml MANAGE HERALD RULES https://reviews.facebook.net/herald/view/differential/ WHY DID I GET THIS EMAIL? https://reviews.facebook.net/herald/transcript/37887/ To: JIRA, navis
          Hide
          Ashutosh Chauhan added a comment -

          Dupe of HIVE-4518 ? Navis You may want to review patch on HIVE-4518 which seems to be more comprehensive than this one.

          Show
          Ashutosh Chauhan added a comment - Dupe of HIVE-4518 ? Navis You may want to review patch on HIVE-4518 which seems to be more comprehensive than this one.
          Hide
          Navis added a comment -

          Removing counter in Operator is the first step. There would be following works related to this, something like,

          1. replace serde2.io.*Writable with hadoop.io.*Writable
          2. remove shims for comparing Text
          3. possibly removing ByteArrayRef in Lazy<Non-Primitives>
          4. etc.

          I'll check the HIVE-4518, but it seemed a little intrusive at first look.

          Show
          Navis added a comment - Removing counter in Operator is the first step. There would be following works related to this, something like, 1. replace serde2.io.*Writable with hadoop.io.*Writable 2. remove shims for comparing Text 3. possibly removing ByteArrayRef in Lazy<Non-Primitives> 4. etc. I'll check the HIVE-4518 , but it seemed a little intrusive at first look.
          Hide
          Navis added a comment -

          Seemed mostly fixed in HIVE-4518 and HIVE-5786. Replacing serde2.io.*Writable seemed to be a huge work and has not much meaning. So closing this as duplicate.

          Show
          Navis added a comment - Seemed mostly fixed in HIVE-4518 and HIVE-5786 . Replacing serde2.io.*Writable seemed to be a huge work and has not much meaning. So closing this as duplicate.
          Hide
          Phabricator added a comment -

          navis has abandoned the revision "HIVE-5342 [jira] Remove pre hadoop-0.20.0 related codes".

          REVISION DETAIL
          https://reviews.facebook.net/D13047

          To: JIRA, navis

          Show
          Phabricator added a comment - navis has abandoned the revision " HIVE-5342 [jira] Remove pre hadoop-0.20.0 related codes". REVISION DETAIL https://reviews.facebook.net/D13047 To: JIRA, navis
          Hide
          Jason Dere added a comment -

          Found some miscellaneous references to hadoop 0.17 workarounds in the code, will look into trying to remove some of those.

          Show
          Jason Dere added a comment - Found some miscellaneous references to hadoop 0.17 workarounds in the code, will look into trying to remove some of those.
          Hide
          Jason Dere added a comment -
          • Tried removing 0.17 workarounds from LazySimpleSerde/LazyBinarySerde, which has different code for Text/BytesWritable because they didn't share common interface in 0.17. Ashutosh Chauhan Does this look correct?
          • exec/Utilities has some 0.17 workarounds relating to file paths. Not everything could be removed here since it looks like both file:/// and just the path string are supposed to work. The changes now allow file:/ to work in addition to file:///.
          Show
          Jason Dere added a comment - Tried removing 0.17 workarounds from LazySimpleSerde/LazyBinarySerde, which has different code for Text/BytesWritable because they didn't share common interface in 0.17. Ashutosh Chauhan Does this look correct? exec/Utilities has some 0.17 workarounds relating to file paths. Not everything could be removed here since it looks like both file:/// and just the path string are supposed to work. The changes now allow file:/ to work in addition to file:/// .
          Hide
          Ashutosh Chauhan added a comment -
          • In SkewJoinHandler, you can get rid of catch(IOException) block altogether.
          • In LazySimpleSerde::deserialize() lets not do any instanceof check. Since this is on performance critical path, we should avoid any if branch. Lets just cast it to BinaryComparable. In worst case, ClasscastException would be thrown, which is fine.
          • Same as above for LazyBinarySerde.
          Show
          Ashutosh Chauhan added a comment - In SkewJoinHandler, you can get rid of catch(IOException) block altogether. In LazySimpleSerde::deserialize() lets not do any instanceof check. Since this is on performance critical path, we should avoid any if branch. Lets just cast it to BinaryComparable. In worst case, ClasscastException would be thrown, which is fine. Same as above for LazyBinarySerde.
          Hide
          Jason Dere added a comment -

          Incorporating Ashutosh's feedback.

          Show
          Jason Dere added a comment - Incorporating Ashutosh's feedback.
          Hide
          Ashutosh Chauhan added a comment -

          +1

          Show
          Ashutosh Chauhan added a comment - +1
          Hide
          Hive QA added a comment -

          Overall: -1 no tests executed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12644454/HIVE-5342.2.patch

          Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/184/testReport
          Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/184/console

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          

          This message is automatically generated.

          ATTACHMENT ID: 12644454

          Show
          Hive QA added a comment - Overall : -1 no tests executed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12644454/HIVE-5342.2.patch Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/184/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/184/console Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase This message is automatically generated. ATTACHMENT ID: 12644454
          Hide
          Jason Dere added a comment -

          The test report link didn't look like it was for this patch. Uploading patch again.

          Show
          Jason Dere added a comment - The test report link didn't look like it was for this patch. Uploading patch again.
          Hide
          Ashutosh Chauhan added a comment -

          Re-upload for Hive QA to pick up.

          Show
          Ashutosh Chauhan added a comment - Re-upload for Hive QA to pick up.
          Hide
          Hive QA added a comment -

          Overall: -1 at least one tests failed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12645712/HIVE-5342.2.patch

          ERROR: -1 due to 11 failed/errored test(s), 5526 tests executed
          Failed tests:

          org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_root_dir_external_table
          org.apache.hive.hcatalog.pig.TestHCatLoader.testReadDataPrimitiveTypes
          org.apache.hive.hcatalog.pig.TestOrcHCatPigStorer.testWriteDecimal
          org.apache.hive.hcatalog.pig.TestOrcHCatPigStorer.testWriteDecimalX
          org.apache.hive.hcatalog.pig.TestOrcHCatPigStorer.testWriteDecimalXY
          org.apache.hive.hcatalog.templeton.TestWebHCatE2e.getHadoopVersion
          org.apache.hive.hcatalog.templeton.TestWebHCatE2e.getHiveVersion
          org.apache.hive.hcatalog.templeton.TestWebHCatE2e.getPigVersion
          org.apache.hive.hcatalog.templeton.TestWebHCatE2e.getStatus
          org.apache.hive.hcatalog.templeton.TestWebHCatE2e.invalidPath
          org.apache.hive.jdbc.miniHS2.TestHiveServer2.testConnection
          

          Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/241/testReport
          Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/241/console
          Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logsPreCommit-HIVE-Build-241/

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests exited with: TestsFailedException: 11 tests failed
          

          This message is automatically generated.

          ATTACHMENT ID: 12645712

          Show
          Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12645712/HIVE-5342.2.patch ERROR: -1 due to 11 failed/errored test(s), 5526 tests executed Failed tests: org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_root_dir_external_table org.apache.hive.hcatalog.pig.TestHCatLoader.testReadDataPrimitiveTypes org.apache.hive.hcatalog.pig.TestOrcHCatPigStorer.testWriteDecimal org.apache.hive.hcatalog.pig.TestOrcHCatPigStorer.testWriteDecimalX org.apache.hive.hcatalog.pig.TestOrcHCatPigStorer.testWriteDecimalXY org.apache.hive.hcatalog.templeton.TestWebHCatE2e.getHadoopVersion org.apache.hive.hcatalog.templeton.TestWebHCatE2e.getHiveVersion org.apache.hive.hcatalog.templeton.TestWebHCatE2e.getPigVersion org.apache.hive.hcatalog.templeton.TestWebHCatE2e.getStatus org.apache.hive.hcatalog.templeton.TestWebHCatE2e.invalidPath org.apache.hive.jdbc.miniHS2.TestHiveServer2.testConnection Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/241/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/241/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logsPreCommit-HIVE-Build-241/ Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 11 tests failed This message is automatically generated. ATTACHMENT ID: 12645712
          Hide
          Ashutosh Chauhan added a comment -

          Committed to trunk. Thanks, Jason!

          Show
          Ashutosh Chauhan added a comment - Committed to trunk. Thanks, Jason!
          Hide
          Thejas M Nair added a comment -

          This has been fixed in 0.14 release. Please open new jira if you see any issues.

          Show
          Thejas M Nair added a comment - This has been fixed in 0.14 release. Please open new jira if you see any issues.

            People

            • Assignee:
              Jason Dere
              Reporter:
              Navis
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development