Hive
  1. Hive
  2. HIVE-6105

LongWritable.compareTo needs shimming

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.12.0
    • Fix Version/s: 0.13.0
    • Component/s: None
    • Labels:
      None

      Description

      Compiled hive against hadoop-2. Running tests on hadoop 1 gives:

      Caused by: java.lang.NoSuchMethodError: org.apache.hadoop.io.LongWritable.compareTo(Lorg/apache/hadoop/io/LongWritable;)I
      at org.apache.hadoop.hive.ql.udf.UDAFPercentile$MyComparator.compare(UDAFPercentile.java:62)
      at org.apache.hadoop.hive.ql.udf.UDAFPercentile$MyComparator.compare(UDAFPercentile.java:58)
      at java.util.Arrays.mergeSort(Arrays.java:1270)
      at java.util.Arrays.sort(Arrays.java:1210)
      at java.util.Collections.sort(Collections.java:157)
      at org.apache.hadoop.hive.ql.udf.UDAFPercentile$PercentileLongEvaluator.terminate(UDAFPercentile.java:194)
      ... 20 more

      I think the problem is that the compareTo function has changed from using Object to using LongWritable as an argument. Seems this needs a shim. However, we should also make sure that this shim doesn't hit performance too hard.

      1. HIVE-6105.1.patch
        8 kB
        Gunther Hagleitner
      2. HIVE-6105.1.patch.txt
        8 kB
        Navis

        Issue Links

          Activity

          Gunther Hagleitner created issue -
          Navis made changes -
          Field Original Value New Value
          Attachment HIVE-6105.1.patch.txt [ 12621031 ]
          Navis made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Assignee Navis [ navis ]
          Navis made changes -
          Remote Link This issue links to "review board (Web Link)" [ 13615 ]
          Hide
          Gunther Hagleitner added a comment -

          LGTM +1, will commit once tests have run.

          Show
          Gunther Hagleitner added a comment - LGTM +1, will commit once tests have run.
          Hide
          Hive QA added a comment -

          Overall: -1 at least one tests failed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12621031/HIVE-6105.1.patch.txt

          ERROR: -1 due to 1 failed/errored test(s), 4874 tests executed
          Failed tests:

          org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_disable_merge_for_bucketing
          

          Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/786/testReport
          Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/786/console

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests exited with: TestsFailedException: 1 tests failed
          

          This message is automatically generated.

          ATTACHMENT ID: 12621031

          Show
          Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12621031/HIVE-6105.1.patch.txt ERROR: -1 due to 1 failed/errored test(s), 4874 tests executed Failed tests: org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_disable_merge_for_bucketing Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/786/testReport Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/786/console Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 1 tests failed This message is automatically generated. ATTACHMENT ID: 12621031
          Navis made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Hide
          Navis added a comment -

          Cannot reproduce the failure. Rerunning test.

          Show
          Navis added a comment - Cannot reproduce the failure. Rerunning test.
          Navis made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Hide
          Gunther Hagleitner added a comment -

          Cannot reproduce either. Probably a fluke. Let's see what the new run brings.

          Show
          Gunther Hagleitner added a comment - Cannot reproduce either. Probably a fluke. Let's see what the new run brings.
          Hide
          Gunther Hagleitner added a comment -

          Don't see it in the q. Reuploading...

          Show
          Gunther Hagleitner added a comment - Don't see it in the q. Reuploading...
          Gunther Hagleitner made changes -
          Attachment HIVE-6105.1.patch [ 12621723 ]
          Gunther Hagleitner made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Gunther Hagleitner made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Hide
          Hive QA added a comment -

          Overall: +1 all checks pass

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12621723/HIVE-6105.1.patch

          SUCCESS: +1 4881 tests passed

          Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/819/testReport
          Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/819/console

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          

          This message is automatically generated.

          ATTACHMENT ID: 12621723

          Show
          Hive QA added a comment - Overall : +1 all checks pass Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12621723/HIVE-6105.1.patch SUCCESS: +1 4881 tests passed Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/819/testReport Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/819/console Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase This message is automatically generated. ATTACHMENT ID: 12621723
          Hide
          Gunther Hagleitner added a comment -

          Committed to trunk. Thanks Navis!

          Show
          Gunther Hagleitner added a comment - Committed to trunk. Thanks Navis!
          Gunther Hagleitner made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Fix Version/s 0.13.0 [ 12324986 ]
          Resolution Fixed [ 1 ]
          Thejas M Nair made changes -
          Link This issue breaks HIVE-6238 [ HIVE-6238 ]

            People

            • Assignee:
              Navis
              Reporter:
              Gunther Hagleitner
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development