Hive
  1. Hive
  2. HIVE-4790

MapredLocalTask task does not make virtual columns

    Details

    • Type: Bug Bug
    • Status: Open
    • Priority: Minor Minor
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: Query Processor
    • Labels:
      None

      Description

      From mailing list, http://www.mail-archive.com/user@hive.apache.org/msg08264.html

                      SELECT *,b.BLOCK__OFFSET__INSIDE__FILE FROM a JOIN b ON b.rownumber = a.number;
      fails with this error:
       
          > SELECT *,b.BLOCK__OFFSET__INSIDE__FILE FROM a JOIN b ON b.rownumber = a.number;
      Automatically selecting local only mode for query
      Total MapReduce jobs = 1
      setting HADOOP_USER_NAME        pmarron
      13/06/25 10:52:56 WARN conf.HiveConf: DEPRECATED: Configuration property hive.metastore.local no longer has any effect. Make sure to provide a valid value for hive.metastore.uris if you are connecting to a remote metastore.
      Execution log at: /tmp/pmarron/.log
      2013-06-25 10:52:56     Starting to launch local task to process map join;      maximum memory = 932118528
      java.lang.RuntimeException: cannot find field block__offset__inside__file from [0:rownumber, 1:offset]
              at org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.getStandardStructFieldRef(ObjectInspectorUtils.java:366)
              at org.apache.hadoop.hive.serde2.lazy.objectinspector.LazySimpleStructObjectInspector.getStructFieldRef(LazySimpleStructObjectInspector.java:168)
              at org.apache.hadoop.hive.serde2.objectinspector.DelegatedStructObjectInspector.getStructFieldRef(DelegatedStructObjectInspector.java:74)
              at org.apache.hadoop.hive.ql.exec.ExprNodeColumnEvaluator.initialize(ExprNodeColumnEvaluator.java:57)
              at org.apache.hadoop.hive.ql.exec.JoinUtil.getObjectInspectorsFromEvaluators(JoinUtil.java:68)
              at org.apache.hadoop.hive.ql.exec.HashTableSinkOperator.initializeOp(HashTableSinkOperator.java:222)
              at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:375)
              at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:451)
              at org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:407)
              at org.apache.hadoop.hive.ql.exec.TableScanOperator.initializeOp(TableScanOperator.java:186)
              at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:375)
              at org.apache.hadoop.hive.ql.exec.MapredLocalTask.initializeOperators(MapredLocalTask.java:394)
              at org.apache.hadoop.hive.ql.exec.MapredLocalTask.executeFromChildJVM(MapredLocalTask.java:277)
              at org.apache.hadoop.hive.ql.exec.ExecDriver.main(ExecDriver.java:676)
              at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
              at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
              at java.lang.reflect.Method.invoke(Method.java:597)
              at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
      Execution failed with exit status: 2
      
      1. HIVE-4790.D11511.1.patch
        27 kB
        Phabricator
      2. HIVE-4790.D11511.2.patch
        27 kB
        Phabricator
      3. D11511.3.patch
        22 kB
        Phabricator
      4. D11511.4.patch
        30 kB
        Phabricator
      5. HIVE-4790.5.patch.txt
        32 kB
        Navis
      6. HIVE-4790.6.patch.txt
        34 kB
        Navis
      7. HIVE-4790.7.patch.txt
        37 kB
        Navis

        Issue Links

          Activity

          Navis made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Hide
          Navis added a comment -

          Things are fucked-up rebasing on trunk. We should skip this in hive-0.13.0.

          Show
          Navis added a comment - Things are fucked-up rebasing on trunk. We should skip this in hive-0.13.0.
          Hide
          Hive QA added a comment -

          Overall: -1 at least one tests failed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12639138/HIVE-4790.7.patch.txt

          ERROR: -1 due to 25 failed/errored test(s), 5555 tests executed
          Failed tests:

          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_11
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketcontext_1
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketcontext_2
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketcontext_3
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketcontext_4
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketcontext_5
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketcontext_6
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketcontext_7
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketcontext_8
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketizedhiveinputformat_auto
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin1
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin11
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin12
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin13
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin2
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin3
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin4
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin5
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin8
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_filters
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_mapjoin_15
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sort_merge_join_desc_4
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sort_merge_join_desc_6
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats11
          org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_bucketmapjoin7
          

          Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/2180/testReport
          Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/2180/console

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests exited with: TestsFailedException: 25 tests failed
          

          This message is automatically generated.

          ATTACHMENT ID: 12639138

          Show
          Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12639138/HIVE-4790.7.patch.txt ERROR: -1 due to 25 failed/errored test(s), 5555 tests executed Failed tests: org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_11 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketcontext_1 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketcontext_2 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketcontext_3 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketcontext_4 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketcontext_5 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketcontext_6 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketcontext_7 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketcontext_8 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketizedhiveinputformat_auto org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin1 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin11 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin12 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin13 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin2 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin3 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin4 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin5 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin8 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_filters org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_mapjoin_15 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sort_merge_join_desc_4 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sort_merge_join_desc_6 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats11 org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_bucketmapjoin7 Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/2180/testReport Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/2180/console Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 25 tests failed This message is automatically generated. ATTACHMENT ID: 12639138
          Navis made changes -
          Attachment HIVE-4790.7.patch.txt [ 12639138 ]
          Hide
          Harish Butani added a comment -

          Navis this patch doesn't apply anymore. Can you take a look? This is one of the jiras you wanted in 0.13.

          Show
          Harish Butani added a comment - Navis this patch doesn't apply anymore. Can you take a look? This is one of the jiras you wanted in 0.13.
          Hide
          Hive QA added a comment -

          Overall: +1 all checks pass

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12622562/HIVE-4790.6.patch.txt

          SUCCESS: +1 4917 tests passed

          Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/874/testReport
          Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/874/console

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          

          This message is automatically generated.

          ATTACHMENT ID: 12622562

          Show
          Hive QA added a comment - Overall : +1 all checks pass Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12622562/HIVE-4790.6.patch.txt SUCCESS: +1 4917 tests passed Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/874/testReport Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/874/console Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase This message is automatically generated. ATTACHMENT ID: 12622562
          Navis made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Navis made changes -
          Attachment HIVE-4790.6.patch.txt [ 12622562 ]
          Navis made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Hide
          Hive QA added a comment -

          Overall: -1 at least one tests failed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12621556/HIVE-4790.5.patch.txt

          ERROR: -1 due to 2 failed/errored test(s), 4875 tests executed
          Failed tests:

          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_metadata_only_queries
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_only_null
          

          Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/809/testReport
          Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/809/console

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests exited with: TestsFailedException: 2 tests failed
          

          This message is automatically generated.

          ATTACHMENT ID: 12621556

          Show
          Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12621556/HIVE-4790.5.patch.txt ERROR: -1 due to 2 failed/errored test(s), 4875 tests executed Failed tests: org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_metadata_only_queries org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_only_null Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/809/testReport Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/809/console Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 2 tests failed This message is automatically generated. ATTACHMENT ID: 12621556
          Navis made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Navis made changes -
          Remote Link This issue links to "review board (Web Link)" [ 13624 ]
          Navis made changes -
          Attachment HIVE-4790.5.patch.txt [ 12621556 ]
          Hide
          Navis added a comment -

          Rebased to trunk

          Show
          Navis added a comment - Rebased to trunk
          Phabricator made changes -
          Attachment D11511.4.patch [ 12616191 ]
          Hide
          Phabricator added a comment -

          navis updated the revision "HIVE-4790 [jira] MapredLocalTask task does not make virtual columns".

          Fix test fails

          Reviewers: JIRA

          REVISION DETAIL
          https://reviews.facebook.net/D11511

          CHANGE SINCE LAST DIFF
          https://reviews.facebook.net/D11511?vs=44409&id=44589#toc

          AFFECTED FILES
          ql/pom.xml
          ql/src/java/org/apache/hadoop/hive/ql/exec/ColumnStatsTask.java
          ql/src/java/org/apache/hadoop/hive/ql/exec/FetchOperator.java
          ql/src/java/org/apache/hadoop/hive/ql/exec/FetchTask.java
          ql/src/java/org/apache/hadoop/hive/ql/exec/HashTableSinkOperator.java
          ql/src/java/org/apache/hadoop/hive/ql/exec/PartitionKeySampler.java
          ql/src/java/org/apache/hadoop/hive/ql/exec/SMBMapJoinOperator.java
          ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecMapperContext.java
          ql/src/java/org/apache/hadoop/hive/ql/exec/mr/MapredLocalTask.java
          ql/src/java/org/apache/hadoop/hive/ql/plan/ColumnStatsWork.java
          ql/src/test/queries/clientpositive/join_vc.q
          ql/src/test/results/clientpositive/join_vc.q.out

          To: JIRA, navis

          Show
          Phabricator added a comment - navis updated the revision " HIVE-4790 [jira] MapredLocalTask task does not make virtual columns". Fix test fails Reviewers: JIRA REVISION DETAIL https://reviews.facebook.net/D11511 CHANGE SINCE LAST DIFF https://reviews.facebook.net/D11511?vs=44409&id=44589#toc AFFECTED FILES ql/pom.xml ql/src/java/org/apache/hadoop/hive/ql/exec/ColumnStatsTask.java ql/src/java/org/apache/hadoop/hive/ql/exec/FetchOperator.java ql/src/java/org/apache/hadoop/hive/ql/exec/FetchTask.java ql/src/java/org/apache/hadoop/hive/ql/exec/HashTableSinkOperator.java ql/src/java/org/apache/hadoop/hive/ql/exec/PartitionKeySampler.java ql/src/java/org/apache/hadoop/hive/ql/exec/SMBMapJoinOperator.java ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecMapperContext.java ql/src/java/org/apache/hadoop/hive/ql/exec/mr/MapredLocalTask.java ql/src/java/org/apache/hadoop/hive/ql/plan/ColumnStatsWork.java ql/src/test/queries/clientpositive/join_vc.q ql/src/test/results/clientpositive/join_vc.q.out To: JIRA, navis
          Navis made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Hide
          Hive QA added a comment -

          Overall: -1 at least one tests failed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12615796/D11511.3.patch

          ERROR: -1 due to 25 failed/errored test(s), 4687 tests executed
          Failed tests:

          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_11
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketcontext_1
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketcontext_2
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketcontext_3
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketcontext_4
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketcontext_5
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketcontext_6
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketcontext_7
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketcontext_8
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketizedhiveinputformat_auto
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin1
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin11
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin12
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin13
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin2
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin3
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin4
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin5
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin8
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_filters
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_mapjoin_15
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sort_merge_join_desc_4
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sort_merge_join_desc_6
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats11
          org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_bucketmapjoin7
          

          Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/454/testReport
          Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/454/console

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests failed with: TestsFailedException: 25 tests failed
          

          This message is automatically generated.

          ATTACHMENT ID: 12615796

          Show
          Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12615796/D11511.3.patch ERROR: -1 due to 25 failed/errored test(s), 4687 tests executed Failed tests: org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_11 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketcontext_1 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketcontext_2 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketcontext_3 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketcontext_4 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketcontext_5 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketcontext_6 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketcontext_7 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketcontext_8 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketizedhiveinputformat_auto org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin1 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin11 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin12 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin13 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin2 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin3 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin4 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin5 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin8 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_filters org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_mapjoin_15 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sort_merge_join_desc_4 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sort_merge_join_desc_6 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats11 org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_bucketmapjoin7 Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/454/testReport Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/454/console Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests failed with: TestsFailedException: 25 tests failed This message is automatically generated. ATTACHMENT ID: 12615796
          Navis made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Phabricator made changes -
          Attachment D11511.3.patch [ 12615796 ]
          Hide
          Phabricator added a comment -

          navis updated the revision "HIVE-4790 [jira] MapredLocalTask task does not make virtual columns".

          Rebased to trunk

          Reviewers: JIRA

          REVISION DETAIL
          https://reviews.facebook.net/D11511

          CHANGE SINCE LAST DIFF
          https://reviews.facebook.net/D11511?vs=35637&id=44409#toc

          AFFECTED FILES
          ql/pom.xml
          ql/src/java/org/apache/hadoop/hive/ql/exec/ColumnStatsTask.java
          ql/src/java/org/apache/hadoop/hive/ql/exec/FetchOperator.java
          ql/src/java/org/apache/hadoop/hive/ql/exec/FetchTask.java
          ql/src/java/org/apache/hadoop/hive/ql/exec/PartitionKeySampler.java
          ql/src/java/org/apache/hadoop/hive/ql/exec/SMBMapJoinOperator.java
          ql/src/java/org/apache/hadoop/hive/ql/exec/mr/MapredLocalTask.java
          ql/src/java/org/apache/hadoop/hive/ql/plan/ColumnStatsWork.java
          ql/src/test/queries/clientpositive/join_vc.q
          ql/src/test/results/clientpositive/join_vc.q.out

          To: JIRA, navis

          Show
          Phabricator added a comment - navis updated the revision " HIVE-4790 [jira] MapredLocalTask task does not make virtual columns". Rebased to trunk Reviewers: JIRA REVISION DETAIL https://reviews.facebook.net/D11511 CHANGE SINCE LAST DIFF https://reviews.facebook.net/D11511?vs=35637&id=44409#toc AFFECTED FILES ql/pom.xml ql/src/java/org/apache/hadoop/hive/ql/exec/ColumnStatsTask.java ql/src/java/org/apache/hadoop/hive/ql/exec/FetchOperator.java ql/src/java/org/apache/hadoop/hive/ql/exec/FetchTask.java ql/src/java/org/apache/hadoop/hive/ql/exec/PartitionKeySampler.java ql/src/java/org/apache/hadoop/hive/ql/exec/SMBMapJoinOperator.java ql/src/java/org/apache/hadoop/hive/ql/exec/mr/MapredLocalTask.java ql/src/java/org/apache/hadoop/hive/ql/plan/ColumnStatsWork.java ql/src/test/queries/clientpositive/join_vc.q ql/src/test/results/clientpositive/join_vc.q.out To: JIRA, navis
          Navis made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Phabricator made changes -
          Attachment HIVE-4790.D11511.2.patch [ 12592017 ]
          Hide
          Phabricator added a comment -

          navis updated the revision "HIVE-4790 [jira] MapredLocalTask task does not make virtual columns".

          Rebased to trunk

          Reviewers: JIRA

          REVISION DETAIL
          https://reviews.facebook.net/D11511

          CHANGE SINCE LAST DIFF
          https://reviews.facebook.net/D11511?vs=35157&id=35637#toc

          AFFECTED FILES
          ql/src/java/org/apache/hadoop/hive/ql/exec/ColumnStatsTask.java
          ql/src/java/org/apache/hadoop/hive/ql/exec/FetchOperator.java
          ql/src/java/org/apache/hadoop/hive/ql/exec/FetchTask.java
          ql/src/java/org/apache/hadoop/hive/ql/exec/PartitionKeySampler.java
          ql/src/java/org/apache/hadoop/hive/ql/exec/SMBMapJoinOperator.java
          ql/src/java/org/apache/hadoop/hive/ql/exec/mr/MapredLocalTask.java
          ql/src/java/org/apache/hadoop/hive/ql/plan/ColumnStatsWork.java
          ql/src/java/org/apache/hadoop/hive/ql/plan/FetchWork.java
          ql/src/test/queries/clientpositive/join_vc.q
          ql/src/test/results/clientpositive/join_vc.q.out

          To: JIRA, navis

          Show
          Phabricator added a comment - navis updated the revision " HIVE-4790 [jira] MapredLocalTask task does not make virtual columns". Rebased to trunk Reviewers: JIRA REVISION DETAIL https://reviews.facebook.net/D11511 CHANGE SINCE LAST DIFF https://reviews.facebook.net/D11511?vs=35157&id=35637#toc AFFECTED FILES ql/src/java/org/apache/hadoop/hive/ql/exec/ColumnStatsTask.java ql/src/java/org/apache/hadoop/hive/ql/exec/FetchOperator.java ql/src/java/org/apache/hadoop/hive/ql/exec/FetchTask.java ql/src/java/org/apache/hadoop/hive/ql/exec/PartitionKeySampler.java ql/src/java/org/apache/hadoop/hive/ql/exec/SMBMapJoinOperator.java ql/src/java/org/apache/hadoop/hive/ql/exec/mr/MapredLocalTask.java ql/src/java/org/apache/hadoop/hive/ql/plan/ColumnStatsWork.java ql/src/java/org/apache/hadoop/hive/ql/plan/FetchWork.java ql/src/test/queries/clientpositive/join_vc.q ql/src/test/results/clientpositive/join_vc.q.out To: JIRA, navis
          Navis made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Phabricator made changes -
          Field Original Value New Value
          Attachment HIVE-4790.D11511.1.patch [ 12589717 ]
          Hide
          Phabricator added a comment -

          navis requested code review of "HIVE-4790 [jira] MapredLocalTask task does not make virtual columns".

          Reviewers: JIRA

          DPAL-4790 MapredLocalTask task does not make virtual columns

          From mailing list, http://www.mail-archive.com/user@hive.apache.org/msg08264.html

          SELECT *,b.BLOCK_OFFSETINSIDE_FILE FROM a JOIN b ON b.rownumber = a.number;
          fails with this error:

          > SELECT *,b.BLOCK_OFFSETINSIDE_FILE FROM a JOIN b ON b.rownumber = a.number;
          Automatically selecting local only mode for query
          Total MapReduce jobs = 1
          setting HADOOP_USER_NAME pmarron
          13/06/25 10:52:56 WARN conf.HiveConf: DEPRECATED: Configuration property hive.metastore.local no longer has any effect. Make sure to provide a valid value for hive.metastore.uris if you are connecting to a remote metastore.
          Execution log at: /tmp/pmarron/.log
          2013-06-25 10:52:56 Starting to launch local task to process map join; maximum memory = 932118528
          java.lang.RuntimeException: cannot find field block_offsetinside_file from [0:rownumber, 1:offset]
          at org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.getStandardStructFieldRef(ObjectInspectorUtils.java:366)
          at org.apache.hadoop.hive.serde2.lazy.objectinspector.LazySimpleStructObjectInspector.getStructFieldRef(LazySimpleStructObjectInspector.java:168)
          at org.apache.hadoop.hive.serde2.objectinspector.DelegatedStructObjectInspector.getStructFieldRef(DelegatedStructObjectInspector.java:74)
          at org.apache.hadoop.hive.ql.exec.ExprNodeColumnEvaluator.initialize(ExprNodeColumnEvaluator.java:57)
          at org.apache.hadoop.hive.ql.exec.JoinUtil.getObjectInspectorsFromEvaluators(JoinUtil.java:68)
          at org.apache.hadoop.hive.ql.exec.HashTableSinkOperator.initializeOp(HashTableSinkOperator.java:222)
          at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:375)
          at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:451)
          at org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:407)
          at org.apache.hadoop.hive.ql.exec.TableScanOperator.initializeOp(TableScanOperator.java:186)
          at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:375)
          at org.apache.hadoop.hive.ql.exec.MapredLocalTask.initializeOperators(MapredLocalTask.java:394)
          at org.apache.hadoop.hive.ql.exec.MapredLocalTask.executeFromChildJVM(MapredLocalTask.java:277)
          at org.apache.hadoop.hive.ql.exec.ExecDriver.main(ExecDriver.java:676)
          at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
          at java.lang.reflect.Method.invoke(Method.java:597)
          at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
          Execution failed with exit status: 2

          TEST PLAN
          EMPTY

          REVISION DETAIL
          https://reviews.facebook.net/D11511

          AFFECTED FILES
          ql/src/java/org/apache/hadoop/hive/ql/exec/ColumnStatsTask.java
          ql/src/java/org/apache/hadoop/hive/ql/exec/FetchOperator.java
          ql/src/java/org/apache/hadoop/hive/ql/exec/FetchTask.java
          ql/src/java/org/apache/hadoop/hive/ql/exec/MapredLocalTask.java
          ql/src/java/org/apache/hadoop/hive/ql/exec/SMBMapJoinOperator.java
          ql/src/java/org/apache/hadoop/hive/ql/plan/ColumnStatsWork.java
          ql/src/java/org/apache/hadoop/hive/ql/plan/FetchWork.java
          ql/src/test/queries/clientpositive/join_vc.q
          ql/src/test/results/clientpositive/join_vc.q.out

          MANAGE HERALD RULES
          https://reviews.facebook.net/herald/view/differential/

          WHY DID I GET THIS EMAIL?
          https://reviews.facebook.net/herald/transcript/27237/

          To: JIRA, navis

          Show
          Phabricator added a comment - navis requested code review of " HIVE-4790 [jira] MapredLocalTask task does not make virtual columns". Reviewers: JIRA DPAL-4790 MapredLocalTask task does not make virtual columns From mailing list, http://www.mail-archive.com/user@hive.apache.org/msg08264.html SELECT *,b.BLOCK_ OFFSET INSIDE _FILE FROM a JOIN b ON b.rownumber = a.number; fails with this error: > SELECT *,b.BLOCK_ OFFSET INSIDE _FILE FROM a JOIN b ON b.rownumber = a.number; Automatically selecting local only mode for query Total MapReduce jobs = 1 setting HADOOP_USER_NAME pmarron 13/06/25 10:52:56 WARN conf.HiveConf: DEPRECATED: Configuration property hive.metastore.local no longer has any effect. Make sure to provide a valid value for hive.metastore.uris if you are connecting to a remote metastore. Execution log at: /tmp/pmarron/.log 2013-06-25 10:52:56 Starting to launch local task to process map join; maximum memory = 932118528 java.lang.RuntimeException: cannot find field block_ offset inside _file from [0:rownumber, 1:offset] at org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.getStandardStructFieldRef(ObjectInspectorUtils.java:366) at org.apache.hadoop.hive.serde2.lazy.objectinspector.LazySimpleStructObjectInspector.getStructFieldRef(LazySimpleStructObjectInspector.java:168) at org.apache.hadoop.hive.serde2.objectinspector.DelegatedStructObjectInspector.getStructFieldRef(DelegatedStructObjectInspector.java:74) at org.apache.hadoop.hive.ql.exec.ExprNodeColumnEvaluator.initialize(ExprNodeColumnEvaluator.java:57) at org.apache.hadoop.hive.ql.exec.JoinUtil.getObjectInspectorsFromEvaluators(JoinUtil.java:68) at org.apache.hadoop.hive.ql.exec.HashTableSinkOperator.initializeOp(HashTableSinkOperator.java:222) at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:375) at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:451) at org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:407) at org.apache.hadoop.hive.ql.exec.TableScanOperator.initializeOp(TableScanOperator.java:186) at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:375) at org.apache.hadoop.hive.ql.exec.MapredLocalTask.initializeOperators(MapredLocalTask.java:394) at org.apache.hadoop.hive.ql.exec.MapredLocalTask.executeFromChildJVM(MapredLocalTask.java:277) at org.apache.hadoop.hive.ql.exec.ExecDriver.main(ExecDriver.java:676) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.RunJar.main(RunJar.java:156) Execution failed with exit status: 2 TEST PLAN EMPTY REVISION DETAIL https://reviews.facebook.net/D11511 AFFECTED FILES ql/src/java/org/apache/hadoop/hive/ql/exec/ColumnStatsTask.java ql/src/java/org/apache/hadoop/hive/ql/exec/FetchOperator.java ql/src/java/org/apache/hadoop/hive/ql/exec/FetchTask.java ql/src/java/org/apache/hadoop/hive/ql/exec/MapredLocalTask.java ql/src/java/org/apache/hadoop/hive/ql/exec/SMBMapJoinOperator.java ql/src/java/org/apache/hadoop/hive/ql/plan/ColumnStatsWork.java ql/src/java/org/apache/hadoop/hive/ql/plan/FetchWork.java ql/src/test/queries/clientpositive/join_vc.q ql/src/test/results/clientpositive/join_vc.q.out MANAGE HERALD RULES https://reviews.facebook.net/herald/view/differential/ WHY DID I GET THIS EMAIL? https://reviews.facebook.net/herald/transcript/27237/ To: JIRA, navis
          Navis created issue -

            People

            • Assignee:
              Navis
              Reporter:
              Navis
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:

                Development