Hive
  1. Hive
  2. HIVE-6429

MapJoinKey has large memory overhead in typical cases

    Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.13.0
    • Component/s: None
    • Labels:
      None

      Description

      The only thing that MJK really needs it hashCode and equals (well, and construction), so there's no need to have array of writables in there. Assuming all the keys for a table have the same structure, for the common case where keys are primitive types, we can store something like a byte array combination of keys to reduce the memory usage. Will probably speed up compares too.

      1. HIVE-6429.10.patch
        56 kB
        Sergey Shelukhin
      2. HIVE-6429.09.patch
        59 kB
        Sergey Shelukhin
      3. HIVE-6429.08.patch
        78 kB
        Sergey Shelukhin
      4. HIVE-6429.07.patch
        76 kB
        Sergey Shelukhin
      5. HIVE-6429.06.patch
        76 kB
        Sergey Shelukhin
      6. HIVE-6429.05.patch
        75 kB
        Sergey Shelukhin
      7. HIVE-6429.04.patch
        54 kB
        Sergey Shelukhin
      8. HIVE-6429.03.patch
        51 kB
        Sergey Shelukhin
      9. HIVE-6429.02.patch
        50 kB
        Sergey Shelukhin
      10. HIVE-6429.01.patch
        50 kB
        Sergey Shelukhin
      11. HIVE-6429.patch
        49 kB
        Sergey Shelukhin
      12. HIVE-6429.WIP.patch
        49 kB
        Sergey Shelukhin

        Issue Links

          Activity

          Transition Time In Source Status Execution Times Last Executer Last Execution Date
          Patch Available Patch Available Open Open
          10d 3h 3m 1 Sergey Shelukhin 28/Feb/14 21:54
          Open Open Patch Available Patch Available
          4d 17h 18m 2 Sergey Shelukhin 01/Mar/14 00:51
          Patch Available Patch Available Resolved Resolved
          1d 22h 52m 1 Sergey Shelukhin 02/Mar/14 23:44
          Hide
          Lefty Leverenz added a comment -

          Added comment to HIVE-6586 so the description of hive.mapjoin.optimized.keys won't get lost in the shuffle when HIVE-6037 changes HiveConf.java.

          Show
          Lefty Leverenz added a comment - Added comment to HIVE-6586 so the description of hive.mapjoin.optimized.keys won't get lost in the shuffle when HIVE-6037 changes HiveConf.java.
          Lefty Leverenz made changes -
          Link This issue is related to HIVE-6188 [ HIVE-6188 ]
          Hide
          Lefty Leverenz added a comment -

          Just for the record, hive.mapjoin.optimized.keys is documented in the wiki with the description added to hive-default.xml.template by HIVE-6188:

          Whether MapJoin hashtable should use optimized (size-wise) keys, allowing the table to take less memory. Depending on key, the memory savings for entire table can be 5-15% or so.

          Show
          Lefty Leverenz added a comment - Just for the record, hive.mapjoin.optimized.keys is documented in the wiki with the description added to hive-default.xml.template by HIVE-6188 : Whether MapJoin hashtable should use optimized (size-wise) keys, allowing the table to take less memory. Depending on key, the memory savings for entire table can be 5-15% or so. patch for HIVE-6188 Configuration Properties – hive.mapjoin.optimized.keys
          Sergey Shelukhin made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Fix Version/s 0.13.0 [ 12324986 ]
          Resolution Fixed [ 1 ]
          Hide
          Sergey Shelukhin added a comment -

          committed to trunk

          Show
          Sergey Shelukhin added a comment - committed to trunk
          Hide
          Hive QA added a comment -

          Overall: -1 at least one tests failed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12631905/HIVE-6429.10.patch

          ERROR: -1 due to 1 failed/errored test(s), 5200 tests executed
          Failed tests:

          org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_auto_sortmerge_join_16
          

          Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1580/testReport
          Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1580/console

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests exited with: TestsFailedException: 1 tests failed
          

          This message is automatically generated.

          ATTACHMENT ID: 12631905

          Show
          Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12631905/HIVE-6429.10.patch ERROR: -1 due to 1 failed/errored test(s), 5200 tests executed Failed tests: org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_auto_sortmerge_join_16 Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1580/testReport Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1580/console Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 1 tests failed This message is automatically generated. ATTACHMENT ID: 12631905
          Hide
          Jitendra Nath Pandey added a comment -

          +1

          Show
          Jitendra Nath Pandey added a comment - +1
          Hide
          Sergey Shelukhin added a comment -

          tez tests pass for me

          Show
          Sergey Shelukhin added a comment - tez tests pass for me
          Hide
          Gunther Hagleitner added a comment -

          +1 (assuming tests pass)

          Show
          Gunther Hagleitner added a comment - +1 (assuming tests pass)
          Sergey Shelukhin made changes -
          Attachment HIVE-6429.10.patch [ 12631905 ]
          Hide
          Sergey Shelukhin added a comment -

          RB feedback + some internal discussion; mostly moving some key-specific stuff to key, and changing the vectorization path to go thru the elaborate writer/writable/oi path, and not raw values. Few tez tests appear to pass, I'll run the rest

          Show
          Sergey Shelukhin added a comment - RB feedback + some internal discussion; mostly moving some key-specific stuff to key, and changing the vectorization path to go thru the elaborate writer/writable/oi path, and not raw values. Few tez tests appear to pass, I'll run the rest
          Sergey Shelukhin made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Sergey Shelukhin made changes -
          Attachment HIVE-6429.09.patch [ 12631853 ]
          Hide
          Sergey Shelukhin added a comment -

          removed leftover BSS changes, filed HIVE-6526.
          Should be ready to go... would be nice to have HiveQA too

          Show
          Sergey Shelukhin added a comment - removed leftover BSS changes, filed HIVE-6526 . Should be ready to go... would be nice to have HiveQA too
          Sergey Shelukhin made changes -
          Link This issue is related to HIVE-6526 [ HIVE-6526 ]
          Sergey Shelukhin made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Sergey Shelukhin made changes -
          Attachment HIVE-6429.08.patch [ 12631660 ]
          Hide
          Sergey Shelukhin added a comment -

          after some more discussion, we decided to rewrite once again using LazyBinarySerde. I preserved some refactoring done to BinarySortableSerde.

          So here's a 3rd way to do this. There are many more untapped serde-s out there...

          I've ran a few tests that failed previously and a couple Tez tests, they all pass. I will run all tez tests now, and all tests overnight if I don't forget

          Show
          Sergey Shelukhin added a comment - after some more discussion, we decided to rewrite once again using LazyBinarySerde. I preserved some refactoring done to BinarySortableSerde. So here's a 3rd way to do this. There are many more untapped serde-s out there... I've ran a few tests that failed previously and a couple Tez tests, they all pass. I will run all tez tests now, and all tests overnight if I don't forget
          Sergey Shelukhin made changes -
          Attachment HIVE-6429.07.patch [ 12631367 ]
          Hide
          Sergey Shelukhin added a comment -

          fix the vectorization test

          Show
          Sergey Shelukhin added a comment - fix the vectorization test
          Hide
          Hive QA added a comment -

          Overall: -1 at least one tests failed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12630885/HIVE-6429.06.patch

          ERROR: -1 due to 3 failed/errored test(s), 5178 tests executed
          Failed tests:

          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_left_outer_join
          org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_auto_sortmerge_join_16
          org.apache.hive.service.cli.TestEmbeddedThriftBinaryCLIService.testExecuteStatementAsync
          

          Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1489/testReport
          Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1489/console

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests exited with: TestsFailedException: 3 tests failed
          

          This message is automatically generated.

          ATTACHMENT ID: 12630885

          Show
          Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12630885/HIVE-6429.06.patch ERROR: -1 due to 3 failed/errored test(s), 5178 tests executed Failed tests: org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_left_outer_join org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_auto_sortmerge_join_16 org.apache.hive.service.cli.TestEmbeddedThriftBinaryCLIService.testExecuteStatementAsync Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1489/testReport Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1489/console Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 3 tests failed This message is automatically generated. ATTACHMENT ID: 12630885
          Hide
          Sergey Shelukhin added a comment -

          note that I also rearranged code so diff with trunk would be easier, as I was diffing looking for bugs. Diff between iterations might be harder. Timestamp serialization was broken, some null checks were missing.

          Show
          Sergey Shelukhin added a comment - note that I also rearranged code so diff with trunk would be easier, as I was diffing looking for bugs. Diff between iterations might be harder. Timestamp serialization was broken, some null checks were missing.
          Sergey Shelukhin made changes -
          Attachment HIVE-6429.06.patch [ 12630885 ]
          Hide
          Sergey Shelukhin added a comment -

          Fixed bugs

          Show
          Sergey Shelukhin added a comment - Fixed bugs
          Hide
          Hive QA added a comment -

          Overall: -1 at least one tests failed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12630607/HIVE-6429.05.patch

          ERROR: -1 due to 137 failed/errored test(s), 5175 tests executed
          Failed tests:

          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join0
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join1
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join10
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join11
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join12
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join13
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join14_hadoop20
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join15
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join17
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join19
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join2
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join20
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join21
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join22
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join23
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join24
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join26
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join27
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join29
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join3
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join30
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join31
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join4
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join5
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join8
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join9
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join_filters
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join_nulls
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join_without_localtask
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_1
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_10
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_11
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_12
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_14
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_3
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_4
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_5
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_6
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_7
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_9
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucket_map_join_1
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucket_map_join_2
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketcontext_1
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketcontext_2
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketcontext_3
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketcontext_4
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketcontext_5
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketcontext_6
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketcontext_7
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketcontext_8
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketizedhiveinputformat_auto
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin1
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin10
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin11
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin12
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin13
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin2
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin3
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin4
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin5
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin8
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin9
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketsortoptimize_insert_2
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketsortoptimize_insert_4
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketsortoptimize_insert_5
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_column_access_stats
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer1
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer3
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer4
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer5
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer6
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer7
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cp_mj_rc
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_infer_bucket_sort_convert_join
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join25
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join26
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join27
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join28
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join29
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join30
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join31
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join32
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join32_lessSize
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join33
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join34
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join35
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join36
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join37
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join38
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join39
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join40
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_filters
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_filters_overlap
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_map_ppr
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_nulls
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_nullsafe
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_reorder4
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_star
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_mapjoin1
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_mapjoin_distinct
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_mapjoin_filter_on_outerjoin
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_mapjoin_hook
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_mapjoin_mapjoin
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_mapjoin_subquery
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_mapjoin_subquery2
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_mapjoin_test_outer
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_multiMapJoin1
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_multiMapJoin2
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_nonblock_op_deduplicate
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_partition_wise_fileformat13
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_partition_wise_fileformat14
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_select_transform_hint
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_semijoin
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_skewjoin
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_mapjoin_13
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_mapjoin_14
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_mapjoin_15
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sort_merge_join_desc_4
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sort_merge_join_desc_6
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sort_merge_join_desc_7
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats11
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_table_access_keys_stats
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_timestamp_3
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union22
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union34
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_decimal_mapjoin
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_left_outer_join
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorization_13
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorization_14
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorization_15
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorization_16
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorization_short_regress
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorized_context
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorized_mapjoin
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_windowing_rank
          org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_bucketmapjoin6
          org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_bucketmapjoin7
          

          Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1475/testReport
          Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1475/console

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests exited with: TestsFailedException: 137 tests failed
          

          This message is automatically generated.

          ATTACHMENT ID: 12630607

          Show
          Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12630607/HIVE-6429.05.patch ERROR: -1 due to 137 failed/errored test(s), 5175 tests executed Failed tests: org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join0 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join1 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join10 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join11 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join12 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join13 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join14_hadoop20 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join15 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join17 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join19 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join2 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join20 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join21 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join22 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join23 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join24 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join26 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join27 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join29 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join3 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join30 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join31 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join4 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join5 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join8 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join9 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join_filters org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join_nulls org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join_without_localtask org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_1 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_10 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_11 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_12 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_14 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_3 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_4 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_5 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_6 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_7 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_9 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucket_map_join_1 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucket_map_join_2 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketcontext_1 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketcontext_2 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketcontext_3 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketcontext_4 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketcontext_5 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketcontext_6 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketcontext_7 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketcontext_8 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketizedhiveinputformat_auto org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin1 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin10 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin11 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin12 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin13 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin2 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin3 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin4 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin5 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin8 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin9 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketsortoptimize_insert_2 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketsortoptimize_insert_4 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketsortoptimize_insert_5 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_column_access_stats org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer1 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer3 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer4 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer5 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer6 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer7 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cp_mj_rc org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_infer_bucket_sort_convert_join org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join25 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join26 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join27 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join28 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join29 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join30 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join31 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join32 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join32_lessSize org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join33 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join34 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join35 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join36 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join37 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join38 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join39 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join40 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_filters org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_filters_overlap org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_map_ppr org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_nulls org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_nullsafe org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_reorder4 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_star org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_mapjoin1 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_mapjoin_distinct org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_mapjoin_filter_on_outerjoin org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_mapjoin_hook org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_mapjoin_mapjoin org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_mapjoin_subquery org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_mapjoin_subquery2 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_mapjoin_test_outer org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_multiMapJoin1 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_multiMapJoin2 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_nonblock_op_deduplicate org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_partition_wise_fileformat13 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_partition_wise_fileformat14 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_select_transform_hint org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_semijoin org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_skewjoin org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_mapjoin_13 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_mapjoin_14 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_mapjoin_15 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sort_merge_join_desc_4 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sort_merge_join_desc_6 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sort_merge_join_desc_7 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats11 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_table_access_keys_stats org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_timestamp_3 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union22 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union34 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_decimal_mapjoin org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_left_outer_join org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorization_13 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorization_14 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorization_15 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorization_16 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorization_short_regress org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorized_context org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorized_mapjoin org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_windowing_rank org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_bucketmapjoin6 org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_bucketmapjoin7 Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1475/testReport Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1475/console Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 137 tests failed This message is automatically generated. ATTACHMENT ID: 12630607
          Sergey Shelukhin made changes -
          Attachment HIVE-6429.05.patch [ 12630607 ]
          Hide
          Sergey Shelukhin added a comment -

          05 contains changes to move to BinarySortableSerde encoding... imho it's not such a good idea.

          Show
          Sergey Shelukhin added a comment - 05 contains changes to move to BinarySortableSerde encoding... imho it's not such a good idea.
          Hide
          Hive QA added a comment -

          Overall: -1 at least one tests failed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12630468/HIVE-6429.04.patch

          ERROR: -1 due to 2 failed/errored test(s), 5175 tests executed
          Failed tests:

          org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_schemeAuthority2
          org.apache.hive.service.cli.TestEmbeddedThriftBinaryCLIService.testExecuteStatementAsync
          

          Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1448/testReport
          Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1448/console

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests exited with: TestsFailedException: 2 tests failed
          

          This message is automatically generated.

          ATTACHMENT ID: 12630468

          Show
          Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12630468/HIVE-6429.04.patch ERROR: -1 due to 2 failed/errored test(s), 5175 tests executed Failed tests: org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_schemeAuthority2 org.apache.hive.service.cli.TestEmbeddedThriftBinaryCLIService.testExecuteStatementAsync Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1448/testReport Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1448/console Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 2 tests failed This message is automatically generated. ATTACHMENT ID: 12630468
          Sergey Shelukhin made changes -
          Attachment HIVE-6429.04.patch [ 12630468 ]
          Hide
          Sergey Shelukhin added a comment -

          for now address the other feedback... I will have separate patch to use BinarySortableSerDe, just need to hack around vectorized path, but I don't think it's worth it, it's convoluted and still has to keep type array and separate path for vectorization; there
          s also additional changes because for example hasAnyNulls would be complicated and expensive with BSSD format, so it has to be additionally retrieved at key creation time for the big table key in MJO.

          Show
          Sergey Shelukhin added a comment - for now address the other feedback... I will have separate patch to use BinarySortableSerDe, just need to hack around vectorized path, but I don't think it's worth it, it's convoluted and still has to keep type array and separate path for vectorization; there s also additional changes because for example hasAnyNulls would be complicated and expensive with BSSD format, so it has to be additionally retrieved at key creation time for the big table key in MJO.
          Hide
          Lefty Leverenz added a comment -

          hive.mapjoin.optimized.keys needs a definition ... but I'm not sure where, because that depends on the state of HIVE-6037 which will put the config param definitions into HiveConf.java and then generate hive-default.xml.template from HiveConf.java.

          See comment on HIVE-6455 for details (but note that HIVE-6037 has been reopened): 17 Feb 2014 22:26 comment .

          So if this commits before HIVE-6037, hive.mapjoin.optimized.keys should be documented in hive-default.xml.template as usual but if it commits after HIVE-6037 a definition should be added to the patched version of HiveConf.java. In any case, I'll add it to the wiki with a release note.

          Show
          Lefty Leverenz added a comment - hive.mapjoin.optimized.keys needs a definition ... but I'm not sure where, because that depends on the state of HIVE-6037 which will put the config param definitions into HiveConf.java and then generate hive-default.xml.template from HiveConf.java. See comment on HIVE-6455 for details (but note that HIVE-6037 has been reopened): 17 Feb 2014 22:26 comment . So if this commits before HIVE-6037 , hive.mapjoin.optimized.keys should be documented in hive-default.xml.template as usual but if it commits after HIVE-6037 a definition should be added to the patched version of HiveConf.java. In any case, I'll add it to the wiki with a release note.
          Hide
          Sergey Shelukhin added a comment -

          Grafting this onto binarysortableserde will take a little bit of effort... will attach patch late evening today, or on Sunday evening

          Show
          Sergey Shelukhin added a comment - Grafting this onto binarysortableserde will take a little bit of effort... will attach patch late evening today, or on Sunday evening
          Sergey Shelukhin made changes -
          Attachment HIVE-6429.03.patch [ 12630350 ]
          Hide
          Gunther Hagleitner added a comment -

          partial review on rb.

          Show
          Gunther Hagleitner added a comment - partial review on rb.
          Sergey Shelukhin made changes -
          Link This issue relates to HIVE-6430 [ HIVE-6430 ]
          Sergey Shelukhin made changes -
          Attachment HIVE-6429.02.patch [ 12630136 ]
          Hide
          Sergey Shelukhin added a comment -

          one fix and one small change

          Show
          Sergey Shelukhin added a comment - one fix and one small change
          Hide
          Sergey Shelukhin added a comment -

          Measured memory savings; 1 million row hashtable; option one - double+int key, double+string value.

          Before (not sure why there's one less key, might be heap dump or profiler glitch)

          Class Objects Shallow Size Retained Size
          java.lang.Object[] 2999997 79999920 256383712
          org.apache.hadoop.hive.serde2.io.DoubleWritable 2999997 71999928 71999928
          byte[] 999999 32383968 32383968
          java.util.HashMap$Entry 999999 31999968 352383616
          java.util.ArrayList 999999 23999976 208383760
          org.apache.hadoop.hive.ql.exec.persistence.MapJoinRowContainer 999999 23999976 232383736
          org.apache.hadoop.hive.ql.exec.persistence.MapJoinRowContainer$NoCopyingArrayList 999999 23999976 160383808
          org.apache.hadoop.io.LongWritable 999999 23999976 23999976
          org.apache.hadoop.io.Text 999999 23999976 56383912
          org.apache.hadoop.hive.ql.exec.persistence.MapJoinKey 999999 15999984 87999912
          java.util.HashMap$Entry[] 1 8388624 360772272
          java.util.HashMap 1 48 360772320

          After

          Class Objects Shallow Size Retained Size
          byte[] 2000000 72384000 72384000
          java.lang.Object[] 2000000 56000000 184384000
          org.apache.hadoop.hive.serde2.io.DoubleWritable 2000000 48000000 48000000
          java.util.HashMap$Entry 1000000 32000000 320384000
          java.util.ArrayList 1000000 24000000 208384000
          org.apache.hadoop.hive.ql.exec.persistence.MapJoinRowContainer 1000000 24000000 232384000
          org.apache.hadoop.hive.ql.exec.persistence.MapJoinRowContainer$NoCopyingArrayList 1000000 24000000 160384000
          org.apache.hadoop.io.Text 1000000 24000000 56384000
          org.apache.hadoop.hive.ql.exec.persistence.MapJoinKeyBytes 1000000 16000000 56000000
          java.util.HashMap$Entry[] 1 8388624 328772016
          java.util.HashMap 1 48 328772064

          Savings are ~9%.
          Not as large as expected, as apparently Java has large overhead for byte arrays, 9-byte array takes 32 bytes and 17-byte one has 40.
          Still, reasonably good, perhaps the patch can be extended.
          Or different hashtable used...

          Show
          Sergey Shelukhin added a comment - Measured memory savings; 1 million row hashtable; option one - double+int key, double+string value. Before (not sure why there's one less key, might be heap dump or profiler glitch) Class Objects Shallow Size Retained Size java.lang.Object[] 2999997 79999920 256383712 org.apache.hadoop.hive.serde2.io.DoubleWritable 2999997 71999928 71999928 byte[] 999999 32383968 32383968 java.util.HashMap$Entry 999999 31999968 352383616 java.util.ArrayList 999999 23999976 208383760 org.apache.hadoop.hive.ql.exec.persistence.MapJoinRowContainer 999999 23999976 232383736 org.apache.hadoop.hive.ql.exec.persistence.MapJoinRowContainer$NoCopyingArrayList 999999 23999976 160383808 org.apache.hadoop.io.LongWritable 999999 23999976 23999976 org.apache.hadoop.io.Text 999999 23999976 56383912 org.apache.hadoop.hive.ql.exec.persistence.MapJoinKey 999999 15999984 87999912 java.util.HashMap$Entry[] 1 8388624 360772272 java.util.HashMap 1 48 360772320 After Class Objects Shallow Size Retained Size byte[] 2000000 72384000 72384000 java.lang.Object[] 2000000 56000000 184384000 org.apache.hadoop.hive.serde2.io.DoubleWritable 2000000 48000000 48000000 java.util.HashMap$Entry 1000000 32000000 320384000 java.util.ArrayList 1000000 24000000 208384000 org.apache.hadoop.hive.ql.exec.persistence.MapJoinRowContainer 1000000 24000000 232384000 org.apache.hadoop.hive.ql.exec.persistence.MapJoinRowContainer$NoCopyingArrayList 1000000 24000000 160384000 org.apache.hadoop.io.Text 1000000 24000000 56384000 org.apache.hadoop.hive.ql.exec.persistence.MapJoinKeyBytes 1000000 16000000 56000000 java.util.HashMap$Entry[] 1 8388624 328772016 java.util.HashMap 1 48 328772064 Savings are ~9%. Not as large as expected, as apparently Java has large overhead for byte arrays, 9-byte array takes 32 bytes and 17-byte one has 40. Still, reasonably good, perhaps the patch can be extended. Or different hashtable used...
          Hide
          Hive QA added a comment -

          Overall: -1 no tests executed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12629611/HIVE-6429.01.patch

          Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1391/testReport
          Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1391/console

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Tests exited with: NonZeroExitCodeException
          Command 'bash /data/hive-ptest/working/scratch/source-prep.sh' failed with exit status 1 and output '+ [[ -n '' ]]
          + export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
          + ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
          + export 'M2_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost -Dhttp.proxyPort=3128'
          + M2_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost -Dhttp.proxyPort=3128'
          + cd /data/hive-ptest/working/
          + tee /data/hive-ptest/logs/PreCommit-HIVE-Build-1391/source-prep.txt
          + [[ false == \t\r\u\e ]]
          + mkdir -p maven ivy
          + [[ svn = \s\v\n ]]
          + [[ -n '' ]]
          + [[ -d apache-svn-trunk-source ]]
          + [[ ! -d apache-svn-trunk-source/.svn ]]
          + [[ ! -d apache-svn-trunk-source ]]
          + cd apache-svn-trunk-source
          + svn revert -R .
          ++ egrep -v '^X|^Performing status on external'
          ++ svn status --no-ignore
          ++ awk '{print $2}'
          + rm -rf target datanucleus.log ant/target shims/target shims/0.20/target shims/0.20S/target shims/0.23/target shims/aggregator/target shims/common/target shims/common-secure/target packaging/target hbase-handler/target testutils/target jdbc/target metastore/target itests/target itests/hcatalog-unit/target itests/test-serde/target itests/qtest/target itests/hive-unit/target itests/custom-serde/target itests/util/target hcatalog/target hcatalog/storage-handlers/hbase/target hcatalog/server-extensions/target hcatalog/core/target hcatalog/webhcat/svr/target hcatalog/webhcat/java-client/target hcatalog/hcatalog-pig-adapter/target hwi/target common/target common/src/gen contrib/target service/target serde/target beeline/target odbc/target cli/target ql/dependency-reduced-pom.xml ql/target
          + svn update
          
          Fetching external item into 'hcatalog/src/test/e2e/harness'
          External at revision 1569607.
          
          At revision 1569607.
          + patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh
          + patchFilePath=/data/hive-ptest/working/scratch/build.patch
          + [[ -f /data/hive-ptest/working/scratch/build.patch ]]
          + chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh
          + /data/hive-ptest/working/scratch/smart-apply-patch.sh /data/hive-ptest/working/scratch/build.patch
          The patch does not appear to apply with p0, p1, or p2
          + exit 1
          '
          

          This message is automatically generated.

          ATTACHMENT ID: 12629611

          Show
          Hive QA added a comment - Overall : -1 no tests executed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12629611/HIVE-6429.01.patch Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1391/testReport Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1391/console Messages: Executing org.apache.hive.ptest.execution.PrepPhase Tests exited with: NonZeroExitCodeException Command 'bash /data/hive-ptest/working/scratch/source-prep.sh' failed with exit status 1 and output '+ [[ -n '' ]] + export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m ' + ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m ' + export 'M2_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost -Dhttp.proxyPort=3128' + M2_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost -Dhttp.proxyPort=3128' + cd /data/hive-ptest/working/ + tee /data/hive-ptest/logs/PreCommit-HIVE-Build-1391/source-prep.txt + [[ false == \t\r\u\e ]] + mkdir -p maven ivy + [[ svn = \s\v\n ]] + [[ -n '' ]] + [[ -d apache-svn-trunk-source ]] + [[ ! -d apache-svn-trunk-source/.svn ]] + [[ ! -d apache-svn-trunk-source ]] + cd apache-svn-trunk-source + svn revert -R . ++ egrep -v '^X|^Performing status on external' ++ svn status --no-ignore ++ awk '{print $2}' + rm -rf target datanucleus.log ant/target shims/target shims/0.20/target shims/0.20S/target shims/0.23/target shims/aggregator/target shims/common/target shims/common-secure/target packaging/target hbase-handler/target testutils/target jdbc/target metastore/target itests/target itests/hcatalog-unit/target itests/test-serde/target itests/qtest/target itests/hive-unit/target itests/custom-serde/target itests/util/target hcatalog/target hcatalog/storage-handlers/hbase/target hcatalog/server-extensions/target hcatalog/core/target hcatalog/webhcat/svr/target hcatalog/webhcat/java-client/target hcatalog/hcatalog-pig-adapter/target hwi/target common/target common/src/gen contrib/target service/target serde/target beeline/target odbc/target cli/target ql/dependency-reduced-pom.xml ql/target + svn update Fetching external item into 'hcatalog/src/test/e2e/harness' External at revision 1569607. At revision 1569607. + patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh + patchFilePath=/data/hive-ptest/working/scratch/build.patch + [[ -f /data/hive-ptest/working/scratch/build.patch ]] + chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh + /data/hive-ptest/working/scratch/smart-apply-patch.sh /data/hive-ptest/working/scratch/build.patch The patch does not appear to apply with p0, p1, or p2 + exit 1 ' This message is automatically generated. ATTACHMENT ID: 12629611
          Hide
          Sergey Shelukhin added a comment -

          all tez tests passed with HIVE-6037 backed out

          Show
          Sergey Shelukhin added a comment - all tez tests passed with HIVE-6037 backed out
          Sergey Shelukhin made changes -
          Attachment HIVE-6429.01.patch [ 12629611 ]
          Hide
          Sergey Shelukhin added a comment -

          added config setting, other minor fixes

          Show
          Sergey Shelukhin added a comment - added config setting, other minor fixes
          Show
          Sergey Shelukhin added a comment - https://reviews.apache.org/r/18230/
          Sergey Shelukhin made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Sergey Shelukhin made changes -
          Attachment HIVE-6429.patch [ 12629592 ]
          Hide
          Sergey Shelukhin added a comment -

          Tez tests are halfway and passing so far.
          I still need to add a config setting

          Show
          Sergey Shelukhin added a comment - Tez tests are halfway and passing so far. I still need to add a config setting
          Sergey Shelukhin made changes -
          Field Original Value New Value
          Attachment HIVE-6429.WIP.patch [ 12629479 ]
          Hide
          Sergey Shelukhin added a comment -

          WIP patch. Some tests appear to pass, but it cannot deal with lazy primitive serdes as I have just discovered. I will address this tomorrow.
          Safety config to disable this (enabled by default) is probably needed Gunther Hagleitner Jitendra Nath Pandey fyi

          Show
          Sergey Shelukhin added a comment - WIP patch. Some tests appear to pass, but it cannot deal with lazy primitive serdes as I have just discovered. I will address this tomorrow. Safety config to disable this (enabled by default) is probably needed Gunther Hagleitner Jitendra Nath Pandey fyi
          Sergey Shelukhin created issue -

            People

            • Assignee:
              Sergey Shelukhin
              Reporter:
              Sergey Shelukhin
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development