Details

      Description

      HIVE-6707 fixed an issue with lazy maps. Lazy maps scans for keys using key separators and maintains a counter for the number of keys i.e; map size. However keys may include duplicates in which returning the number of keys for map size will be wrong as after deserialization the actual map will not have duplicate keys.

      This is a subtask of HIVE-6707 to fix the getMapSize() api.

      1. HIVE-6714.1.patch
        4 kB
        Prasanth Jayachandran
      2. HIVE-6714.2.patch
        5 kB
        Prasanth Jayachandran
      3. HIVE-6714.2.patch
        5 kB
        Prasanth Jayachandran
      4. HIVE-6714.3.patch
        5 kB
        Prasanth Jayachandran
      5. HIVE-6714.3.patch
        5 kB
        Prasanth Jayachandran

        Issue Links

          Activity

          Hide
          Thejas M Nair added a comment -

          This has been fixed in 0.14 release. Please open new jira if you see any issues.

          Show
          Thejas M Nair added a comment - This has been fixed in 0.14 release. Please open new jira if you see any issues.
          Hide
          Gunther Hagleitner added a comment -

          Committed to trunk and branch. Thanks Prasanth Jayachandran!

          Show
          Gunther Hagleitner added a comment - Committed to trunk and branch. Thanks Prasanth Jayachandran !
          Hide
          Harish Butani added a comment -

          +1 for 0.13

          Show
          Harish Butani added a comment - +1 for 0.13
          Hide
          Hive QA added a comment -

          Overall: +1 all checks pass

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12636758/HIVE-6714.3.patch

          SUCCESS: +1 5457 tests passed

          Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1959/testReport
          Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1959/console

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          

          This message is automatically generated.

          ATTACHMENT ID: 12636758

          Show
          Hive QA added a comment - Overall : +1 all checks pass Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12636758/HIVE-6714.3.patch SUCCESS: +1 5457 tests passed Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1959/testReport Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1959/console Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase This message is automatically generated. ATTACHMENT ID: 12636758
          Hide
          Prasanth Jayachandran added a comment -

          Reuploading for jenkins.

          Show
          Prasanth Jayachandran added a comment - Reuploading for jenkins.
          Hide
          Prasanth Jayachandran added a comment -

          Failed to initialize a variable caused the tests to fail. Fixed it in this patch.

          Show
          Prasanth Jayachandran added a comment - Failed to initialize a variable caused the tests to fail. Fixed it in this patch.
          Hide
          Hive QA added a comment -

          Overall: -1 at least one tests failed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12636374/HIVE-6714.2.patch

          ERROR: -1 due to 14 failed/errored test(s), 5443 tests executed
          Failed tests:

          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_select
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_avro_nullable_fields
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_columnarserde_create_shortcut
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_complex_types
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_complex_types_multi_single_reducer
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_input_columnarserde
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_input_dynamicserde
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_input_lazyserde
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_insert_overwrite_local_directory_1
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_orc_create
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_parquet_create
          org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_binary_map_queries
          org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_queries
          org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_auto_sortmerge_join_16
          

          Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1952/testReport
          Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1952/console

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests exited with: TestsFailedException: 14 tests failed
          

          This message is automatically generated.

          ATTACHMENT ID: 12636374

          Show
          Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12636374/HIVE-6714.2.patch ERROR: -1 due to 14 failed/errored test(s), 5443 tests executed Failed tests: org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_select org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_avro_nullable_fields org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_columnarserde_create_shortcut org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_complex_types org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_complex_types_multi_single_reducer org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_input_columnarserde org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_input_dynamicserde org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_input_lazyserde org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_insert_overwrite_local_directory_1 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_orc_create org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_parquet_create org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_binary_map_queries org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_queries org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_auto_sortmerge_join_16 Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1952/testReport Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1952/console Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 14 tests failed This message is automatically generated. ATTACHMENT ID: 12636374
          Hide
          Harish Butani added a comment -

          +1 for 0.13

          Show
          Harish Butani added a comment - +1 for 0.13
          Hide
          Prasanth Jayachandran added a comment -

          Reuploading again for jenkins.

          Show
          Prasanth Jayachandran added a comment - Reuploading again for jenkins.
          Hide
          Gunther Hagleitner added a comment -

          +1 LGTM

          Show
          Gunther Hagleitner added a comment - +1 LGTM
          Hide
          Prasanth Jayachandran added a comment -

          This patch addressed Gunther Hagleitner's review comments. The parse() method now eliminates the duplicate keys (duplicate key/value offsets are not stored).

          Show
          Prasanth Jayachandran added a comment - This patch addressed Gunther Hagleitner 's review comments. The parse() method now eliminates the duplicate keys (duplicate key/value offsets are not stored).
          Hide
          Gunther Hagleitner added a comment -

          I think a cleaner way would be to eliminate duplicate keys during parse stage. The deserialized structs would only have each key once. That way you don't have to carry both actual (deduped) and original size. Is that possible?

          Show
          Gunther Hagleitner added a comment - I think a cleaner way would be to eliminate duplicate keys during parse stage. The deserialized structs would only have each key once. That way you don't have to carry both actual (deduped) and original size. Is that possible?
          Hide
          Prasanth Jayachandran added a comment -

          This patch is generate on top of HIVE-6707

          Show
          Prasanth Jayachandran added a comment - This patch is generate on top of HIVE-6707

            People

            • Assignee:
              Prasanth Jayachandran
              Reporter:
              Prasanth Jayachandran
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development