Hive
  1. Hive
  2. HIVE-5916

No need to aggregate statistics collected via counter mechanism

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.13.0
    • Fix Version/s: 0.13.0
    • Component/s: Statistics
    • Labels:
      None

      Description

      This results in unnecessary computations and waste of cluster resources which is not required since aggregation of counter is anyway done by JobTracker.

      1. HIVE-5916.patch
        4 kB
        Ashutosh Chauhan
      2. HIVE-5916.2.patch
        22 kB
        Ashutosh Chauhan
      3. HIVE-5916.3.patch
        23 kB
        Ashutosh Chauhan
      4. HIVE-5916.4.patch
        52 kB
        Ashutosh Chauhan
      5. HIVE-5916.5.patch
        52 kB
        Ashutosh Chauhan

        Issue Links

          Activity

          Hide
          Ashutosh Chauhan added a comment -

          Committed to trunk. Thanks, Navis for review.

          Show
          Ashutosh Chauhan added a comment - Committed to trunk. Thanks, Navis for review.
          Hide
          Hive QA added a comment -

          Overall: +1 all checks pass

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12617112/HIVE-5916.5.patch

          SUCCESS: +1 4457 tests passed

          Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/526/testReport
          Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/526/console

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          

          This message is automatically generated.

          ATTACHMENT ID: 12617112

          Show
          Hive QA added a comment - Overall : +1 all checks pass Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12617112/HIVE-5916.5.patch SUCCESS: +1 4457 tests passed Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/526/testReport Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/526/console Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase This message is automatically generated. ATTACHMENT ID: 12617112
          Hide
          Ashutosh Chauhan added a comment -

          Hopefully, Hive QA will pick this up.

          Show
          Ashutosh Chauhan added a comment - Hopefully, Hive QA will pick this up.
          Hide
          Ashutosh Chauhan added a comment -

          Thanks, Navis for review. Updated patch with review comments. Also, fixed many whitespace changes.

          Show
          Ashutosh Chauhan added a comment - Thanks, Navis for review. Updated patch with review comments. Also, fixed many whitespace changes.
          Hide
          Navis added a comment -

          nit: formats. except that, +1

          Show
          Navis added a comment - nit: formats. except that, +1
          Hide
          Ashutosh Chauhan added a comment -
          Show
          Ashutosh Chauhan added a comment - Review request at https://reviews.apache.org/r/16026/
          Hide
          Ashutosh Chauhan added a comment -

          Patch is ready for review.
          Navis Major change in this patch is it changes the aggKey such that no aggregation is required on client side in StatsTask. This also shortens the key so that it won't run into limits. Now we just use dbName.TblName/p1=v1/ instead of long elaborate filesystem paths. This patch also lowers the number of unique counters required. Earlier they were num of Partitions * number of Tasks * number of Stats, now we just use num of Partitions * num of Tasks.
          This will conflict with HIVE-5936 in major fashion. Since, it has bunch of additional improvements, do you think it makes sense to get this one in first.

          Show
          Ashutosh Chauhan added a comment - Patch is ready for review. Navis Major change in this patch is it changes the aggKey such that no aggregation is required on client side in StatsTask. This also shortens the key so that it won't run into limits. Now we just use dbName.TblName/p1=v1/ instead of long elaborate filesystem paths. This patch also lowers the number of unique counters required. Earlier they were num of Partitions * number of Tasks * number of Stats, now we just use num of Partitions * num of Tasks. This will conflict with HIVE-5936 in major fashion. Since, it has bunch of additional improvements, do you think it makes sense to get this one in first.
          Hide
          Hive QA added a comment -

          Overall: -1 at least one tests failed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12617026/HIVE-5916.3.patch

          ERROR: -1 due to 1 failed/errored test(s), 4456 tests executed
          Failed tests:

          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_decimal_udf
          

          Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/514/testReport
          Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/514/console

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests exited with: TestsFailedException: 1 tests failed
          

          This message is automatically generated.

          ATTACHMENT ID: 12617026

          Show
          Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12617026/HIVE-5916.3.patch ERROR: -1 due to 1 failed/errored test(s), 4456 tests executed Failed tests: org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_decimal_udf Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/514/testReport Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/514/console Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 1 tests failed This message is automatically generated. ATTACHMENT ID: 12617026
          Hide
          Ashutosh Chauhan added a comment -

          Getting closer. Trigger Hive QA to see if this introduces any failures.

          Show
          Ashutosh Chauhan added a comment - Getting closer. Trigger Hive QA to see if this introduces any failures.
          Hide
          Ashutosh Chauhan added a comment -

          Attached wrong patch. Now with test-case.

          Show
          Ashutosh Chauhan added a comment - Attached wrong patch. Now with test-case.
          Hide
          Ashutosh Chauhan added a comment -

          Working patch with attached testcase. Need to clean-up to remove logging and need to do more testing.

          Show
          Ashutosh Chauhan added a comment - Working patch with attached testcase. Need to clean-up to remove logging and need to do more testing.
          Hide
          Ashutosh Chauhan added a comment -

          Unfinished patch illustrating the idea. More work needs to be done. Navis Can you take a look to finish this off, if you have time?

          Show
          Ashutosh Chauhan added a comment - Unfinished patch illustrating the idea. More work needs to be done. Navis Can you take a look to finish this off, if you have time?
          Hide
          Ashutosh Chauhan added a comment - - edited

          Currently from TableScanOp we publish statistics to JobTracker with aggrKey as counter group name, actual statistics type (numRows etc) as counter and value of statistics as counter value. We can simply use partition spec as counter group name, statistics as counter name and then we need not to do invoke stats aggregation from hive client when query finishes. This has following advantages:

          • Client don't need to do any aggregation. After retrieving statistics from JobTracker (via JobClient) it can directly add them to metastore.
          • It lowers memory footprint on JobTracker, since instead of having counters per task per partition, it will have counters per partition only.
          Show
          Ashutosh Chauhan added a comment - - edited Currently from TableScanOp we publish statistics to JobTracker with aggrKey as counter group name, actual statistics type (numRows etc) as counter and value of statistics as counter value. We can simply use partition spec as counter group name, statistics as counter name and then we need not to do invoke stats aggregation from hive client when query finishes. This has following advantages: Client don't need to do any aggregation. After retrieving statistics from JobTracker (via JobClient) it can directly add them to metastore. It lowers memory footprint on JobTracker, since instead of having counters per task per partition, it will have counters per partition only.

            People

            • Assignee:
              Ashutosh Chauhan
              Reporter:
              Ashutosh Chauhan
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development