Hive
  1. Hive
  2. HIVE-305

Port Hadoop streaming's counters/status reporters to Hive Transforms

    Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.12.0
    • Component/s: Query Processor
    • Labels:
      None

      Description

      https://issues.apache.org/jira/browse/HADOOP-1328

      " Introduced a way for a streaming process to update global counters and status using stderr stream to emit information. Use "reporter:counter:<group>,<counter>,<amount> " to update a counter. Use "reporter:status:<message>" to update status. "

      1. HIVE-305.patch.txt
        5 kB
        Edward Capriolo
      2. hive-305.3.diff.txt
        4 kB
        Edward Capriolo
      3. HIVE-305.2.patch
        3 kB
        Guo Hongjie
      4. HIVE-305.1.patch
        2 kB
        Guo Hongjie

        Issue Links

          Activity

          Venky Iyer created issue -
          Carl Steinbach made changes -
          Field Original Value New Value
          Component/s Query Processor [ 12312586 ]
          Guo Hongjie made changes -
          Attachment HIVE-305.1.patch [ 12564135 ]
          Guo Hongjie made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Release Note I use the trunk to create this patch . http://svn.apache.org/repos/asf/hive/trunk
          Carl Steinbach made changes -
          Link This issue is duplicated by HIVE-1649 [ HIVE-1649 ]
          Hide
          Carl Steinbach added a comment -

          @Guo: please submit a review request on reviewboard or phabricator. Thanks.

          Show
          Carl Steinbach added a comment - @Guo: please submit a review request on reviewboard or phabricator. Thanks.
          Carl Steinbach made changes -
          Assignee Guo Hongjie [ fu2you ]
          Guo Hongjie made changes -
          Attachment HIVE-305.2.patch [ 12564541 ]
          Hide
          Guo Hongjie added a comment -

          thanks @Carl,do you mean "submit patch" button? I cannot find any button about "review request".

          Show
          Guo Hongjie added a comment - thanks @Carl,do you mean "submit patch" button? I cannot find any button about "review request".
          Hide
          Guo Hongjie added a comment -

          i found that in the mail list.thanks

          Show
          Guo Hongjie added a comment - i found that in the mail list.thanks
          Hide
          Carl Steinbach added a comment -

          @Guo: please include a link to the review request. Thanks.

          Show
          Carl Steinbach added a comment - @Guo: please include a link to the review request. Thanks.
          Hide
          Carl Steinbach added a comment -
          Show
          Carl Steinbach added a comment - review request: https://reviews.apache.org/r/8933/
          Hide
          Edward Capriolo added a comment -

          Guo Hongjie Guo, I am sorry this patch has sat in jira for this long. I will help you drive this through. I am going to rebase your patch a bit, because I would like to add a conf variable to turn this on/off.

          Show
          Edward Capriolo added a comment - Guo Hongjie Guo, I am sorry this patch has sat in jira for this long. I will help you drive this through. I am going to rebase your patch a bit, because I would like to add a conf variable to turn this on/off.
          Edward Capriolo made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Edward Capriolo made changes -
          Attachment hive-305.3.diff.txt [ 12592702 ]
          Edward Capriolo made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Hide
          Edward Capriolo added a comment - - edited

          I corrected the status typo and did a minor refactor on Guo's work.

          Show
          Edward Capriolo added a comment - - edited I corrected the status typo and did a minor refactor on Guo's work.
          Edward Capriolo made changes -
          Attachment HIVE-305.patch.txt [ 12593116 ]
          Hide
          Edward Capriolo added a comment -

          Added default configuration values to the xml file.

          Show
          Edward Capriolo added a comment - Added default configuration values to the xml file.
          Hide
          Hive QA added a comment -

          Overall: -1 build exited with an error

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12593116/HIVE-305.patch.txt

          ERROR: -1 due to 1 failed/errored test(s)
          Failed tests:
          org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_bucketmapjoin6

          Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/93/testReport
          Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/93/console

          Messages:
          Executing org.apache.hive.ptest.execution.CleanupPhase
          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests failed with: TestsFailedException: 1 tests failed

          This message is automatically generated.

          Show
          Hive QA added a comment - Overall : -1 build exited with an error Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12593116/HIVE-305.patch.txt ERROR: -1 due to 1 failed/errored test(s) Failed tests: org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_bucketmapjoin6 Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/93/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/93/console Messages: Executing org.apache.hive.ptest.execution.CleanupPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests failed with: TestsFailedException: 1 tests failed This message is automatically generated.
          Hide
          Brock Noland added a comment -

          The latest patch HIVE-305.patch.txt looks good and that test which failed is flaky. The only question I have is, shouldn't the configuration variables start with hive? as opposed to stream?

          Show
          Brock Noland added a comment - The latest patch HIVE-305 .patch.txt looks good and that test which failed is flaky. The only question I have is, shouldn't the configuration variables start with hive? as opposed to stream?
          Hide
          Edward Capriolo added a comment -

          In this patch we are using the same variable as the upstream hadoop.

          public MRErrorThread() {
          +      this.reporterPrefix = job_.get("stream.stderr.reporter.prefix", "reporter:");
          

          I am ok with this because the meaning is the same, and duplicating hadoop variables is something we are trying to avoid. (like the ticket on deprecating hive's speculative execution settings.

          Show
          Edward Capriolo added a comment - In this patch we are using the same variable as the upstream hadoop. public MRErrorThread() { + this .reporterPrefix = job_.get( "stream.stderr.reporter.prefix" , "reporter:" ); I am ok with this because the meaning is the same, and duplicating hadoop variables is something we are trying to avoid. (like the ticket on deprecating hive's speculative execution settings.
          Hide
          Brock Noland added a comment -

          Gotcha. Sounds good

          Show
          Brock Noland added a comment - Gotcha. Sounds good
          Hide
          Edward Capriolo added a comment -

          If someone does the commit please make sure Guo Hongjie is listed as he did most of the work as the assignee.

          Show
          Edward Capriolo added a comment - If someone does the commit please make sure Guo Hongjie is listed as he did most of the work as the assignee.
          Hide
          Hive QA added a comment -

          Overall: +1 all checks pass

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12593116/HIVE-305.patch.txt

          SUCCESS: +1 2646 tests passed

          Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/135/testReport
          Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/135/console

          Messages:

          Executing org.apache.hive.ptest.execution.CleanupPhase
          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          

          This message is automatically generated.

          Show
          Hive QA added a comment - Overall : +1 all checks pass Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12593116/HIVE-305.patch.txt SUCCESS: +1 2646 tests passed Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/135/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/135/console Messages: Executing org.apache.hive.ptest.execution.CleanupPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase This message is automatically generated.
          Hide
          Edward Capriolo added a comment -

          ::crickets::

          Show
          Edward Capriolo added a comment - ::crickets::
          Hide
          Brock Noland added a comment -

          +1

          Show
          Brock Noland added a comment - +1
          Brock Noland made changes -
          Release Note I use the trunk to create this patch . http://svn.apache.org/repos/asf/hive/trunk
          Hide
          Brock Noland added a comment -

          Committed to trunk! Thank you for your contribution Guo and Edward!

          Show
          Brock Noland added a comment - Committed to trunk! Thank you for your contribution Guo and Edward!
          Brock Noland made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Fix Version/s 0.12.0 [ 12324312 ]
          Resolution Fixed [ 1 ]
          Hide
          Ashutosh Chauhan added a comment -

          This issue has been fixed and released as part of 0.12 release. If you find further issues, please create a new jira and link it to this one.

          Show
          Ashutosh Chauhan added a comment - This issue has been fixed and released as part of 0.12 release. If you find further issues, please create a new jira and link it to this one.
          Ashutosh Chauhan made changes -
          Status Resolved [ 5 ] Closed [ 6 ]

            People

            • Assignee:
              Guo Hongjie
              Reporter:
              Venky Iyer
            • Votes:
              1 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development