Hive
  1. Hive
  2. HIVE-3235

Support jobId notification for MaredTasks

    Details

    • Type: Improvement Improvement
    • Status: Patch Available
    • Priority: Trivial Trivial
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: Diagnosability
    • Labels:
      None

      Description

      JobId for each MapredTask is requested for monitoring purpose.

        Activity

        Hide
        Hive QA added a comment -

        Overall: -1 no tests executed

        Here are the results of testing the latest attachment:
        https://issues.apache.org/jira/secure/attachment/12570783/HIVE-3235.D4017.2.patch

        Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3718/testReport
        Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3718/console
        Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-3718/

        Messages:

        Executing org.apache.hive.ptest.execution.PrepPhase
        Tests exited with: NonZeroExitCodeException
        Command 'bash /data/hive-ptest/working/scratch/source-prep.sh' failed with exit status 1 and output '+ [[ -n /usr/java/jdk1.7.0_45-cloudera ]]
        + export JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera
        + JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera
        + export PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin
        + PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin
        + export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
        + ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
        + export 'M2_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost -Dhttp.proxyPort=3128'
        + M2_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost -Dhttp.proxyPort=3128'
        + cd /data/hive-ptest/working/
        + tee /data/hive-ptest/logs/PreCommit-HIVE-TRUNK-Build-3718/source-prep.txt
        + [[ false == \t\r\u\e ]]
        + mkdir -p maven ivy
        + [[ git = \s\v\n ]]
        + [[ git = \g\i\t ]]
        + [[ -z master ]]
        + [[ -d apache-github-source-source ]]
        + [[ ! -d apache-github-source-source/.git ]]
        + [[ ! -d apache-github-source-source ]]
        + cd apache-github-source-source
        + git fetch origin
        + git reset --hard HEAD
        HEAD is now at 8fb8052 HIVE-10571 : HiveMetaStoreClient should close existing thrift connection before its reconnect (Chaoyu Tang via Szehon)
        + git clean -f -d
        Removing common/src/java/org/apache/hadoop/hive/conf/HiveConf.java.orig
        + git checkout master
        Already on 'master'
        + git reset --hard origin/master
        HEAD is now at 8fb8052 HIVE-10571 : HiveMetaStoreClient should close existing thrift connection before its reconnect (Chaoyu Tang via Szehon)
        + git merge --ff-only origin/master
        Already up-to-date.
        + git gc
        + patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh
        + patchFilePath=/data/hive-ptest/working/scratch/build.patch
        + [[ -f /data/hive-ptest/working/scratch/build.patch ]]
        + chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh
        + /data/hive-ptest/working/scratch/smart-apply-patch.sh /data/hive-ptest/working/scratch/build.patch
        The patch does not appear to apply with p0, p1, or p2
        + exit 1
        '
        

        This message is automatically generated.

        ATTACHMENT ID: 12570783 - PreCommit-HIVE-TRUNK-Build

        Show
        Hive QA added a comment - Overall : -1 no tests executed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12570783/HIVE-3235.D4017.2.patch Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3718/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3718/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-3718/ Messages: Executing org.apache.hive.ptest.execution.PrepPhase Tests exited with: NonZeroExitCodeException Command 'bash /data/hive-ptest/working/scratch/source-prep.sh' failed with exit status 1 and output '+ [[ -n /usr/java/jdk1.7.0_45-cloudera ]] + export JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera + JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera + export PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin + PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin + export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m ' + ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m ' + export 'M2_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost -Dhttp.proxyPort=3128' + M2_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost -Dhttp.proxyPort=3128' + cd /data/hive-ptest/working/ + tee /data/hive-ptest/logs/PreCommit-HIVE-TRUNK-Build-3718/source-prep.txt + [[ false == \t\r\u\e ]] + mkdir -p maven ivy + [[ git = \s\v\n ]] + [[ git = \g\i\t ]] + [[ -z master ]] + [[ -d apache-github-source-source ]] + [[ ! -d apache-github-source-source/.git ]] + [[ ! -d apache-github-source-source ]] + cd apache-github-source-source + git fetch origin + git reset --hard HEAD HEAD is now at 8fb8052 HIVE-10571 : HiveMetaStoreClient should close existing thrift connection before its reconnect (Chaoyu Tang via Szehon) + git clean -f -d Removing common/src/java/org/apache/hadoop/hive/conf/HiveConf.java.orig + git checkout master Already on 'master' + git reset --hard origin/master HEAD is now at 8fb8052 HIVE-10571 : HiveMetaStoreClient should close existing thrift connection before its reconnect (Chaoyu Tang via Szehon) + git merge --ff-only origin/master Already up-to-date. + git gc + patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh + patchFilePath=/data/hive-ptest/working/scratch/build.patch + [[ -f /data/hive-ptest/working/scratch/build.patch ]] + chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh + /data/hive-ptest/working/scratch/smart-apply-patch.sh /data/hive-ptest/working/scratch/build.patch The patch does not appear to apply with p0, p1, or p2 + exit 1 ' This message is automatically generated. ATTACHMENT ID: 12570783 - PreCommit-HIVE-TRUNK-Build
        Hide
        Aihua Xu added a comment -

        Navis Can we get this reviewed and committed?

        Show
        Aihua Xu added a comment - Navis Can we get this reviewed and committed?
        Hide
        Phabricator added a comment -

        navis updated the revision "HIVE-3235 [jira] Support jobId notification for MaredTasks".

        Addressed comments & rebased to trunk

        Reviewers: JIRA

        REVISION DETAIL
        https://reviews.facebook.net/D4017

        CHANGE SINCE LAST DIFF
        https://reviews.facebook.net/D4017?vs=12885&id=28605#toc

        AFFECTED FILES
        common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
        ql/src/java/org/apache/hadoop/hive/ql/exec/ExecDriver.java
        ql/src/java/org/apache/hadoop/hive/ql/exec/Task.java

        To: JIRA, navis
        Cc: zhenxiao

        Show
        Phabricator added a comment - navis updated the revision " HIVE-3235 [jira] Support jobId notification for MaredTasks". Addressed comments & rebased to trunk Reviewers: JIRA REVISION DETAIL https://reviews.facebook.net/D4017 CHANGE SINCE LAST DIFF https://reviews.facebook.net/D4017?vs=12885&id=28605#toc AFFECTED FILES common/src/java/org/apache/hadoop/hive/conf/HiveConf.java ql/src/java/org/apache/hadoop/hive/ql/exec/ExecDriver.java ql/src/java/org/apache/hadoop/hive/ql/exec/Task.java To: JIRA, navis Cc: zhenxiao
        Hide
        Phabricator added a comment -

        zhenxiao has commented on the revision "HIVE-3235 [jira] Support jobId notification for MaredTasks".

        INLINE COMMENTS
        ql/src/java/org/apache/hadoop/hive/ql/exec/Task.java:156 log error instead of warning?
        ql/src/java/org/apache/hadoop/hive/ql/exec/Task.java:152 log error instead of warning?

        REVISION DETAIL
        https://reviews.facebook.net/D4017

        To: JIRA, navis
        Cc: zhenxiao

        Show
        Phabricator added a comment - zhenxiao has commented on the revision " HIVE-3235 [jira] Support jobId notification for MaredTasks". INLINE COMMENTS ql/src/java/org/apache/hadoop/hive/ql/exec/Task.java:156 log error instead of warning? ql/src/java/org/apache/hadoop/hive/ql/exec/Task.java:152 log error instead of warning? REVISION DETAIL https://reviews.facebook.net/D4017 To: JIRA, navis Cc: zhenxiao
        Hide
        Zhenxiao Luo added a comment -

        comments on Phabricator

        Show
        Zhenxiao Luo added a comment - comments on Phabricator
        Hide
        Zhenxiao Luo added a comment -

        How about file a Hadoop Jira to get JobStartNotification, and make this Jira depends on that one?

        Show
        Zhenxiao Luo added a comment - How about file a Hadoop Jira to get JobStartNotification, and make this Jira depends on that one?
        Hide
        Navis added a comment -

        End notification is delegated to hadoop, but I couldn't find start notification in that.
        Modifying hadoop could be better but I simply added job start notification in ExecDriver (notifying code is also copied from hadoop except retrying policy).

        Show
        Navis added a comment - End notification is delegated to hadoop, but I couldn't find start notification in that. Modifying hadoop could be better but I simply added job start notification in ExecDriver (notifying code is also copied from hadoop except retrying policy).
        Hide
        Edward Capriolo added a comment -

        Q: How is this different then hadoop's job notification? Can we leverage that code?

        Show
        Edward Capriolo added a comment - Q: How is this different then hadoop's job notification? Can we leverage that code?
        Hide
        Navis added a comment -
        Show
        Navis added a comment - https://reviews.facebook.net/D4017

          People

          • Assignee:
            Navis
            Reporter:
            Navis
          • Votes:
            1 Vote for this issue
            Watchers:
            8 Start watching this issue

            Dates

            • Created:
              Updated:

              Development