Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-3649

Job End notification gives an error on calling back.

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.23.1
    • Fix Version/s: 0.23.1
    • Component/s: mrv2
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      When calling job end notification for oozie the AM fails with the following trace:

      2012-01-09 23:45:41,732 WARN [AsyncDispatcher event handler] org.mortbay.log: Job end notification to http://HOST:11000/oozie/v0/callback?id=0000000-120109234442311-oozie-oozi-W@mr-node&status=SUCCEEDED& failed
      java.net.UnknownServiceException: no content-type
      	at java.net.URLConnection.getContentHandler(URLConnection.java:1192)
      	at java.net.URLConnection.getContent(URLConnection.java:689)
      	at org.apache.hadoop.mapreduce.v2.app.JobEndNotifier.notifyURLOnce(JobEndNotifier.java:95)
      	at org.apache.hadoop.mapreduce.v2.app.JobEndNotifier.notify(JobEndNotifier.java:139)
      	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobFinishEventHandler.handle(MRAppMaster.java:388)
      	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobFinishEventHandler.handle(MRAppMaster.java:375)
      	at org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:125)
      	at org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:82)
      
      1. MAPREDUCE-3649.patch
        2 kB
        Ravi Prakash
      2. MAPREDUCE-3649.patch
        3 kB
        Ravi Prakash

        Activity

        Hide
        Alejandro Abdelnur added a comment -

        Looking at the code in the JobNotifier class:

              URLConnection conn = urlToNotify.openConnection();
              conn.setConnectTimeout(5*1000);
              conn.setReadTimeout(5*1000);
              conn.setAllowUserInteraction(false);
              InputStream is = conn.getInputStream();
              conn.getContent();
              is.close();
        

        Instead it should be

        
              HttpURLConnection conn = (HttpURLConnection) urlToNotify.openConnection();
              conn.setConnectTimeout(5*1000);
              conn.setReadTimeout(5*1000);
              conn.setAllowUserInteraction(false);
              if (conn.getResponseCode() != HttpURLConnection.HTTP_OK) {
                //LOG WARNING
              }
        
        Show
        Alejandro Abdelnur added a comment - Looking at the code in the JobNotifier class: URLConnection conn = urlToNotify.openConnection(); conn.setConnectTimeout(5*1000); conn.setReadTimeout(5*1000); conn.setAllowUserInteraction( false ); InputStream is = conn.getInputStream(); conn.getContent(); is.close(); Instead it should be HttpURLConnection conn = (HttpURLConnection) urlToNotify.openConnection(); conn.setConnectTimeout(5*1000); conn.setReadTimeout(5*1000); conn.setAllowUserInteraction( false ); if (conn.getResponseCode() != HttpURLConnection.HTTP_OK) { //LOG WARNING }
        Hide
        Ravi Prakash added a comment -

        Agreed! We saw the same exception in MAPREDUCE-3484 . I was guessing that the notification did happen, although the server didn't return any content. This would be a good fix though.

        +1 to the change

        Attaching a patch which I tested, sends out the HTTP notification. I haven't checked if it warns about the no-content, but from the code it seems it shouldn't =D

        Show
        Ravi Prakash added a comment - Agreed! We saw the same exception in MAPREDUCE-3484 . I was guessing that the notification did happen, although the server didn't return any content. This would be a good fix though. +1 to the change Attaching a patch which I tested, sends out the HTTP notification. I haven't checked if it warns about the no-content, but from the code it seems it shouldn't =D
        Hide
        Ravi Prakash added a comment -

        Sneaking a couple of javadoc changes inside this patch.

        Show
        Ravi Prakash added a comment - Sneaking a couple of javadoc changes inside this patch.
        Hide
        Mahadev konar added a comment -

        Correcting the assignment.

        Show
        Mahadev konar added a comment - Correcting the assignment.
        Hide
        Mahadev konar added a comment -

        @Ravi,
        Did you test out the patch on a cluster?

        Show
        Mahadev konar added a comment - @Ravi, Did you test out the patch on a cluster?
        Hide
        Mahadev konar added a comment -

        Nevermind,
        I ran the patch with an oozie cluster. It works fine. Ill go ahead and check this in.

        Show
        Mahadev konar added a comment - Nevermind, I ran the patch with an oozie cluster. It works fine. Ill go ahead and check this in.
        Hide
        Mahadev konar added a comment -

        I just committed this. Thanks Ravi and Alejandro!

        Show
        Mahadev konar added a comment - I just committed this. Thanks Ravi and Alejandro!
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-0.23-Commit #371 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/371/)
        MAPREDUCE-3649. Job End notification gives an error on calling back. (Ravi Prakash via mahadev) - Merging r1232126 from trunk.

        mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1232128
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/JobEndNotifier.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Commit #371 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/371/ ) MAPREDUCE-3649 . Job End notification gives an error on calling back. (Ravi Prakash via mahadev) - Merging r1232126 from trunk. mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1232128 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/JobEndNotifier.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk-Commit #1620 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1620/)
        MAPREDUCE-3649. Job End notification gives an error on calling back. (Ravi Prakash via mahadev)

        mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1232126
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/JobEndNotifier.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #1620 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1620/ ) MAPREDUCE-3649 . Job End notification gives an error on calling back. (Ravi Prakash via mahadev) mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1232126 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/JobEndNotifier.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-trunk-Commit #1547 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1547/)
        MAPREDUCE-3649. Job End notification gives an error on calling back. (Ravi Prakash via mahadev)

        mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1232126
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/JobEndNotifier.java
        Show
        Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #1547 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1547/ ) MAPREDUCE-3649 . Job End notification gives an error on calling back. (Ravi Prakash via mahadev) mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1232126 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/JobEndNotifier.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-0.23-Commit #381 (See https://builds.apache.org/job/Hadoop-Common-0.23-Commit/381/)
        MAPREDUCE-3649. Job End notification gives an error on calling back. (Ravi Prakash via mahadev) - Merging r1232126 from trunk.

        mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1232128
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/JobEndNotifier.java
        Show
        Hudson added a comment - Integrated in Hadoop-Common-0.23-Commit #381 (See https://builds.apache.org/job/Hadoop-Common-0.23-Commit/381/ ) MAPREDUCE-3649 . Job End notification gives an error on calling back. (Ravi Prakash via mahadev) - Merging r1232126 from trunk. mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1232128 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/JobEndNotifier.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-0.23-Commit #394 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/394/)
        MAPREDUCE-3649. Job End notification gives an error on calling back. (Ravi Prakash via mahadev) - Merging r1232126 from trunk.

        mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1232128
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/JobEndNotifier.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-0.23-Commit #394 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/394/ ) MAPREDUCE-3649 . Job End notification gives an error on calling back. (Ravi Prakash via mahadev) - Merging r1232126 from trunk. mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1232128 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/JobEndNotifier.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk-Commit #1565 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1565/)
        MAPREDUCE-3649. Job End notification gives an error on calling back. (Ravi Prakash via mahadev)

        mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1232126
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/JobEndNotifier.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #1565 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1565/ ) MAPREDUCE-3649 . Job End notification gives an error on calling back. (Ravi Prakash via mahadev) mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1232126 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/JobEndNotifier.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk #928 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/928/)
        MAPREDUCE-3649. Job End notification gives an error on calling back. (Ravi Prakash via mahadev)

        mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1232126
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/JobEndNotifier.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #928 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/928/ ) MAPREDUCE-3649 . Job End notification gives an error on calling back. (Ravi Prakash via mahadev) mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1232126 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/JobEndNotifier.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-0.23-Build #141 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/141/)
        MAPREDUCE-3649. Job End notification gives an error on calling back. (Ravi Prakash via mahadev) - Merging r1232126 from trunk.

        mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1232128
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/JobEndNotifier.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Build #141 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/141/ ) MAPREDUCE-3649 . Job End notification gives an error on calling back. (Ravi Prakash via mahadev) - Merging r1232126 from trunk. mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1232128 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/JobEndNotifier.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-0.23-Build #163 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/163/)
        MAPREDUCE-3649. Job End notification gives an error on calling back. (Ravi Prakash via mahadev) - Merging r1232126 from trunk.

        mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1232128
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/JobEndNotifier.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-0.23-Build #163 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/163/ ) MAPREDUCE-3649 . Job End notification gives an error on calling back. (Ravi Prakash via mahadev) - Merging r1232126 from trunk. mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1232128 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/JobEndNotifier.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #961 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/961/)
        MAPREDUCE-3649. Job End notification gives an error on calling back. (Ravi Prakash via mahadev)

        mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1232126
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/JobEndNotifier.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #961 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/961/ ) MAPREDUCE-3649 . Job End notification gives an error on calling back. (Ravi Prakash via mahadev) mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1232126 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/JobEndNotifier.java

          People

          • Assignee:
            Ravi Prakash
            Reporter:
            Mahadev konar
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development