Details

    • Type: Bug
    • Status: Closed
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.5.4
    • Component/s: None
    • Labels:
      None
    • Target Version/s:

      Description

      We are not publishing any version info to Timeline. This will be useful to compare different dags/apps over time and also to catch issues if needed.

      1. TEZ-1931.1.patch
        101 kB
        Hitesh Shah
      2. TEZ-1931.1.branch-0.5.patch
        12 kB
        Hitesh Shah

        Issue Links

          Activity

          Hide
          hitesh Hitesh Shah added a comment -

          Jonathan Eagles Should have a patch soon. Are you ok with putting this into 0.6.0 ?

          Show
          hitesh Hitesh Shah added a comment - Jonathan Eagles Should have a patch soon. Are you ok with putting this into 0.6.0 ?
          Hide
          jeagles Jonathan Eagles added a comment -

          This will be good to go into 0.6.0, Hitesh Shah

          Show
          jeagles Jonathan Eagles added a comment - This will be good to go into 0.6.0, Hitesh Shah
          Hide
          hitesh Hitesh Shah added a comment -

          Patch is slightly bigger as I noticed that the unit tests were present in the tez-yarn-timeline-history dir instead of tez-yarn-timeline-history-with-acls. Had to reverse the symlink as part of this patch.

          Show
          hitesh Hitesh Shah added a comment - Patch is slightly bigger as I noticed that the unit tests were present in the tez-yarn-timeline-history dir instead of tez-yarn-timeline-history-with-acls. Had to reverse the symlink as part of this patch.
          Hide
          hitesh Hitesh Shah added a comment -
          Show
          hitesh Hitesh Shah added a comment - Jonathan Eagles Prakash Ramachandran please review.
          Hide
          jeagles Jonathan Eagles added a comment -

          Trying this out now to verify correctness of publishing to timeline server. In addition, I'll file a follow-up JIRA to make this version available in the Tez UI.

          Show
          jeagles Jonathan Eagles added a comment - Trying this out now to verify correctness of publishing to timeline server. In addition, I'll file a follow-up JIRA to make this version available in the Tez UI.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12691422/TEZ-1931.1.patch
          against master revision 4c4bcdd.

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/13//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12691422/TEZ-1931.1.patch against master revision 4c4bcdd. -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/13//console This message is automatically generated.
          Hide
          jeagles Jonathan Eagles added a comment -

          +1. Verified the code is correct and that the output seems reasonable for a sample job. I was a little surprised at the build time format, but that seems to be correct as well.

          {
          
              "entities": [
                  {
                      "events": [ ],
                      "entitytype": "TEZ_APPLICATION",
                      "entity": "tez_application_1421187504058_0003",
                      "starttime": 1421188513349,
                      "domain": "Tez_ATS_application_1421187504058_0003",
                      "relatedentities": {
                          "TEZ_DAG_ID": [
                              "dag_1421187504058_0003_1"
                          ]
                      },
                      "primaryfilters": {
                          "user": [
                              "jeagles"
                          ]
                      },
                      "otherinfo": {
                          "config": { ...
                          },
                          "tezVersion": {
                              "buildTime": "20150113-1603",
                              "revision": "83261659809f7904b786c9c81def4451dca27078",
                              "version": "0.7.0-SNAPSHOT"
                          }
                      }
                  }
              ]
          }
          
          Show
          jeagles Jonathan Eagles added a comment - +1. Verified the code is correct and that the output seems reasonable for a sample job. I was a little surprised at the build time format, but that seems to be correct as well. { "entities" : [ { "events" : [ ], "entitytype" : "TEZ_APPLICATION" , "entity" : "tez_application_1421187504058_0003" , "starttime" : 1421188513349, "domain" : "Tez_ATS_application_1421187504058_0003" , "relatedentities" : { "TEZ_DAG_ID" : [ "dag_1421187504058_0003_1" ] }, "primaryfilters" : { "user" : [ "jeagles" ] }, "otherinfo" : { "config" : { ... }, "tezVersion" : { "buildTime" : "20150113-1603" , "revision" : "83261659809f7904b786c9c81def4451dca27078" , "version" : "0.7.0-SNAPSHOT" } } } ] }
          Hide
          hitesh Hitesh Shah added a comment - - edited

          Jonathan Eagles Thanks. Any concerns/suggestions on the timestamp format? I can try and switch that around to a more parseable format if that helps?

          Show
          hitesh Hitesh Shah added a comment - - edited Jonathan Eagles Thanks. Any concerns/suggestions on the timestamp format? I can try and switch that around to a more parseable format if that helps?
          Hide
          jeagles Jonathan Eagles added a comment -

          Hitesh Shah, not sure about a better format. Most of the timestamps in timeline entities are in epoch so far but I'm not sure we have the granularity.

          Show
          jeagles Jonathan Eagles added a comment - Hitesh Shah , not sure about a better format. Most of the timestamps in timeline entities are in epoch so far but I'm not sure we have the granularity.
          Hide
          hitesh Hitesh Shah added a comment -

          Looking at the build.time support in maven, it only supports format specified by http://docs.oracle.com/javase/6/docs/api/java/text/SimpleDateFormat.html. This does not support seconds since epoch as a format.

          Show
          hitesh Hitesh Shah added a comment - Looking at the build.time support in maven, it only supports format specified by http://docs.oracle.com/javase/6/docs/api/java/text/SimpleDateFormat.html . This does not support seconds since epoch as a format.
          Hide
          hitesh Hitesh Shah added a comment - - edited

          Jonathan Eagles Given your +1, I will go ahead and commit this patch later today. If there are recommendations on the timestamp change, I will file a separate jira to modify the build as needed. Let me know if there are any concerns following this path.

          Show
          hitesh Hitesh Shah added a comment - - edited Jonathan Eagles Given your +1, I will go ahead and commit this patch later today. If there are recommendations on the timestamp change, I will file a separate jira to modify the build as needed. Let me know if there are any concerns following this path.
          Hide
          jeagles Jonathan Eagles added a comment -

          Makes sense. Thanks for investigating the build date format details

          Show
          jeagles Jonathan Eagles added a comment - Makes sense. Thanks for investigating the build date format details
          Hide
          hitesh Hitesh Shah added a comment -

          Committed to master and branch 0.6

          Done as 3 commits to ensure all the moves and symlink changes happened correctly.

          Show
          hitesh Hitesh Shah added a comment - Committed to master and branch 0.6 Done as 3 commits to ensure all the moves and symlink changes happened correctly.
          Hide
          hitesh Hitesh Shah added a comment -

          Seems like a safe enough and useful option in case anyone wants to trying running the UI against a 0.5 release.

          Jonathan Eagles Prakash Ramachandran review please.

          Show
          hitesh Hitesh Shah added a comment - Seems like a safe enough and useful option in case anyone wants to trying running the UI against a 0.5 release. Jonathan Eagles Prakash Ramachandran review please.
          Show
          hitesh Hitesh Shah added a comment - Jonathan Eagles Prakash Ramachandran Siddharth Seth Rajesh Balamohan ping.
          Hide
          sseth Siddharth Seth added a comment -

          +1 for the 0.5 patch.

          Show
          sseth Siddharth Seth added a comment - +1 for the 0.5 patch.
          Hide
          hitesh Hitesh Shah added a comment -

          Thanks Siddharth Seth - committed to branch 0.5

          Show
          hitesh Hitesh Shah added a comment - Thanks Siddharth Seth - committed to branch 0.5
          Hide
          hitesh Hitesh Shah added a comment -

          Closing issue as 0.5.4, 0.6.1 and 0.7.0 have been released.

          Show
          hitesh Hitesh Shah added a comment - Closing issue as 0.5.4, 0.6.1 and 0.7.0 have been released.

            People

            • Assignee:
              hitesh Hitesh Shah
              Reporter:
              hitesh Hitesh Shah
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development