Uploaded image for project: 'Oozie'
  1. Oozie
  2. OOZIE-2251

Expose instrumental matrices in Realtime Graphing tool

    Details

    • Type: New Feature
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 4.3.0
    • Component/s: monitoring
    • Labels:
      None

      Description

      We have been logging so many important matrices in oozie-instrumentation.log . These information is very useful for oozie functional monitoring. But it is always difficult to get the meaning from flat file. If we expose this information on some graphing tool, We can get the lot of meaning out of it and can take some actions based on it.

      1. OOZIE-2251-v0.patch
        14 kB
        Narayan Periwal
      2. OOZIE-2251-v1.patch
        14 kB
        Narayan Periwal
      3. OOZIE-2251-v10.patch
        15 kB
        Narayan Periwal
      4. OOZIE-2251-v11.patch
        16 kB
        Narayan Periwal
      5. OOZIE-2251-v2.patch
        14 kB
        Narayan Periwal
      6. OOZIE-2251-v3.patch
        14 kB
        Narayan Periwal
      7. OOZIE-2251-v4.patch
        13 kB
        Narayan Periwal
      8. OOZIE-2251-v5.patch
        8 kB
        Narayan Periwal
      9. OOZIE-2251-v6.patch
        12 kB
        Narayan Periwal
      10. OOZIE-2251-v7.patch
        13 kB
        Narayan Periwal
      11. OOZIE-2251-v8.patch
        13 kB
        Narayan Periwal
      12. OOZIE-2251-v9.patch
        15 kB
        Narayan Periwal

        Issue Links

          Activity

          Hide
          rkanter Robert Kanter added a comment -

          Closing issue; Oozie 4.3.0 is released.

          Show
          rkanter Robert Kanter added a comment - Closing issue; Oozie 4.3.0 is released.
          Hide
          nperiwal Narayan Periwal added a comment -
          Show
          nperiwal Narayan Periwal added a comment - Thanks Robert Kanter
          Hide
          rkanter Robert Kanter added a comment -

          Thanks Narayan. Committed to trunk!

          Show
          rkanter Robert Kanter added a comment - Thanks Narayan. Committed to trunk!
          Hide
          rkanter Robert Kanter added a comment -

          Jenkins seems to be broken for some reason. I ran a bunch of the related tests locally again, and they all pass now.

          +1

          Show
          rkanter Robert Kanter added a comment - Jenkins seems to be broken for some reason. I ran a bunch of the related tests locally again, and they all pass now. +1
          Hide
          hadoopqa Hadoop QA added a comment -

          Testing JIRA OOZIE-2251

          Cleaning local git workspace

          ----------------------------

          +1 PATCH_APPLIES
          +1 CLEAN
          +1 RAW_PATCH_ANALYSIS
          . +1 the patch does not introduce any @author tags
          . +1 the patch does not introduce any tabs
          . +1 the patch does not introduce any trailing spaces
          . +1 the patch does not introduce any line longer than 132
          . +1 the patch does adds/modifies 1 testcase(s)
          +1 RAT
          . +1 the patch does not seem to introduce new RAT warnings
          +1 JAVADOC
          . +1 the patch does not seem to introduce new Javadoc warnings
          +1 COMPILE
          . +1 HEAD compiles
          . +1 patch compiles
          . +1 the patch does not seem to introduce new javac warnings
          +1 BACKWARDS_COMPATIBILITY
          . +1 the patch does not change any JPA Entity/Colum/Basic/Lob/Transient annotations
          . +1 the patch does not modify JPA files
          -1 TESTS - patch does not compile, cannot run testcases
          +1 DISTRO
          . +1 distro tarball builds with the patch

          ----------------------------
          -1 Overall result, please check the reported -1(s)

          The full output of the test-patch run is available at

          . https://builds.apache.org/job/oozie-trunk-precommit-build/2580/

          Show
          hadoopqa Hadoop QA added a comment - Testing JIRA OOZIE-2251 Cleaning local git workspace ---------------------------- +1 PATCH_APPLIES +1 CLEAN +1 RAW_PATCH_ANALYSIS . +1 the patch does not introduce any @author tags . +1 the patch does not introduce any tabs . +1 the patch does not introduce any trailing spaces . +1 the patch does not introduce any line longer than 132 . +1 the patch does adds/modifies 1 testcase(s) +1 RAT . +1 the patch does not seem to introduce new RAT warnings +1 JAVADOC . +1 the patch does not seem to introduce new Javadoc warnings +1 COMPILE . +1 HEAD compiles . +1 patch compiles . +1 the patch does not seem to introduce new javac warnings +1 BACKWARDS_COMPATIBILITY . +1 the patch does not change any JPA Entity/Colum/Basic/Lob/Transient annotations . +1 the patch does not modify JPA files -1 TESTS - patch does not compile, cannot run testcases +1 DISTRO . +1 distro tarball builds with the patch ---------------------------- -1 Overall result, please check the reported -1(s) The full output of the test-patch run is available at . https://builds.apache.org/job/oozie-trunk-precommit-build/2580/
          Hide
          nperiwal Narayan Periwal added a comment -

          Robert Kanter, thanks for the reply. I have resolved the test case failure and updated the patch. As you told, it was a minor change in the test.

          Show
          nperiwal Narayan Periwal added a comment - Robert Kanter , thanks for the reply. I have resolved the test case failure and updated the patch. As you told, it was a minor change in the test.
          Hide
          rkanter Robert Kanter added a comment -

          Also, don't worry about RB for the next version, you can just post it here. It's probably some simple change in the test.

          Show
          rkanter Robert Kanter added a comment - Also, don't worry about RB for the next version, you can just post it here. It's probably some simple change in the test.
          Hide
          rkanter Robert Kanter added a comment -

          I was just running the relevant unit tests, and it looks like TestMetricsInstrumentation has a bunch of failures now. Narayan Periwal, can you take a look and make a new patch? I promise I won't take as long to review it this time

          Looks like they're all failing with

          testInstrumentationTimer(org.apache.oozie.util.TestMetricsInstrumentation)  Time elapsed: 0.012 sec  <<< ERROR!
          java.lang.NullPointerException
                  at org.apache.oozie.service.ConfigurationService.getBoolean(ConfigurationService.java:524)
                  at org.apache.oozie.util.MetricsInstrumentation.<init>(MetricsInstrumentation.java:105)
                  at org.apache.oozie.util.TestMetricsInstrumentation.testInstrumentationTimer(TestMetricsInstrumentation.java:62)
                  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
                  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
                  at java.lang.reflect.Method.invoke(Method.java:606)
                  at junit.framework.TestCase.runTest(TestCase.java:168)
                  at junit.framework.TestCase.runBare(TestCase.java:134)
                  at junit.framework.TestResult$1.protect(TestResult.java:110)
                  at junit.framework.TestResult.runProtected(TestResult.java:128)
                  at junit.framework.TestResult.run(TestResult.java:113)
                  at junit.framework.TestCase.run(TestCase.java:124)
                  at junit.framework.TestSuite.runTest(TestSuite.java:243)
                  at junit.framework.TestSuite.run(TestSuite.java:238)
                  at org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:83)
                  at org.junit.runners.Suite.runChild(Suite.java:128)
                  at org.junit.runners.Suite.runChild(Suite.java:24)
                  at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
                  at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
                  at java.util.concurrent.FutureTask.run(FutureTask.java:262)
                  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
                  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
                  at java.lang.Thread.run(Thread.java:745)
          
          Show
          rkanter Robert Kanter added a comment - I was just running the relevant unit tests, and it looks like TestMetricsInstrumentation has a bunch of failures now. Narayan Periwal , can you take a look and make a new patch? I promise I won't take as long to review it this time Looks like they're all failing with testInstrumentationTimer(org.apache.oozie.util.TestMetricsInstrumentation) Time elapsed: 0.012 sec <<< ERROR! java.lang.NullPointerException at org.apache.oozie.service.ConfigurationService.getBoolean(ConfigurationService.java:524) at org.apache.oozie.util.MetricsInstrumentation.<init>(MetricsInstrumentation.java:105) at org.apache.oozie.util.TestMetricsInstrumentation.testInstrumentationTimer(TestMetricsInstrumentation.java:62) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at junit.framework.TestCase.runTest(TestCase.java:168) at junit.framework.TestCase.runBare(TestCase.java:134) at junit.framework.TestResult$1.protect(TestResult.java:110) at junit.framework.TestResult.runProtected(TestResult.java:128) at junit.framework.TestResult.run(TestResult.java:113) at junit.framework.TestCase.run(TestCase.java:124) at junit.framework.TestSuite.runTest(TestSuite.java:243) at junit.framework.TestSuite.run(TestSuite.java:238) at org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:83) at org.junit.runners.Suite.runChild(Suite.java:128) at org.junit.runners.Suite.runChild(Suite.java:24) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745)
          Hide
          rkanter Robert Kanter added a comment - - edited

          Sorry for the super long delay. The latest patch (v10) looks good to me. +1

          Show
          rkanter Robert Kanter added a comment - - edited Sorry for the super long delay. The latest patch (v10) looks good to me. +1
          Hide
          hadoopqa Hadoop QA added a comment -

          Testing JIRA OOZIE-2251

          Cleaning local git workspace

          ----------------------------

          +1 PATCH_APPLIES
          +1 CLEAN
          -1 RAW_PATCH_ANALYSIS
          . +1 the patch does not introduce any @author tags
          . +1 the patch does not introduce any tabs
          . +1 the patch does not introduce any trailing spaces
          . +1 the patch does not introduce any line longer than 132
          . -1 the patch does not add/modify any testcase
          +1 RAT
          . +1 the patch does not seem to introduce new RAT warnings
          +1 JAVADOC
          . +1 the patch does not seem to introduce new Javadoc warnings
          +1 COMPILE
          . +1 HEAD compiles
          . +1 patch compiles
          . +1 the patch does not seem to introduce new javac warnings
          +1 BACKWARDS_COMPATIBILITY
          . +1 the patch does not change any JPA Entity/Colum/Basic/Lob/Transient annotations
          . +1 the patch does not modify JPA files
          -1 TESTS
          . Tests run: 1694
          . Tests failed: 3
          . Tests errors: 5

          . The patch failed the following testcases:

          . testCoordStatus_Failed(org.apache.oozie.command.coord.TestCoordChangeXCommand)
          . testMaxMatThrottleNotPicked(org.apache.oozie.service.TestCoordMaterializeTriggerService)
          . testForNoDuplicates(org.apache.oozie.event.TestEventGeneration)

          +1 DISTRO
          . +1 distro tarball builds with the patch

          ----------------------------
          -1 Overall result, please check the reported -1(s)

          The full output of the test-patch run is available at

          . https://builds.apache.org/job/oozie-trunk-precommit-build/2553/

          Show
          hadoopqa Hadoop QA added a comment - Testing JIRA OOZIE-2251 Cleaning local git workspace ---------------------------- +1 PATCH_APPLIES +1 CLEAN -1 RAW_PATCH_ANALYSIS . +1 the patch does not introduce any @author tags . +1 the patch does not introduce any tabs . +1 the patch does not introduce any trailing spaces . +1 the patch does not introduce any line longer than 132 . -1 the patch does not add/modify any testcase +1 RAT . +1 the patch does not seem to introduce new RAT warnings +1 JAVADOC . +1 the patch does not seem to introduce new Javadoc warnings +1 COMPILE . +1 HEAD compiles . +1 patch compiles . +1 the patch does not seem to introduce new javac warnings +1 BACKWARDS_COMPATIBILITY . +1 the patch does not change any JPA Entity/Colum/Basic/Lob/Transient annotations . +1 the patch does not modify JPA files -1 TESTS . Tests run: 1694 . Tests failed: 3 . Tests errors: 5 . The patch failed the following testcases: . testCoordStatus_Failed(org.apache.oozie.command.coord.TestCoordChangeXCommand) . testMaxMatThrottleNotPicked(org.apache.oozie.service.TestCoordMaterializeTriggerService) . testForNoDuplicates(org.apache.oozie.event.TestEventGeneration) +1 DISTRO . +1 distro tarball builds with the patch ---------------------------- -1 Overall result, please check the reported -1(s) The full output of the test-patch run is available at . https://builds.apache.org/job/oozie-trunk-precommit-build/2553/
          Hide
          nperiwal Narayan Periwal added a comment -

          Srikanth Sundarrajan, rebased the patch and also updated it in review board

          Show
          nperiwal Narayan Periwal added a comment - Srikanth Sundarrajan , rebased the patch and also updated it in review board
          Hide
          sriksun Srikanth Sundarrajan added a comment -

          Looks good to me, but this needs to be rebased.

          Show
          sriksun Srikanth Sundarrajan added a comment - Looks good to me, but this needs to be rebased.
          Hide
          nperiwal Narayan Periwal added a comment -

          Robert Kanter, Please have a look at the patch. I have made the changes as per the review.

          Show
          nperiwal Narayan Periwal added a comment - Robert Kanter , Please have a look at the patch. I have made the changes as per the review.
          Hide
          hadoopqa Hadoop QA added a comment -

          Testing JIRA OOZIE-2251

          Cleaning local git workspace

          ----------------------------

          +1 PATCH_APPLIES
          +1 CLEAN
          -1 RAW_PATCH_ANALYSIS
          . +1 the patch does not introduce any @author tags
          . +1 the patch does not introduce any tabs
          . +1 the patch does not introduce any trailing spaces
          . +1 the patch does not introduce any line longer than 132
          . -1 the patch does not add/modify any testcase
          +1 RAT
          . +1 the patch does not seem to introduce new RAT warnings
          +1 JAVADOC
          . +1 the patch does not seem to introduce new Javadoc warnings
          +1 COMPILE
          . +1 HEAD compiles
          . +1 patch compiles
          . +1 the patch does not seem to introduce new javac warnings
          +1 BACKWARDS_COMPATIBILITY
          . +1 the patch does not change any JPA Entity/Colum/Basic/Lob/Transient annotations
          . +1 the patch does not modify JPA files
          -1 TESTS
          . Tests run: 1673
          . Tests failed: 4
          . Tests errors: 5

          . The patch failed the following testcases:

          . testCoordStatusTransitWithLock(org.apache.oozie.service.TestStatusTransitService)
          . testNone(org.apache.oozie.command.coord.TestCoordActionInputCheckXCommandNonUTC)
          . testCoordMaterializeTriggerService3(org.apache.oozie.service.TestCoordMaterializeTriggerService)
          . testStreamingWithMultipleOozieServers_coordActionList(org.apache.oozie.service.TestZKXLogStreamingService)

          +1 DISTRO
          . +1 distro tarball builds with the patch

          ----------------------------
          -1 Overall result, please check the reported -1(s)

          The full output of the test-patch run is available at

          . https://builds.apache.org/job/oozie-trunk-precommit-build/2441/

          Show
          hadoopqa Hadoop QA added a comment - Testing JIRA OOZIE-2251 Cleaning local git workspace ---------------------------- +1 PATCH_APPLIES +1 CLEAN -1 RAW_PATCH_ANALYSIS . +1 the patch does not introduce any @author tags . +1 the patch does not introduce any tabs . +1 the patch does not introduce any trailing spaces . +1 the patch does not introduce any line longer than 132 . -1 the patch does not add/modify any testcase +1 RAT . +1 the patch does not seem to introduce new RAT warnings +1 JAVADOC . +1 the patch does not seem to introduce new Javadoc warnings +1 COMPILE . +1 HEAD compiles . +1 patch compiles . +1 the patch does not seem to introduce new javac warnings +1 BACKWARDS_COMPATIBILITY . +1 the patch does not change any JPA Entity/Colum/Basic/Lob/Transient annotations . +1 the patch does not modify JPA files -1 TESTS . Tests run: 1673 . Tests failed: 4 . Tests errors: 5 . The patch failed the following testcases: . testCoordStatusTransitWithLock(org.apache.oozie.service.TestStatusTransitService) . testNone(org.apache.oozie.command.coord.TestCoordActionInputCheckXCommandNonUTC) . testCoordMaterializeTriggerService3(org.apache.oozie.service.TestCoordMaterializeTriggerService) . testStreamingWithMultipleOozieServers_coordActionList(org.apache.oozie.service.TestZKXLogStreamingService) +1 DISTRO . +1 distro tarball builds with the patch ---------------------------- -1 Overall result, please check the reported -1(s) The full output of the test-patch run is available at . https://builds.apache.org/job/oozie-trunk-precommit-build/2441/
          Hide
          nperiwal Narayan Periwal added a comment -

          Robert Kanter, changes made as per the review.

          Show
          nperiwal Narayan Periwal added a comment - Robert Kanter , changes made as per the review.
          Hide
          sriksun Srikanth Sundarrajan added a comment -

          Narayan Periwal, If some user were to use Ganglia in their environment, then they will need additionally add the excluded dependency to the class path. It would be useful to cover this in the docs.

          Show
          sriksun Srikanth Sundarrajan added a comment - Narayan Periwal , If some user were to use Ganglia in their environment, then they will need additionally add the excluded dependency to the class path. It would be useful to cover this in the docs.
          Hide
          nperiwal Narayan Periwal added a comment -

          @robert kanter, the licence-incompatible dependencies that we are excluding is required only at compile time.
          I have made the changes as per the review. Please have a look.

          Show
          nperiwal Narayan Periwal added a comment - @robert kanter, the licence-incompatible dependencies that we are excluding is required only at compile time. I have made the changes as per the review. Please have a look.
          Hide
          rkanter Robert Kanter added a comment -

          Narayan Periwal I left you some comments on RB.

          Also, are any of the license-incompatible dependencies that we're excluding required at runtime? Or just at compile time? If we also need them at runtime, can you add some docs on what to add so users can do it?

          Show
          rkanter Robert Kanter added a comment - Narayan Periwal I left you some comments on RB. Also, are any of the license-incompatible dependencies that we're excluding required at runtime? Or just at compile time? If we also need them at runtime, can you add some docs on what to add so users can do it?
          Hide
          hadoopqa Hadoop QA added a comment -

          Testing JIRA OOZIE-2251

          Cleaning local git workspace

          ----------------------------

          +1 PATCH_APPLIES
          +1 CLEAN
          -1 RAW_PATCH_ANALYSIS
          . +1 the patch does not introduce any @author tags
          . +1 the patch does not introduce any tabs
          . +1 the patch does not introduce any trailing spaces
          . +1 the patch does not introduce any line longer than 132
          . -1 the patch does not add/modify any testcase
          +1 RAT
          . +1 the patch does not seem to introduce new RAT warnings
          +1 JAVADOC
          . +1 the patch does not seem to introduce new Javadoc warnings
          +1 COMPILE
          . +1 HEAD compiles
          . +1 patch compiles
          . +1 the patch does not seem to introduce new javac warnings
          +1 BACKWARDS_COMPATIBILITY
          . +1 the patch does not change any JPA Entity/Colum/Basic/Lob/Transient annotations
          . +1 the patch does not modify JPA files
          -1 TESTS
          . Tests run: 1653
          . Tests failed: 1
          . Tests errors: 5

          . The patch failed the following testcases:

          . testForNoDuplicates(org.apache.oozie.event.TestEventGeneration)

          +1 DISTRO
          . +1 distro tarball builds with the patch

          ----------------------------
          -1 Overall result, please check the reported -1(s)

          The full output of the test-patch run is available at

          . https://builds.apache.org/job/oozie-trunk-precommit-build/2414/

          Show
          hadoopqa Hadoop QA added a comment - Testing JIRA OOZIE-2251 Cleaning local git workspace ---------------------------- +1 PATCH_APPLIES +1 CLEAN -1 RAW_PATCH_ANALYSIS . +1 the patch does not introduce any @author tags . +1 the patch does not introduce any tabs . +1 the patch does not introduce any trailing spaces . +1 the patch does not introduce any line longer than 132 . -1 the patch does not add/modify any testcase +1 RAT . +1 the patch does not seem to introduce new RAT warnings +1 JAVADOC . +1 the patch does not seem to introduce new Javadoc warnings +1 COMPILE . +1 HEAD compiles . +1 patch compiles . +1 the patch does not seem to introduce new javac warnings +1 BACKWARDS_COMPATIBILITY . +1 the patch does not change any JPA Entity/Colum/Basic/Lob/Transient annotations . +1 the patch does not modify JPA files -1 TESTS . Tests run: 1653 . Tests failed: 1 . Tests errors: 5 . The patch failed the following testcases: . testForNoDuplicates(org.apache.oozie.event.TestEventGeneration) +1 DISTRO . +1 distro tarball builds with the patch ---------------------------- -1 Overall result, please check the reported -1(s) The full output of the test-patch run is available at . https://builds.apache.org/job/oozie-trunk-precommit-build/2414/
          Hide
          hadoopqa Hadoop QA added a comment -

          Testing JIRA OOZIE-2251

          Cleaning local git workspace

          ----------------------------

          -1 Patch failed to apply to head of branch

          ----------------------------

          Show
          hadoopqa Hadoop QA added a comment - Testing JIRA OOZIE-2251 Cleaning local git workspace ---------------------------- -1 Patch failed to apply to head of branch ----------------------------
          Hide
          nperiwal Narayan Periwal added a comment -

          Shwetha G S, have made the changes as per review. Please have a look

          Show
          nperiwal Narayan Periwal added a comment - Shwetha G S , have made the changes as per review. Please have a look
          Hide
          shwethags Shwetha G S added a comment -

          You need gmetric4j at compile time, but don't include it in the war

          Show
          shwethags Shwetha G S added a comment - You need gmetric4j at compile time, but don't include it in the war
          Hide
          nperiwal Narayan Periwal added a comment -

          Shwetha G S, you said don't include info.ganglia.gmetric4j:gmetric4j as well. But I need this to initialise the ganglia server.

          Show
          nperiwal Narayan Periwal added a comment - Shwetha G S , you said don't include info.ganglia.gmetric4j:gmetric4j as well. But I need this to initialise the ganglia server.
          Hide
          shwethags Shwetha G S added a comment -

          don't include info.ganglia.gmetric4j:gmetric4j as well

          Show
          shwethags Shwetha G S added a comment - don't include info.ganglia.gmetric4j:gmetric4j as well
          Hide
          shwethags Shwetha G S added a comment -

          Narayan Periwal, can you make sure org.acplt:oncrpc is not part of the war and add documentation that users need to include it if ganglia metrics is enabled

          Show
          shwethags Shwetha G S added a comment - Narayan Periwal , can you make sure org.acplt:oncrpc is not part of the war and add documentation that users need to include it if ganglia metrics is enabled
          Hide
          shwethags Shwetha G S added a comment -

          Some minor comments on reviewboard

          Show
          shwethags Shwetha G S added a comment - Some minor comments on reviewboard
          Hide
          sriksun Srikanth Sundarrajan added a comment -

          +1 for the patch.

          Additional note FWIW, Gmetric pulls in an additional dependency (LGPL) which isn't compatible. This might be an issue if binary releases were to include all transitive dependencies. Some details can be found here. For additional reference see SPARK-1167

          Show
          sriksun Srikanth Sundarrajan added a comment - +1 for the patch. Additional note FWIW, Gmetric pulls in an additional dependency (LGPL) which isn't compatible. This might be an issue if binary releases were to include all transitive dependencies. Some details can be found here . For additional reference see SPARK-1167
          Hide
          nperiwal Narayan Periwal added a comment -

          Srikanth Sundarrajan, changes made as per the review

          Show
          nperiwal Narayan Periwal added a comment - Srikanth Sundarrajan , changes made as per the review
          Hide
          nperiwal Narayan Periwal added a comment -

          Srikanth Sundarrajan, changes made as per the review, please have a look

          Show
          nperiwal Narayan Periwal added a comment - Srikanth Sundarrajan , changes made as per the review, please have a look
          Hide
          sriksun Srikanth Sundarrajan added a comment -

          Thanks Narayan Periwal, fix does address most of the comments from earlier review. Have a few more on the new patch, can you please check.

          Show
          sriksun Srikanth Sundarrajan added a comment - Thanks Narayan Periwal , fix does address most of the comments from earlier review. Have a few more on the new patch, can you please check.
          Hide
          hadoopqa Hadoop QA added a comment -

          Testing JIRA OOZIE-2251

          Cleaning local git workspace

          ----------------------------

          +1 PATCH_APPLIES
          +1 CLEAN
          -1 RAW_PATCH_ANALYSIS
          . +1 the patch does not introduce any @author tags
          . +1 the patch does not introduce any tabs
          . +1 the patch does not introduce any trailing spaces
          . +1 the patch does not introduce any line longer than 132
          . -1 the patch does not add/modify any testcase
          +1 RAT
          . +1 the patch does not seem to introduce new RAT warnings
          +1 JAVADOC
          . +1 the patch does not seem to introduce new Javadoc warnings
          -1 COMPILE
          . +1 HEAD compiles
          . +1 patch compiles
          . -1 the patch seems to introduce 1 new javac warning(s)
          +1 BACKWARDS_COMPATIBILITY
          . +1 the patch does not change any JPA Entity/Colum/Basic/Lob/Transient annotations
          . +1 the patch does not modify JPA files
          -1 TESTS
          . Tests run: 1653
          . Tests failed: 4
          . Tests errors: 0

          . The patch failed the following testcases:

          . testNone(org.apache.oozie.command.coord.TestCoordActionInputCheckXCommandNonUTC)
          . testPauseBundleAndCoordinator(org.apache.oozie.service.TestPauseTransitService)
          . testSamplers(org.apache.oozie.util.TestMetricsInstrumentation)
          . testCoordStatus_Failed(org.apache.oozie.command.coord.TestCoordChangeXCommand)

          +1 DISTRO
          . +1 distro tarball builds with the patch

          ----------------------------
          -1 Overall result, please check the reported -1(s)

          The full output of the test-patch run is available at

          . https://builds.apache.org/job/oozie-trunk-precommit-build/2392/

          Show
          hadoopqa Hadoop QA added a comment - Testing JIRA OOZIE-2251 Cleaning local git workspace ---------------------------- +1 PATCH_APPLIES +1 CLEAN -1 RAW_PATCH_ANALYSIS . +1 the patch does not introduce any @author tags . +1 the patch does not introduce any tabs . +1 the patch does not introduce any trailing spaces . +1 the patch does not introduce any line longer than 132 . -1 the patch does not add/modify any testcase +1 RAT . +1 the patch does not seem to introduce new RAT warnings +1 JAVADOC . +1 the patch does not seem to introduce new Javadoc warnings -1 COMPILE . +1 HEAD compiles . +1 patch compiles . -1 the patch seems to introduce 1 new javac warning(s) +1 BACKWARDS_COMPATIBILITY . +1 the patch does not change any JPA Entity/Colum/Basic/Lob/Transient annotations . +1 the patch does not modify JPA files -1 TESTS . Tests run: 1653 . Tests failed: 4 . Tests errors: 0 . The patch failed the following testcases: . testNone(org.apache.oozie.command.coord.TestCoordActionInputCheckXCommandNonUTC) . testPauseBundleAndCoordinator(org.apache.oozie.service.TestPauseTransitService) . testSamplers(org.apache.oozie.util.TestMetricsInstrumentation) . testCoordStatus_Failed(org.apache.oozie.command.coord.TestCoordChangeXCommand) +1 DISTRO . +1 distro tarball builds with the patch ---------------------------- -1 Overall result, please check the reported -1(s) The full output of the test-patch run is available at . https://builds.apache.org/job/oozie-trunk-precommit-build/2392/
          Hide
          nperiwal Narayan Periwal added a comment -

          Srikanth Sundarrajan, changes made as per the review

          Show
          nperiwal Narayan Periwal added a comment - Srikanth Sundarrajan , changes made as per the review
          Hide
          sriksun Srikanth Sundarrajan added a comment -

          Narayan Periwal, Have shared my comments on review board

          Show
          sriksun Srikanth Sundarrajan added a comment - Narayan Periwal , Have shared my comments on review board
          Hide
          nperiwal Narayan Periwal added a comment -

          Hi Robert Kanter and Srikanth Sundarrajan,
          Please have a look at the patch.

          Show
          nperiwal Narayan Periwal added a comment - Hi Robert Kanter and Srikanth Sundarrajan , Please have a look at the patch.
          Hide
          nperiwal Narayan Periwal added a comment -

          Hi Srikanth Sundarrajan,
          I have added the review board link.

          Show
          nperiwal Narayan Periwal added a comment - Hi Srikanth Sundarrajan , I have added the review board link.
          Hide
          sriksun Srikanth Sundarrajan added a comment -

          Possible to upload this in reviewboard ?

          Show
          sriksun Srikanth Sundarrajan added a comment - Possible to upload this in reviewboard ?
          Hide
          hadoopqa Hadoop QA added a comment -

          Testing JIRA OOZIE-2251

          Cleaning local git workspace

          ----------------------------

          +1 PATCH_APPLIES
          +1 CLEAN
          -1 RAW_PATCH_ANALYSIS
          . +1 the patch does not introduce any @author tags
          . +1 the patch does not introduce any tabs
          . +1 the patch does not introduce any trailing spaces
          . +1 the patch does not introduce any line longer than 132
          . -1 the patch does not add/modify any testcase
          +1 RAT
          . +1 the patch does not seem to introduce new RAT warnings
          +1 JAVADOC
          . +1 the patch does not seem to introduce new Javadoc warnings
          -1 COMPILE
          . +1 HEAD compiles
          . +1 patch compiles
          . -1 the patch seems to introduce 1 new javac warning(s)
          +1 BACKWARDS_COMPATIBILITY
          . +1 the patch does not change any JPA Entity/Colum/Basic/Lob/Transient annotations
          . +1 the patch does not modify JPA files
          -1 TESTS
          . Tests run: 1653
          . Tests failed: 3
          . Tests errors: 0

          . The patch failed the following testcases:

          . testNone(org.apache.oozie.command.coord.TestCoordActionInputCheckXCommand)
          . testSamplers(org.apache.oozie.util.TestMetricsInstrumentation)
          . testActionKillCommandActionNumbers(org.apache.oozie.command.coord.TestCoordActionsKillXCommand)

          +1 DISTRO
          . +1 distro tarball builds with the patch

          ----------------------------
          -1 Overall result, please check the reported -1(s)

          The full output of the test-patch run is available at

          . https://builds.apache.org/job/oozie-trunk-precommit-build/2383/

          Show
          hadoopqa Hadoop QA added a comment - Testing JIRA OOZIE-2251 Cleaning local git workspace ---------------------------- +1 PATCH_APPLIES +1 CLEAN -1 RAW_PATCH_ANALYSIS . +1 the patch does not introduce any @author tags . +1 the patch does not introduce any tabs . +1 the patch does not introduce any trailing spaces . +1 the patch does not introduce any line longer than 132 . -1 the patch does not add/modify any testcase +1 RAT . +1 the patch does not seem to introduce new RAT warnings +1 JAVADOC . +1 the patch does not seem to introduce new Javadoc warnings -1 COMPILE . +1 HEAD compiles . +1 patch compiles . -1 the patch seems to introduce 1 new javac warning(s) +1 BACKWARDS_COMPATIBILITY . +1 the patch does not change any JPA Entity/Colum/Basic/Lob/Transient annotations . +1 the patch does not modify JPA files -1 TESTS . Tests run: 1653 . Tests failed: 3 . Tests errors: 0 . The patch failed the following testcases: . testNone(org.apache.oozie.command.coord.TestCoordActionInputCheckXCommand) . testSamplers(org.apache.oozie.util.TestMetricsInstrumentation) . testActionKillCommandActionNumbers(org.apache.oozie.command.coord.TestCoordActionsKillXCommand) +1 DISTRO . +1 distro tarball builds with the patch ---------------------------- -1 Overall result, please check the reported -1(s) The full output of the test-patch run is available at . https://builds.apache.org/job/oozie-trunk-precommit-build/2383/
          Hide
          nperiwal Narayan Periwal added a comment -

          This patch fixes the error that were coming in the following testcases :
          TestMetricsInstrumentation, TestOozieCLI, TestInstrumentation

          Show
          nperiwal Narayan Periwal added a comment - This patch fixes the error that were coming in the following testcases : TestMetricsInstrumentation, TestOozieCLI, TestInstrumentation
          Hide
          hadoopqa Hadoop QA added a comment -

          Testing JIRA OOZIE-2251

          Cleaning local git workspace

          ----------------------------

          +1 PATCH_APPLIES
          +1 CLEAN
          -1 RAW_PATCH_ANALYSIS
          . +1 the patch does not introduce any @author tags
          . +1 the patch does not introduce any tabs
          . +1 the patch does not introduce any trailing spaces
          . +1 the patch does not introduce any line longer than 132
          . -1 the patch does not add/modify any testcase
          +1 RAT
          . +1 the patch does not seem to introduce new RAT warnings
          +1 JAVADOC
          . +1 the patch does not seem to introduce new Javadoc warnings
          -1 COMPILE
          . +1 HEAD compiles
          . +1 patch compiles
          . -1 the patch seems to introduce 1 new javac warning(s)
          +1 BACKWARDS_COMPATIBILITY
          . +1 the patch does not change any JPA Entity/Colum/Basic/Lob/Transient annotations
          . +1 the patch does not modify JPA files
          -1 TESTS
          . Tests run: 1653
          . Tests failed: 1
          . Tests errors: 10

          . The patch failed the following testcases:

          . testAdminInstrumentation(org.apache.oozie.client.TestOozieCLI)

          +1 DISTRO
          . +1 distro tarball builds with the patch

          ----------------------------
          -1 Overall result, please check the reported -1(s)

          The full output of the test-patch run is available at

          . https://builds.apache.org/job/oozie-trunk-precommit-build/2381/

          Show
          hadoopqa Hadoop QA added a comment - Testing JIRA OOZIE-2251 Cleaning local git workspace ---------------------------- +1 PATCH_APPLIES +1 CLEAN -1 RAW_PATCH_ANALYSIS . +1 the patch does not introduce any @author tags . +1 the patch does not introduce any tabs . +1 the patch does not introduce any trailing spaces . +1 the patch does not introduce any line longer than 132 . -1 the patch does not add/modify any testcase +1 RAT . +1 the patch does not seem to introduce new RAT warnings +1 JAVADOC . +1 the patch does not seem to introduce new Javadoc warnings -1 COMPILE . +1 HEAD compiles . +1 patch compiles . -1 the patch seems to introduce 1 new javac warning(s) +1 BACKWARDS_COMPATIBILITY . +1 the patch does not change any JPA Entity/Colum/Basic/Lob/Transient annotations . +1 the patch does not modify JPA files -1 TESTS . Tests run: 1653 . Tests failed: 1 . Tests errors: 10 . The patch failed the following testcases: . testAdminInstrumentation(org.apache.oozie.client.TestOozieCLI) +1 DISTRO . +1 distro tarball builds with the patch ---------------------------- -1 Overall result, please check the reported -1(s) The full output of the test-patch run is available at . https://builds.apache.org/job/oozie-trunk-precommit-build/2381/
          Hide
          nperiwal Narayan Periwal added a comment -

          Changed the configs used in oozie-default.xml to lowercase

          Show
          nperiwal Narayan Periwal added a comment - Changed the configs used in oozie-default.xml to lowercase
          Hide
          rkanter Robert Kanter added a comment -

          That patch appears to be related to yarn tags; I think you uploaded the wrong one?

          In any case, OOZIE-1817 added an implementation of the instrumentation that uses Codahale Metrics (which looks like it's the Dropwizard metrics): https://dropwizard.github.io/metrics/3.1.0/
          Their disabled by default because their output formatting is slightly different than the original instrumentation, but you can enable them by following the docs here: http://oozie.apache.org/docs/4.1.0/AG_Install.html#Using_Metrics_instead_of_Instrumentation

          We're currently only exposing a JSON reporter from Codahale, but if you want to add better integration with graphite or ganglia, it looks like it should be pretty easy to add some code and oozie-site configs that do this:
          https://dropwizard.github.io/metrics/3.1.0/manual/ganglia/#manual-ganglia
          https://dropwizard.github.io/metrics/3.1.0/manual/graphite/#manual-graphite

          Show
          rkanter Robert Kanter added a comment - That patch appears to be related to yarn tags; I think you uploaded the wrong one? In any case, OOZIE-1817 added an implementation of the instrumentation that uses Codahale Metrics (which looks like it's the Dropwizard metrics): https://dropwizard.github.io/metrics/3.1.0/ Their disabled by default because their output formatting is slightly different than the original instrumentation, but you can enable them by following the docs here: http://oozie.apache.org/docs/4.1.0/AG_Install.html#Using_Metrics_instead_of_Instrumentation We're currently only exposing a JSON reporter from Codahale, but if you want to add better integration with graphite or ganglia, it looks like it should be pretty easy to add some code and oozie-site configs that do this: https://dropwizard.github.io/metrics/3.1.0/manual/ganglia/#manual-ganglia https://dropwizard.github.io/metrics/3.1.0/manual/graphite/#manual-graphite
          Hide
          hadoopqa Hadoop QA added a comment -

          Testing JIRA OOZIE-2251

          Cleaning local git workspace

          ----------------------------

          -1 Patch failed to apply to head of branch

          ----------------------------

          Show
          hadoopqa Hadoop QA added a comment - Testing JIRA OOZIE-2251 Cleaning local git workspace ---------------------------- -1 Patch failed to apply to head of branch ----------------------------
          Hide
          jaydeepvishwakarma Jaydeep Vishwakarma added a comment -

          To enhance the functional monitoring for oozie, we expose some important metrics into the Graphite/ Ganglia, a daemon that listens for some time-series data and render graphs of this data on demand. For the purpose of integrating this in oozie, we can use the dropwizard library to emit the metrics to graphite/ Ganglia. This will be an optional feature, An option will be provided to enable this feature on oozie through a appropriate parameter during run-rime on oozie config.

          Show
          jaydeepvishwakarma Jaydeep Vishwakarma added a comment - To enhance the functional monitoring for oozie, we expose some important metrics into the Graphite/ Ganglia, a daemon that listens for some time-series data and render graphs of this data on demand. For the purpose of integrating this in oozie, we can use the dropwizard library to emit the metrics to graphite/ Ganglia. This will be an optional feature, An option will be provided to enable this feature on oozie through a appropriate parameter during run-rime on oozie config.

            People

            • Assignee:
              nperiwal Narayan Periwal
              Reporter:
              jaydeepvishwakarma Jaydeep Vishwakarma
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development