Derby
  1. Derby
  2. DERBY-1114

derbynet/testSecMec.java fails intermittently and test exits mysteriously when server is shutdown as part of the testrun.

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Duplicate
    • Affects Version/s: 10.2.1.6
    • Fix Version/s: None
    • Component/s: Network Server, Test
    • Labels:
      None
    • Environment:
      Java Version: 1.4.2_06
      Java Vendor: Sun Microsystems Inc.
      Java home: /usr/local/lib/j2sdk1.4.2_06/jre
      OS name: Linux
      OS architecture: i386
      OS version: 2.4.21-27.0.2.ELsmp

      Description

      The test derbynet/testSecMec.java fails intermittently in a strange way that the test actually exits before finishing the full test.

      This issue was noticed by Bryan when he ran derbyall with the derby-1080 fix. Some discussion and analysis that has been done is in Derby-1080. Please read comments in DERBY-1080. Some main links in
      http://issues.apache.org/jira/browse/DERBY-1080#action_12370260
      http://issues.apache.org/jira/browse/DERBY-1080#action_12370374

      In short: the test does some switching of System.out and System.err streams before calling networkserver.shutdown. This code was added to this test as part of fix for derby-273. for some reason, the networkserver.shutdown call makes the test to exit. One theory is that the network server.shutdown is causing the standard out streams to close and thus the test exits.

      – Need to investigate what is the cause for this intermittent failure and how/why networkserver.shutdown is closing the stream or making the test to exit prematurely.

      1. testSecMec_test_output.zip
        68 kB
        Kathey Marsden
      2. Stdout.java
        0.6 kB
        Knut Anders Hatlen
      3. Derby1114.diff.txt
        4 kB
        Sunitha Kambhampati

        Issue Links

          Activity

          Sunitha Kambhampati created issue -
          Hide
          Sunitha Kambhampati added a comment -

          Per the discussion on 1080, one reasonable course of action was to remove the switching of streams in testSecMec.java to see if it solves the intermittent failure. So, I am attaching this patch Derby1114.diff.txt to remove the logic for switching of streams in testSecMec.java.

          Bryan - thanks for running this test so many times for me. I appreciate it. When you can, could you try this patch out and see if the intermittent problem still reproduces. If it doesnt, then can this patch be committed ?

          I ran this test 6 times on my linux machine and I didnt hit the intermittent failure but so far I have not had any luck reproducing the failure. I have started a derbynetmats and derbynetclientmats run. I'll post results as they finish.

          svn stat:
          M java\testing\org\apache\derbyTesting\functionTests\tests\derbynet\testSecMec.java
          M java\testing\org\apache\derbyTesting\functionTests\tests\derbynet\dataSourcePermissions_net.java

          I will post the original testSecMec.java here, so it can be used to debug this issue.

          Thanks.

          Show
          Sunitha Kambhampati added a comment - Per the discussion on 1080, one reasonable course of action was to remove the switching of streams in testSecMec.java to see if it solves the intermittent failure. So, I am attaching this patch Derby1114.diff.txt to remove the logic for switching of streams in testSecMec.java. Bryan - thanks for running this test so many times for me. I appreciate it. When you can, could you try this patch out and see if the intermittent problem still reproduces. If it doesnt, then can this patch be committed ? I ran this test 6 times on my linux machine and I didnt hit the intermittent failure but so far I have not had any luck reproducing the failure. I have started a derbynetmats and derbynetclientmats run. I'll post results as they finish. svn stat: M java\testing\org\apache\derbyTesting\functionTests\tests\derbynet\testSecMec.java M java\testing\org\apache\derbyTesting\functionTests\tests\derbynet\dataSourcePermissions_net.java I will post the original testSecMec.java here, so it can be used to debug this issue. Thanks.
          Sunitha Kambhampati made changes -
          Field Original Value New Value
          Attachment Derby1114.diff.txt [ 12324184 ]
          Hide
          Bryan Pendleton added a comment -

          I can still reproduce the problem even with your changes to remove the switchable streams. So that's one piece of evidence that appears to rule out the switchable streams as the cause.

          Also, I tried another experiment. I modified the definition of SwitchablePrintStream in dataSourcePermissions_net.java so that it overrode the close() method, and I gave the overridden method an empty body (that is, I didn't call super.close()) on the theory that this would show whether somebody was calling the close method on the streams unexpectedly. But I was still able to reproduce the problem even with SwitchablePrintStream.close() doing nothing.

          So, the good news is that I can still reproduce the failure, and I seem to be ruling things out.

          The bad news is that I'm a little short of theories as to what's going wrong.

          I like Sunitha's theory that we're somehow closing the standard output stream prematurely; I'm just not sure how to write code that tests that theory.

          Show
          Bryan Pendleton added a comment - I can still reproduce the problem even with your changes to remove the switchable streams. So that's one piece of evidence that appears to rule out the switchable streams as the cause. Also, I tried another experiment. I modified the definition of SwitchablePrintStream in dataSourcePermissions_net.java so that it overrode the close() method, and I gave the overridden method an empty body (that is, I didn't call super.close()) on the theory that this would show whether somebody was calling the close method on the streams unexpectedly. But I was still able to reproduce the problem even with SwitchablePrintStream.close() doing nothing. So, the good news is that I can still reproduce the failure, and I seem to be ruling things out. The bad news is that I'm a little short of theories as to what's going wrong. I like Sunitha's theory that we're somehow closing the standard output stream prematurely; I'm just not sure how to write code that tests that theory.
          Hide
          Sunitha Kambhampati added a comment -

          Thanks Bryan for trying out the patch. If the problem lies in the networkserver.shutdown and the test change to remove the switching of streams doesnt help, then I guess there isnt much point checking in this test change. So lets just ignore the (Derby1114.diff.txt) patch.

          I thought it was interesting that the shutdown message for server doesnt get to the derby.log for the run where the test exits. In the network server shutdown method (NetworkServerControlImpl.shutdown), the message would be written to the cloudscapeLogWriter. ( goes to consoleWriter method). If you follow that path, it leads to the Monitor.getStream() and there is a comment there that if streams cannot be setup for some reason, it would default to System.err. From a quick look, I think if the stream is setup as System.err it doesnot close the stream.

          I can only think of putting old fashioned println statements in networkserver.shutdown to see what maybe happening.

          Thanks,
          Sunitha.

          Show
          Sunitha Kambhampati added a comment - Thanks Bryan for trying out the patch. If the problem lies in the networkserver.shutdown and the test change to remove the switching of streams doesnt help, then I guess there isnt much point checking in this test change. So lets just ignore the (Derby1114.diff.txt) patch. I thought it was interesting that the shutdown message for server doesnt get to the derby.log for the run where the test exits. In the network server shutdown method (NetworkServerControlImpl.shutdown), the message would be written to the cloudscapeLogWriter. ( goes to consoleWriter method). If you follow that path, it leads to the Monitor.getStream() and there is a comment there that if streams cannot be setup for some reason, it would default to System.err. From a quick look, I think if the stream is setup as System.err it doesnot close the stream. I can only think of putting old fashioned println statements in networkserver.shutdown to see what maybe happening. Thanks, Sunitha.
          Hide
          Bryan Pendleton added a comment -

          I wonder whether the problem might possibly be on the RunTest side, instead.

          That is, I've sort of got three possible theories, so far:

          1) The child process prematurely closes System.out and System.err. This causes the parent process to think that the test run is complete, and the remainder of the output is lost.
          2) The child process prematurely calls System.exit. The test ends prematurely and the remainder of the output is not created.
          3) The RunTest harness mistakenly decides that the child process has finished, when in fact it is still running. RunTest terminates the test prematurely, and does not receive the full output from the child.

          I'm not sure I have any evidence to favor one of these theories over another, yet.

          Show
          Bryan Pendleton added a comment - I wonder whether the problem might possibly be on the RunTest side, instead. That is, I've sort of got three possible theories, so far: 1) The child process prematurely closes System.out and System.err. This causes the parent process to think that the test run is complete, and the remainder of the output is lost. 2) The child process prematurely calls System.exit. The test ends prematurely and the remainder of the output is not created. 3) The RunTest harness mistakenly decides that the child process has finished, when in fact it is still running. RunTest terminates the test prematurely, and does not receive the full output from the child. I'm not sure I have any evidence to favor one of these theories over another, yet.
          Hide
          Bryan Pendleton added a comment -

          To reproduce this bug, I simply run this command over and over:

          java -Dframework=DerbyNet -Dverbose=true org.apache.derbyTesting.functionTests.harness.RunTest derbynet/testSecMec.java

          Show
          Bryan Pendleton added a comment - To reproduce this bug, I simply run this command over and over: java -Dframework=DerbyNet -Dverbose=true org.apache.derbyTesting.functionTests.harness.RunTest derbynet/testSecMec.java
          Hide
          Bryan Pendleton added a comment -

          I've commented out everything in the NetworkServer shutdown code except for the close of the serverSocket. I have to leave the close of the serverSocket, otherwise the test hangs because networkServer.shutdown() uses the socket close to detect when the server is shut down.

          With all this shutdown code commented out, the test still fails for me.

          I am really finding this behavior confusing

          Show
          Bryan Pendleton added a comment - I've commented out everything in the NetworkServer shutdown code except for the close of the serverSocket. I have to leave the close of the serverSocket, otherwise the test hangs because networkServer.shutdown() uses the socket close to detect when the server is shut down. With all this shutdown code commented out, the test still fails for me. I am really finding this behavior confusing
          Hide
          Bryan Pendleton added a comment -

          I've been fiddling around with the main loop in testSecMec.java. This loop iterates through various security mechanisms. For each security mechanism, the loop starts up the Network Server with that mechanism, runs various tests, then shuts it down.

          My findings so far are interesting:

          • Re-ordering all the security mechanisms in the loop didn't affect the bug, So it doesn't seem to have to do with the security mechanism that is used.
          • In fact, I altered testSecMec.java so that it just looped 6 times with no security mechanism at all, and the bug still occurred.
          • Lastly, and most interestingly, when the bug occurs, it always happens after the second time through the loop. Never in dozens of reproduced cases has it ever happened after the 3rd or 4th or 5th time through the loop, nor after the first time.

          It seems that if the bug is going to happen in my environment, it always happens after the second time through the loop.

          Show
          Bryan Pendleton added a comment - I've been fiddling around with the main loop in testSecMec.java. This loop iterates through various security mechanisms. For each security mechanism, the loop starts up the Network Server with that mechanism, runs various tests, then shuts it down. My findings so far are interesting: Re-ordering all the security mechanisms in the loop didn't affect the bug, So it doesn't seem to have to do with the security mechanism that is used. In fact, I altered testSecMec.java so that it just looped 6 times with no security mechanism at all, and the bug still occurred. Lastly, and most interestingly, when the bug occurs, it always happens after the second time through the loop. Never in dozens of reproduced cases has it ever happened after the 3rd or 4th or 5th time through the loop, nor after the first time. It seems that if the bug is going to happen in my environment, it always happens after the second time through the loop.
          Hide
          Bryan Pendleton added a comment -

          Sunitha suggested removing the RunTest harness from the environment to rule that theory in or out. So I ran the test using -Dverbose=true and -Dkeepfiles=true to set up the test conditions, then I extracted the sub-command that RunTest was running, and ran that sub-command over and over to see if I could reproduce the problem.

          And I can.

          So it would seem that it's definitely not the harness's fault. Thanks for the good suggestion, Sunitha!

          Show
          Bryan Pendleton added a comment - Sunitha suggested removing the RunTest harness from the environment to rule that theory in or out. So I ran the test using -Dverbose=true and -Dkeepfiles=true to set up the test conditions, then I extracted the sub-command that RunTest was running, and ran that sub-command over and over to see if I could reproduce the problem. And I can. So it would seem that it's definitely not the harness's fault. Thanks for the good suggestion, Sunitha!
          Hide
          Bryan Pendleton added a comment -

          I tried building both sane jars and insane jars, and I can reproduce the problem either way. So I don't think it's related to SanityManager code.

          Show
          Bryan Pendleton added a comment - I tried building both sane jars and insane jars, and I can reproduce the problem either way. So I don't think it's related to SanityManager code.
          Hide
          Bryan Pendleton added a comment -

          I tried finding all the places where we might have called System.exit and replacing them with "throw new RuntimeException". Several dozen of these places were in other functional tests and in the harness, and since I'd already ruled those out, I didn't change them. I just changed the System.exit calls in the Derby code proper (java/engine/* and java/drda/*).

          And I can still reproduce the problem.

          So I think that I can rule out the "premature System.exit" theory, at least if it's a System.exit call coming from Derby code.

          That seems to leave the "somebody's closing System.out" theory as the most promising.

          Now I just have to figure out a way to pursue that theory. I don't know how to set a breakpoint on this condition, nor do I know how to search for it in the code using find and grep. Hmmm....

          Show
          Bryan Pendleton added a comment - I tried finding all the places where we might have called System.exit and replacing them with "throw new RuntimeException". Several dozen of these places were in other functional tests and in the harness, and since I'd already ruled those out, I didn't change them. I just changed the System.exit calls in the Derby code proper (java/engine/* and java/drda/*). And I can still reproduce the problem. So I think that I can rule out the "premature System.exit" theory, at least if it's a System.exit call coming from Derby code. That seems to leave the "somebody's closing System.out" theory as the most promising. Now I just have to figure out a way to pursue that theory. I don't know how to set a breakpoint on this condition, nor do I know how to search for it in the code using find and grep. Hmmm....
          Hide
          Knut Anders Hatlen added a comment -

          Bryan wrote:

          > That seems to leave the "somebody's closing System.out" theory as
          > the most promising.
          >
          > Now I just have to figure out a way to pursue that theory. I don't
          > know how to set a breakpoint on this condition, nor do I know how to
          > search for it in the code using find and grep. Hmmm....

          Maybe you could create a stream wrapper class which prints a stack
          trace in its close method, and set System.out/System.err to an
          instance of that class. See the attached file for an example.

          Show
          Knut Anders Hatlen added a comment - Bryan wrote: > That seems to leave the "somebody's closing System.out" theory as > the most promising. > > Now I just have to figure out a way to pursue that theory. I don't > know how to set a breakpoint on this condition, nor do I know how to > search for it in the code using find and grep. Hmmm.... Maybe you could create a stream wrapper class which prints a stack trace in its close method, and set System.out/System.err to an instance of that class. See the attached file for an example.
          Knut Anders Hatlen made changes -
          Attachment Stdout.java [ 12324384 ]
          Hide
          Bryan Pendleton added a comment -

          The next two things I'm intending to try are:

          • Knut Anders's suggestion of wrapping System.out to detect a close(), and
          • trying some different JDK vendors and versions to see if it's particular to my JDK.
          Show
          Bryan Pendleton added a comment - The next two things I'm intending to try are: Knut Anders's suggestion of wrapping System.out to detect a close(), and trying some different JDK vendors and versions to see if it's particular to my JDK.
          Bryan Pendleton made changes -
          Assignee Bryan Pendleton [ bryanpendleton ]
          Hide
          Bryan Pendleton added a comment -

          Sunitha, I have satisfied myself that I was encountering a JVM bug here, not a Derby bug.

          Specifically, I believe I was encountering this bug:
          http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=4927336

          Although that bug report seems to indicate that this bug was present only in JDK 1.5 betas, I believe that it was also present in some versions of JDK 1.4.2. This belief is based primarily on some of the comments in the bug report page, and also in this web page:
          http://forum.java.sun.com/thread.jspa?threadID=563298&messageID=3575166

          I am able to reproduce the problem with JDK 1.4.2_06, which is mentioned in the above web pages as being one of the releases afflicted with this problem.

          I tried running with both 1.5.0_05, and with 1.4.2_11. In both cases, I can not reproduce the bug.

          Since switching to a different JVM definitely resolves the behavior for me, and since the bug report seems to match my symptoms closely (I have even seen the "Real-time signal 28" message a few times), I believe that there's enough evidence here to conclusively resolve this bug as being a JVM bug, not a Derby problem.

          If you're comfortable with that resolution, could you please mark this bug closed?

          thanks,

          bryan

          Show
          Bryan Pendleton added a comment - Sunitha, I have satisfied myself that I was encountering a JVM bug here, not a Derby bug. Specifically, I believe I was encountering this bug: http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=4927336 Although that bug report seems to indicate that this bug was present only in JDK 1.5 betas, I believe that it was also present in some versions of JDK 1.4.2. This belief is based primarily on some of the comments in the bug report page, and also in this web page: http://forum.java.sun.com/thread.jspa?threadID=563298&messageID=3575166 I am able to reproduce the problem with JDK 1.4.2_06, which is mentioned in the above web pages as being one of the releases afflicted with this problem. I tried running with both 1.5.0_05, and with 1.4.2_11. In both cases, I can not reproduce the bug. Since switching to a different JVM definitely resolves the behavior for me, and since the bug report seems to match my symptoms closely (I have even seen the "Real-time signal 28" message a few times), I believe that there's enough evidence here to conclusively resolve this bug as being a JVM bug, not a Derby problem. If you're comfortable with that resolution, could you please mark this bug closed? thanks, bryan
          Bryan Pendleton made changes -
          Resolution Won't Fix [ 2 ]
          Status Open [ 1 ] Resolved [ 5 ]
          Hide
          Sunitha Kambhampati added a comment -

          Good work, Bryan. Thanks much for resolving this issue. I agree with your conclusion and am closing this issue.

          I am curious to know which program did you use to find out that the jvm was exiting with signal 28.

          Thanks,
          Sunitha.

          Show
          Sunitha Kambhampati added a comment - Good work, Bryan. Thanks much for resolving this issue. I agree with your conclusion and am closing this issue. I am curious to know which program did you use to find out that the jvm was exiting with signal 28. Thanks, Sunitha.
          Sunitha Kambhampati made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Hide
          Kathey Marsden added a comment -

          With all jvms I have tried and for quite a while I have seen this failure on my machine with Windows XP. I think it actually started when I upgraded my machine to Windows XP from Windows 2000, but I am not totally sure of that. It occurs with both classes and jars, with both derby client and jcc and is not intermittent. The test fails every time.

          I am attaching the test output (testSecMec_test_output.zip) to confirm that it is the same thing, but it sure seems like it because as Sunitha indicated the derby.log only shows two shutdown messages, instead of four.

          The JVM for this run is:

          java -version
          ava version "1.5.0"
          ava(TM) 2 Runtime Environment, Standard Edition (build pwi32dev-20060511 (SR2))
          BM J9 VM (build 2.3, J2RE 1.5.0 IBM J9 2.3 Windows XP x86-32 j9vmwi3223-20060504 (JIT enabled)
          9VM - 20060501_06428_lHdSMR
          IT - 20060428_1800_r8
          C - 20060501_AA)
          CL - 20060511a

          OS is Windows XP Professional version 2002 Service Pack 2

          All Sun and IBM jdk's that I have tried on this machine have the same failure, I am retiring this machine so thought I would record these results in case someone else hit the issue. Sadly, I don't have a solution to offer. One thing I tried, because I know it has caused other issues in the past was switching the test to just use port 1527 instead of 20000. That made no difference.

          When I set up my new machine I will see if I have the same problem.

          Show
          Kathey Marsden added a comment - With all jvms I have tried and for quite a while I have seen this failure on my machine with Windows XP. I think it actually started when I upgraded my machine to Windows XP from Windows 2000, but I am not totally sure of that. It occurs with both classes and jars, with both derby client and jcc and is not intermittent. The test fails every time. I am attaching the test output (testSecMec_test_output.zip) to confirm that it is the same thing, but it sure seems like it because as Sunitha indicated the derby.log only shows two shutdown messages, instead of four. The JVM for this run is: java -version ava version "1.5.0" ava(TM) 2 Runtime Environment, Standard Edition (build pwi32dev-20060511 (SR2)) BM J9 VM (build 2.3, J2RE 1.5.0 IBM J9 2.3 Windows XP x86-32 j9vmwi3223-20060504 (JIT enabled) 9VM - 20060501_06428_lHdSMR IT - 20060428_1800_r8 C - 20060501_AA) CL - 20060511a OS is Windows XP Professional version 2002 Service Pack 2 All Sun and IBM jdk's that I have tried on this machine have the same failure, I am retiring this machine so thought I would record these results in case someone else hit the issue. Sadly, I don't have a solution to offer. One thing I tried, because I know it has caused other issues in the past was switching the test to just use port 1527 instead of 20000. That made no difference. When I set up my new machine I will see if I have the same problem.
          Hide
          Kathey Marsden added a comment -

          have to reopen this to add an attachment.

          Show
          Kathey Marsden added a comment - have to reopen this to add an attachment.
          Kathey Marsden made changes -
          Status Closed [ 6 ] Reopened [ 4 ]
          Resolution Won't Fix [ 2 ]
          Hide
          Kathey Marsden added a comment -

          This is the test output from running the testSecMec.java test with derby client. The test seems to exit after two itterations of bringing the server up and down.

          This was run on the trunk At revision 431576

          Show
          Kathey Marsden added a comment - This is the test output from running the testSecMec.java test with derby client. The test seems to exit after two itterations of bringing the server up and down. This was run on the trunk At revision 431576
          Kathey Marsden made changes -
          Attachment testSecMec_test_output.zip [ 12338877 ]
          Hide
          Daniel John Debrunner added a comment -

          This test failed for me in a derbynetclientmats run, the last line in the output was

          Testing with derby.drda.securityMechanism=USER_ONLY_SECURITY

          before that it matched the master file, so about 300 lines of missng output.

          Windows XP IBM 1.4.2

          Show
          Daniel John Debrunner added a comment - This test failed for me in a derbynetclientmats run, the last line in the output was Testing with derby.drda.securityMechanism=USER_ONLY_SECURITY before that it matched the master file, so about 300 lines of missng output. Windows XP IBM 1.4.2
          Hide
          Daniel John Debrunner added a comment -

          The failure I'm seeing is due to this code: line 241

          // Wait for the NetworkServer to start.
          if (!isServerStarted(networkServer, 60))
          System.exit(-1);

          The test should not call System.exit(). Instead it should print a failure message and then break to end the test.

          More like.
          // Wait for the NetworkServer to start.
          if (!isServerStarted(networkServer, 60))

          { System.out.println("FAIL: Server failed to respond to ping - ending test"); break; }

          The test continued to fail for me, with the above error, but if I increased the loop count for isServerStarted to 120 from 60 it seems to pass every time.

          Show
          Daniel John Debrunner added a comment - The failure I'm seeing is due to this code: line 241 // Wait for the NetworkServer to start. if (!isServerStarted(networkServer, 60)) System.exit(-1); The test should not call System.exit(). Instead it should print a failure message and then break to end the test. More like. // Wait for the NetworkServer to start. if (!isServerStarted(networkServer, 60)) { System.out.println("FAIL: Server failed to respond to ping - ending test"); break; } The test continued to fail for me, with the above error, but if I increased the loop count for isServerStarted to 120 from 60 it seems to pass every time.
          Hide
          Kathey Marsden added a comment -

          Thanks Dan. Upping the time did the trick for me as well.

          Show
          Kathey Marsden added a comment - Thanks Dan. Upping the time did the trick for me as well.
          Hide
          Bryan Pendleton added a comment -

          I'm not quite sure what's going on with this JIRA issue.

          The original problem I was seeing was definitely a JVM bug, which went away when
          I upgraded the patch level on my particular JVM.

          Others have seen the same symptom, but for them it appears to be a timing issue,
          and changing the timing in the test has helped them avoid the problem.

          For now, I will unassign myself, as I am not working on the problem and do not see
          it in any of my Derby environments.

          Show
          Bryan Pendleton added a comment - I'm not quite sure what's going on with this JIRA issue. The original problem I was seeing was definitely a JVM bug, which went away when I upgraded the patch level on my particular JVM. Others have seen the same symptom, but for them it appears to be a timing issue, and changing the timing in the test has helped them avoid the problem. For now, I will unassign myself, as I am not working on the problem and do not see it in any of my Derby environments.
          Bryan Pendleton made changes -
          Assignee Bryan Pendleton [ bryanpendleton ]
          Bryan Pendleton made changes -
          Link This issue relates to DERBY-1751 [ DERBY-1751 ]
          Hide
          Myrna van Lunteren added a comment -

          I'm marking this as duplicate of DERBY-1751.
          In 10.3 (pre-branching), this test has been rewritten as a junit test and I have not seen this fail for a long time.

          Show
          Myrna van Lunteren added a comment - I'm marking this as duplicate of DERBY-1751 . In 10.3 (pre-branching), this test has been rewritten as a junit test and I have not seen this fail for a long time.
          Myrna van Lunteren made changes -
          Resolution Duplicate [ 3 ]
          Status Reopened [ 4 ] Closed [ 6 ]
          Gavin made changes -
          Workflow jira [ 12352017 ] Default workflow, editable Closed status [ 12798103 ]

            People

            • Assignee:
              Unassigned
              Reporter:
              Sunitha Kambhampati
            • Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development