Uploaded image for project: 'Maven Resolver'
  1. Maven Resolver
  2. MRESOLVER-228

Improve the maven dependency resolution speed by a skip & reconcile approach

Details

    • Improvement
    • Status: Closed
    • Major
    • Resolution: Duplicate
    • 1.7.2
    • None
    • Resolver
    • None

    Description

      When comes to resolve the huge amount of dependencies of an enterprise level project, the maven resolver is very slow to resolve the dependency graph/tree. Take one of our app as example, it could take 10minutes+ and 16G memory to print out the result of mvn dependency:tree.

      This is because there are many dependencies declared in the project, and some of the dependencies would introduce 600+ transitive dependencies, and exclusions are widely used to solve dependency conflicts. 

      By checking the code, we know the exclusion is also part of the cache key. This means when the exclusions up the tree differs, the cached resolution result for the same GAV won't be picked up and need s to be recalculated. 

      From above figure, we know:

      • In 1st case, D will be resolved only once as there are no exclusions/same exclusions up the tree.
      • In 2nd case, the B and C have different exclusions and D needs to be recalculated, if D is a heavy dependency which introduce many transitive dependencies, all D and its children needs to be recalculated.  Recalculating all of these nodes introduces 2 issues:
        • Slow in resolving dependencies.
        • Lots of DependencyNodes cached (all calculated/recalculated nodes would be cached) and will consume huge memory.

      To improve the speed of maven resolver's dependency resolution,  I implemented a skip & reconcile approach. Here is the skip part.

      From above figure, the 1st R is resolved at depth 3, and the 2nd R is resolved again because the depth is at 2 which is lower, the 3rd R at depth 3 and the 4th R at depth 4 are simply skipped as R is already resolved at depth 2. This is because the same node with deeper depth is most likely won't be picked up by maven as maven employs a "nearest transitive dependency in the tree depth and the first in resolution" strategy.

      The 3rd R and 4th R will have children set as zero and marked as skipped by the R at depth 2 in 2nd tree path.

       

      Here is the reconcile part:

      When there are dependency conflicts, some of the skipped nodes need to be reconciled.

      In above figure, there are 4 tree paths.

      • The D1 (D with version 1) in the 1st tree path is get resolved, children of E and R at depth 3 are resolved and cached.
      • In the 2nd tree path, when resolving E & R of H, we simply skip these 2 nodes as they are in deeper depth (depth: 4) than the E & R in 1st tree path.
      • In the 3rd tree path, a R node with lower path is resolved, and a E node at depth 5 is skipped.
      • In the 4th path, a D2 (D with version 2) node is resolved, as the depth is lower than D1, so maven will pick D2, this means the E & R's children cached in tree depth 1 should be discarded

      Thus we might need to reconcile the E & R nodes in 2nd, 3rd and 4th tree paths. Here only E in 2nd tree path needs to be reconciled. This is because:

      • R in 3rd tree path won't be picked up as there is already a R in 2nd tree path with a lower depth.
      • E in 3rd tree path won't be picked up as it is enough to reconcile the E in 2nd tree path as the E in 2nd tree path is deeper than E in 3rd tree path.

      Here is what we've updated in the maven-resolver logic:

      1. Resolve dependencies by leveraging a skip approach. The node in deeper depth will be skipped if a node with same GAV has been resolved with a lower depth.
      2. Cloned the nodes in step 1.
        Use maven's ConflictResolver (Transformer) to transform the cloned nodes and find out the conflict winners & the nodes to reconcile.
        Ex, D1 conflicts with D2 in above digram.
        E in 2nd tree path will be reconciled.
      3. Reconcile all skipped nodes identified in above step.

       

      After we enabled the resolver patch in maven, we are seeing 10% ~70% build time reduced for different projects depend on how complex the dependencies are, and the result of mvn dependency:tree and mvn dependency:list remain the same.

      We've verified the resolver performance patch leveraging an automation solution to certify 2000+ apps of our company by comparing the  mvn dependency:tree and mvn dependency:list result with/without the performance patch.

      Please help review the PR.
      https://github.com/apache/maven-resolver/pull/136

       

       

      Attachments

        Issue Links

          Activity

            michael-o Michael Osipov added a comment -

            Please take a look at MNG-6357, this might break your understanding of the second diagram.

            michael-o Michael Osipov added a comment - Please take a look at MNG-6357 , this might break your understanding of the second diagram.
            michael-o Michael Osipov added a comment -

            Question: You have added a boolean property, does this mean that this solution is an opt-out? If yes, why? Why would one want to disable this if the tree does not change and it is much faster?

            Definition: Root node is level 1 (highest) then it goes down to leafs which are level n (lowest)

            I'd like to rephrase your approach for my understanding: If you visit a node at a lower level and if it has been previously resolved at least on the same or higher level and you know the tree is going to be identical because version and exclusions are the same you will skip it?! I do not fully what understand you mean here by reconcile. Especially as explained here "Find out all skipped nodes that is getting affected with D1 as D1 is the loser and D2 is the winner. Reconcile all skipped nodes in above step, for nodes with same GAVs, only the node with the lowest path will be reconciled." Please clarify.

            michael-o Michael Osipov added a comment - Question: You have added a boolean property, does this mean that this solution is an opt-out? If yes, why? Why would one want to disable this if the tree does not change and it is much faster? Definition: Root node is level 1 (highest) then it goes down to leafs which are level n (lowest) I'd like to rephrase your approach for my understanding: If you visit a node at a lower level and if it has been previously resolved at least on the same or higher level and you know the tree is going to be identical because version and exclusions are the same you will skip it?! I do not fully what understand you mean here by reconcile. Especially as explained here "Find out all skipped nodes that is getting affected with D1 as D1 is the loser and D2 is the winner. Reconcile all skipped nodes in above step, for nodes with same GAVs, only the node with the lowest path will be reconciled." Please clarify.
            wecai wei cai added a comment - - edited

            michael-o 

            Regarding MNG-6357, I think this is a different issue.  MNG-6357 is about the dependency order when generating classpath. This Jira is about skipping resolving a node with higher depth if a node with same GAV has been resolved at a lower depth. MNG-6357  could be probably be resolved by leveraging a BFS solution.

             

            Question: You have added a boolean property, does this mean that this solution is an opt-out? If yes, why? Why would one want to disable this if the tree does not change and it is much faster?

            Just for compatibility consideration. As I'm confident with the "skip & reconcile" approach as we've dryrun 2000+ applications in our company, so I would raise both hands in favour of that we don't provide a property to disable this behavior.

             

            Question:  If you visit a node at a lower level and if it has been previously resolved at least on the same or higher level and you know the tree is going to be identical because version and exclusions are the same you will skip it?

            If version differs, it won't be skipped. Different versions are considered as version conflicts and will be always resolved.

            If version are the same and exclusions differs, still it will be skipped as the node with higher depth is most likely won't be picked. 

            Most likely a node at a higher level won't be picked up by maven as maven employs a "nearest transitive dependency in the tree depth and the first in resolution" strategy.
            I mean most likely here, however this is not always true in one case: version conflicts in parent nodes and one of the parent node is the conflict loser, this is why I need "reconcile/fix" later. 

            The strategy is like we should skip as much as possible, and then reconcile the least nodes that should be reconciled.

            Below is the message printed from one of our app.

            Skipped resolving 31459 nodes, and reconciled 8 nodes to solve 71 dependency conflicts. 

            Skip:

            A > B > C
               > D(excl E) -> B -> C

            The red B would be skipped as above blue B is with lower depth, even the exclusion is different, we skip resolving B as B is most likely won't be picked up by maven.  As a skip, we simply set B's children with empty, then record B is skipped by the B with path (A>B). Originally if exclusions up the tree (exclusions can be inherited) are different, maven would resolve B again, this means both B and B will be resolved by maven. And now we only resolve the blue B.

            The skip of B is safe as maven won't pick up B at all, this explains why the resolution could be much faster in this way because we skipped calculating many nodes of such cases. 

            Reconcile:

            A -> B -> D:2.0 -> E ->F
               -> C -> G -> H -> E -> F ===> this E would be skipped as above E is at lower depth.
               -> D:1.0 -> G    ===> D1.0 is with lower depth, D:2.0 is the conflict loser, this means E in the 1st tree path is no longer invalid as D2.0 is not picked up by maven, however E in 2nd tree path is skipped by the E in the 1st tree path. Thus we need to reconcile/fix the E in 2nd tree path as this E would be the winner.

             

            wecai wei cai added a comment - - edited michael-o   Regarding MNG-6357 , I think this is a different issue.   MNG-6357 is about the dependency order when generating classpath. This Jira is about skipping resolving a node with higher depth if a node with same GAV has been resolved at a lower depth. MNG-6357  could be probably be resolved by leveraging a BFS solution.   Question: You have added a boolean property, does this mean that this solution is an opt-out? If yes, why? Why would one want to disable this if the tree does not change and it is much faster? Just for compatibility consideration. As I'm confident with the "skip & reconcile" approach as we've dryrun 2000+ applications in our company, so I would raise both hands in favour of that we don't provide a property to disable this behavior.   Question:  If you visit a node at a lower level and if it has been previously resolved at least on the same or higher level and you know the tree is going to be identical because version and exclusions are the same you will skip it? If version differs, it won't be skipped. Different versions are considered as version conflicts and will be always resolved. If version are the same and exclusions differs, still it will be skipped as the node with higher depth is most likely won't be picked.  Most likely a node at a higher level won't be picked up by maven as maven employs a "nearest transitive dependency in the tree depth and the first in resolution" strategy. I mean most likely here, however this is not always true in one case: version conflicts in parent nodes and one of the parent node is the conflict loser, this is why I need "reconcile/fix" later.  The strategy is like we should skip as much as possible, and then reconcile the least nodes that should be reconciled. Below is the message printed from one of our app. Skipped resolving 31459 nodes, and reconciled 8 nodes to solve 71 dependency conflicts. Skip: A > B > C    > D(excl E) -> B -> C The red B  would be skipped as above blue B is with lower depth, even the exclusion is different, we skip resolving B as B is most likely won't be picked up by maven.  As a skip, we simply set B 's children with empty, then record B  is skipped by the B with path (A> B ). Originally if exclusions up the tree (exclusions can be inherited) are different, maven would resolve B again, this means both B and B will be resolved by maven. And now we only resolve the blue B . The skip of B  is safe as maven won't pick up B at all, this explains why the resolution could be much faster in this way because we skipped calculating many nodes of such cases.  Reconcile: A -> B -> D:2.0 -> E ->F    -> C -> G -> H -> E -> F ===> this E would be skipped as above E is at lower depth.    -> D:1.0 -> G    ===> D1.0 is with lower depth, D:2.0 is the conflict loser, this means E in the 1st tree path is no longer invalid as D2.0 is not picked up by maven, however E in 2nd tree path is skipped by the E in the 1st tree path. Thus we need to reconcile/fix the E in 2nd tree path  as this E would be the winner.  
            wecai wei cai added a comment - - edited

            michael-o 

            Details of the simplified algorithm:

            https://github.com/apache/maven-resolver/pull/136#issuecomment-998529747

            Originally was trying to filter out the least nodes to be reconciled from thousands of skipped nodes. With the updated algorithm, it can quickly locate the nodes to be reconciled by walking through the cloned graph. 

            The nodes need to be reconciled should satisfy:

            • A selected node:
              Node is a selected node (no winner set in DependencyNode.getData) which means this node is being selected by maven.
            • No children:
              Node has no children & original node also has no children (Skipped by other nodes with lower depth previously).  

            The overall idea is:

            Skip -> Reconcile (Transform rehearsal) -> Transform graph

            Skip:

            Skip resolving if node has deeper depth than cached when resolving all nodes from the root following up the original depth-first strategy.

            Reconcile (Transform rehearsal):
            1. Cloned the nodes  -> Transformer to transform the cloned root node -> Get the conflicts & the nodes to reconcile.

            2. Do some house sweeping for the NodesWithDepth cache, and reconcile the nodes in above step. 

            99%+ of apps don't need to a reconcile as the nodes found in above step is empty. 

             

            wecai wei cai added a comment - - edited michael-o   Details of the simplified algorithm: https://github.com/apache/maven-resolver/pull/136#issuecomment-998529747 Originally was trying to filter out the least nodes to be reconciled from thousands of skipped nodes. With the updated algorithm, it can quickly locate the nodes to be reconciled by walking through the cloned graph.  The nodes need to be reconciled should satisfy: A selected node: Node is a selected node (no winner set in DependencyNode.getData) which means this node is being selected by maven. No children: Node has no children & original node also has no children (Skipped by other nodes with lower depth previously).   The overall idea is: Skip -> Reconcile (Transform rehearsal) -> Transform graph Skip: Skip resolving if node has deeper depth than cached when resolving all nodes from the root following up the original depth-first strategy. Reconcile (Transform rehearsal): 1. Cloned the nodes  -> Transformer to transform the cloned root node -> Get the conflicts & the nodes to reconcile. 2. Do some house sweeping for the NodesWithDepth cache, and reconcile the nodes in above step.  99%+ of apps don't need to a reconcile as the nodes found in above step is empty.   
            ibabiankou Ivan Babiankou added a comment -

            Wow wecai this is amazing explanation of the problem and proposed solution! 

            I've stumbled upon this ticket after michael-o mentioned MRESOLVER-133 in my PR. I also realised that if BFS is used, the algorithm might be even simpler and more efficient:
            If you walk the dependency graph level by level, after each level you uniquely identify version for each dependency, which is also the nearest version to the root. This way you should never even try to resolve different versions and don't need the reconcile part.

            Does it make sense?

            Combined together with MRESOLVER-7 and MRESOLVER-133 this should give incredible performance boost. Especially for the projects using version ranges.

            ibabiankou Ivan Babiankou added a comment - Wow wecai this is amazing explanation of the problem and proposed solution!  I've stumbled upon this ticket after michael-o mentioned MRESOLVER-133 in my PR . I also realised that if BFS is used, the algorithm might be even simpler and more efficient: If you walk the dependency graph level by level, after each level you uniquely identify version for each dependency, which is also the nearest version to the root. This way you should never even try to resolve different versions and don't need the reconcile part. Does it make sense? Combined together with MRESOLVER-7 and MRESOLVER-133 this should give incredible performance boost. Especially for the projects using version ranges.
            wecai wei cai added a comment -

            ibabiankou 

            Thanks for your comments!  Kudos for the excellent work on MRESOLVER-133  and MRESOLVER-7.

            I agree combining MRESOLVER-133, MRESOLVER-7 and MRESOLVER-228, there would be huge performance improvements, however I don't think BFS + Skip can simply work as it may introduce scope incompatibilities.

            Before I submitted the PR of skip & reconcile approach in this ticket, I actually noticed the MRESOLVER-133 and tried breadth-first approach but no vain, here is what I've tried:

            • DFS + Skip
              1% of 500+ apps failed because some low level transitive dependencies are no longer available in the dependency tree. This is because for a skipped node, I need set children as empty and reuse the result of the node with same GAV but a lower depth.
              When there are version conflicts in the parent paths, a skipped node might should not be skipped and need reconcile somehow.
            • BFS + Skip
              This time, I'm not seeing any failures because of missing some low level transitive dependencies. Instead I witnessed the scope of certain dependency nodes in the dependency tree were incorrect, especially for "provided" or "compile" scopes. Check this code , when there are conflict items with different scopes, maven will pick "compile" scope as long as there is a conflict item using "compile" scope. 
            • DFS + Skip + Reconcile
              Then I come up with this solution. Tested with 2000+ apps in our company, all dependency:tree and dependency:list result remains the same which proves it is enough compatible. If go with BFS + Skip approach, as the scopes of dependencies are no longer the same, we should do some reconcile, however it is hard to implement such reconcile logic to make scope correct with BFS solution.
            wecai wei cai added a comment - ibabiankou   Thanks for your comments!  Kudos for the excellent work on MRESOLVER-133   and MRESOLVER-7 . I agree combining MRESOLVER-133 , MRESOLVER-7 and MRESOLVER-228 , there would be huge performance improvements, however I don't think BFS + Skip can simply work as it may introduce scope incompatibilities. Before I submitted the PR of skip & reconcile approach in this ticket, I actually noticed the MRESOLVER-133 and tried breadth-first approach but no vain, here is what I've tried: DFS + Skip 1% of 500+ apps failed because some low level transitive dependencies are no longer available in the dependency tree. This is because for a skipped node, I need set children as empty and reuse the result of the node with same GAV but a lower depth. When there are version conflicts in the parent paths, a skipped node might should not be skipped and need reconcile somehow. BFS + Skip This time, I'm not seeing any failures because of missing some low level transitive dependencies. Instead I witnessed the scope of certain dependency nodes in the dependency tree were incorrect, especially for "provided" or "compile" scopes. Check this code , when there are conflict items with different scopes, maven will pick "compile" scope as long as there is a conflict item using "compile" scope.  DFS + Skip + Reconcile Then I come up with this solution. Tested with 2000+ apps in our company, all dependency:tree and dependency:list result remains the same which proves it is enough compatible. If go with BFS + Skip approach, as the scopes of dependencies are no longer the same, we should do some reconcile, however it is hard to implement such reconcile logic to make scope correct with BFS solution.
            michael-o Michael Osipov added a comment - - edited

            wecai, thanks! As far as I understand your explanation several approaches do not play nice together. We must decide which combination is best in terms of stability/reliability and then performance.

            michael-o Michael Osipov added a comment - - edited wecai , thanks! As far as I understand your explanation several approaches do not play nice together. We must decide which combination is best in terms of stability/reliability and then performance.
            wecai wei cai added a comment - - edited

            michael-o ibabiankou 

            Cool! Have read the comments in https://github.com/apache/maven-resolver/pull/142.

            Looks like we now have a road map:

            • DFS -> BFS to make sure dependency tree no changes, might solve MRESOLVER-133
            • BFS + Skip & Reconcile to make sure dependency tree no changes which is for MRESOLVER-228
            • Download artifacts in parallel which is for MRESOLVER-7

            One question is, as Ivan pointed out in MRESOLVER-133 as below, it does not have to use a BFS strategy to solve MRESOLVER-133. Shall we still need to go to BFS?

            -----------------------------------------------------------
            This loop should iterate over reversed list of versions, it should only try to get artifact descriptor and break as soon as got one successfully. If no versions are available, then the build could be failed with appropriate message.

            -----------------------------------------------------------

            If we do not need to go with BFS, the road map would be:

            • Ivan's proposal to solve MRESOLVER-133
            • DFS + Skip & Reconcile to make sure dependency tree no changes which is for MRESOLVER-228

            PR: https://github.com/apache/maven-resolver/pull/136 already implemented this.

            • Download artifacts in parallel which is for MRESOLVER-7

            If we need go with BFS, I already did a BFS experiment before I submitted the PR of DFS + Skip & Reconcile approach for MRESOLVER-228, let me prepare the very first PR, the PR should include pure BFS solution, no skip, no reconcile, then I will go with the BFS + Skip & Reconcile as the 2nd PR.

             

            wecai wei cai added a comment - - edited michael-o ibabiankou   Cool! Have read the comments in https://github.com/apache/maven-resolver/pull/142. Looks like we now have a road map: DFS -> BFS to make sure dependency tree no changes, might solve MRESOLVER-133 BFS + Skip & Reconcile to make sure dependency tree no changes which is for MRESOLVER-228 Download artifacts in parallel which is for MRESOLVER-7 One question is, as Ivan pointed out in MRESOLVER-133 as below, it does not have to use a BFS strategy to solve MRESOLVER-133 . Shall we still need to go to BFS? ----------------------------------------------------------- This loop should iterate over reversed list of versions, it should only try to get artifact descriptor and break as soon as got one successfully. If no versions are available, then the build could be failed with appropriate message. ----------------------------------------------------------- If we do not need to go with BFS, the road map would be: Ivan's proposal to solve MRESOLVER-133 DFS + Skip & Reconcile to make sure dependency tree no changes which is for MRESOLVER-228 PR: https://github.com/apache/maven-resolver/pull/136 already implemented this. Download artifacts in parallel which is for MRESOLVER-7 If we need go with BFS, I already did a BFS experiment before I submitted the PR of DFS + Skip & Reconcile approach for MRESOLVER-228 , let me prepare the very first PR, the PR should include pure BFS solution, no skip, no reconcile, then I will go with the BFS + Skip & Reconcile as the 2nd PR.  
            ibabiankou Ivan Babiankou added a comment -

            BFS is needed for the parallel artifact download, because you can only download in parallel descriptors on the same level of the graph. It won't solve MRESOLVER-133 because the problem that ticket want to solve is "Resolve highest version in the range, instead resolving all versions and keeping one". We should probably update title and description of that ticket and create separate one for DFS > BFS.

            I would keep MRESOLVER-133 out of the scope for now, as it does not seem to be as trivial as I suggested in that comment. It also seems to change the behavior, where as the rest only avoids unnecessary work + does the same things in parallel. 

            From what I can tell so far the outlined plan makes the most sense:

            1. DFS > BFS - preparation for parallel download
            2. Skip & Reconcile - avoid unnecessary version resolution
            3. Download descriptors in parallel
            ibabiankou Ivan Babiankou added a comment - BFS is needed for the parallel artifact download, because you can only download in parallel descriptors on the same level of the graph. It won't solve MRESOLVER-133 because the problem that ticket want to solve is "Resolve highest version in the range, instead resolving all versions and keeping one". We should probably update title and description of that ticket and create separate one for DFS > BFS. I would keep MRESOLVER-133 out of the scope for now, as it does not seem to be as trivial as I suggested in that comment. It also seems to change the behavior, where as the rest only avoids unnecessary work + does the same things in parallel.  From what I can tell so far the outlined plan makes the most sense: DFS > BFS - preparation for parallel download Skip & Reconcile - avoid unnecessary version resolution Download descriptors in parallel
            michael-o Michael Osipov added a comment - - edited

            One question is, as Ivan pointed out in MRESOLVER-133 as below, it does not have to use a BFS strategy to solve MRESOLVER-133. Shall we still need to go to BFS?

            I think that resolving highest only should be completely decoupled from BFS. BFS might be necessary, but resolving all versions is yet another optimization.

            I would keep MRESOLVER-133 out of the scope for now, as it does not seem to be as trivial as I suggested in that comment. It also seems to change the behavior, where as the rest only avoids unnecessary work + does the same things in parallel.

            Fully agree. Lets not mix things here.

            ibabiankou, I fully agree with your three step plan.

            Folks, if you find an alternating behavior during your work, i.e., resolution is different then we need to discuss wether the new outcome is a consistency bugfix or a regression. If you look at MNG, there are plenty issues related to dependency resolution.

            michael-o Michael Osipov added a comment - - edited One question is, as Ivan pointed out in MRESOLVER-133 as below, it does not have to use a BFS strategy to solve MRESOLVER-133 . Shall we still need to go to BFS? I think that resolving highest only should be completely decoupled from BFS. BFS might be necessary, but resolving all versions is yet another optimization. I would keep MRESOLVER-133 out of the scope for now, as it does not seem to be as trivial as I suggested in that comment. It also seems to change the behavior, where as the rest only avoids unnecessary work + does the same things in parallel. Fully agree. Lets not mix things here. ibabiankou , I fully agree with your three step plan. Folks, if you find an alternating behavior during your work, i.e., resolution is different then we need to discuss wether the new outcome is a consistency bugfix or a regression. If you look at MNG, there are plenty issues related to dependency resolution.
            michael-o Michael Osipov added a comment -

            Related issues: MNG-5988, MNG-5761, MNG-6357

            michael-o Michael Osipov added a comment - Related issues: MNG-5988 , MNG-5761 , MNG-6357
            wecai wei cai added a comment -

            michael-o  ibabiankou 

            Here is the pure BFS implementation:
            https://github.com/apache/maven-resolver/pull/144

            Please kindly review & merge.
            As discussed, I will continue submit the PR for skip approach once this patch is merged.

            wecai wei cai added a comment - michael-o   ibabiankou   Here is the pure BFS implementation: https://github.com/apache/maven-resolver/pull/144 Please kindly review & merge. As discussed, I will continue submit the PR for skip approach once this patch is merged.
            wecai wei cai added a comment - - edited

            As solution in this Jira is based on DFS, created another Jira https://issues.apache.org/jira/browse/MRESOLVER-247 for the Skip solution based on BFS.

            This Jira is out of date, please check https://issues.apache.org/jira/browse/MRESOLVER-240 and https://issues.apache.org/jira/browse/MRESOLVER-247 for more details.

            wecai wei cai added a comment - - edited As solution in this Jira is based on DFS, created another Jira https://issues.apache.org/jira/browse/MRESOLVER-247 for the Skip solution based on BFS. This Jira is out of date, please check https://issues.apache.org/jira/browse/MRESOLVER-240 and https://issues.apache.org/jira/browse/MRESOLVER-247 for more details.

            People

              michael-o Michael Osipov
              wecai wei cai
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: