Lucene - Core
  1. Lucene - Core
  2. LUCENE-6477

Add BKD tree for spatial shape query intersecting indexed points

    Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 5.3, 6.0
    • Component/s: None
    • Labels:
      None
    • Lucene Fields:
      New

      Description

      I'd like to explore using dedicated spatial trees for faster shape
      intersection filters than postings-based implementations.

      I implemented the tree data structure from
      https://www.cs.duke.edu/~pankaj/publications/papers/bkd-sstd.pdf

      The idea is simple: it builds a full binary tree, partitioning 2D
      space, alternately on lat and then lon, into smaller and smaller
      rectangles until a leaf has <= N (default 1024) points.

      It cannot index shapes (just points), and can then do fast shape
      intersection queries. Multi-valued fields are supported.

      I only implemented the "point is contained in this bounding box" query
      for now, but I think polygon shape querying should be easy to
      implement using the same approach from LUCENE-6450.

      For indexing, you add BKDPointField (takes lat, lon) to your doc, and
      must set up your Codec use BKDTreeDocValuesFormat for that field.
      This DV format wraps Lucene50DVFormat, but then builds the disk-based
      BKD tree structure on the side. BKDPointInBBoxQuery then requires this
      DVFormat, and casts it to gain access to the tree.

      I quantize each incoming double lat/lon to 32 bits precision (so 64
      bits per point) = ~9 milli-meter lon precision at the equator, I
      think.

      1. LUCENE-6477.patch
        130 kB
        Michael McCandless
      2. LUCENE-6477.patch
        157 kB
        Michael McCandless
      3. LUCENE-6477.patch
        157 kB
        Michael McCandless
      4. LUCENE-6477.patch
        159 kB
        Michael McCandless
      5. LUCENE-6477.patch
        120 kB
        Michael McCandless
      6. LUCENE-6477.patch
        116 kB
        Michael McCandless

        Activity

        Hide
        Michael McCandless added a comment -

        Work-in-progress patch, with quite a few nocommits still, but the core
        idea is working.

        Show
        Michael McCandless added a comment - Work-in-progress patch, with quite a few nocommits still, but the core idea is working.
        Hide
        Michael McCandless added a comment -

        I tested performance with the same "bounding boxes around London, UK"
        on 60.8M points test from LUCENE-6450:

          Index size disk: 1.2 GB
          Index size heap: 0.75 MB
          Index time: 636.0 seconds (incl. forceMerge)
          Mean query time: .0068 sec
        

        Indexing time is a bit slower (makes heavy use of OfflineSorter, but
        should be O(N*log(N) overall).

        It's very fast at search time: ~5.7X faster than GeoHashPrefixTree.

        It also uses very little heap (0.75 MB) at search time for the inner
        nodes of the tree.

        It currently allocates FixedBitSet(maxDoc) per query & segment at
        search time (like spatial does with RecursivePrefixTreeStrategy, I
        think?). Really it should use BitDocIdSet.Builder, to start sparse
        and upgrade to FixedBitSet only if result set will be "big-ish", but
        when I do that the query is 2.4X slower, which is frustrating. I think
        we could try using SentinelIntSet for the sparse case, like spatial
        does in some cases.

        Show
        Michael McCandless added a comment - I tested performance with the same "bounding boxes around London, UK" on 60.8M points test from LUCENE-6450 : Index size disk: 1.2 GB Index size heap: 0.75 MB Index time: 636.0 seconds (incl. forceMerge) Mean query time: .0068 sec Indexing time is a bit slower (makes heavy use of OfflineSorter, but should be O(N*log(N) overall). It's very fast at search time: ~5.7X faster than GeoHashPrefixTree. It also uses very little heap (0.75 MB) at search time for the inner nodes of the tree. It currently allocates FixedBitSet(maxDoc) per query & segment at search time (like spatial does with RecursivePrefixTreeStrategy, I think?). Really it should use BitDocIdSet.Builder, to start sparse and upgrade to FixedBitSet only if result set will be "big-ish", but when I do that the query is 2.4X slower, which is frustrating. I think we could try using SentinelIntSet for the sparse case, like spatial does in some cases.
        Hide
        Michael McCandless added a comment -

        New patch, fixing the last nocommits. I think it's ready.

        Show
        Michael McCandless added a comment - New patch, fixing the last nocommits. I think it's ready.
        Hide
        David Smiley added a comment -

        Mike,
        This looks cool; it's been on my backlog of interesting patches to look at. But I noticed it's not in the spatial module and doesn't have "spatial" in the package. Can you please explain to me your rationale why this is so? We have 3 highlighters that have no APIs in common in the highlighter module, we've got lots of suggesters in the suggest module... why wouldn't all spatial capabilities be in the spatial module? It needn't fit any interface or have the same dependencies as anything else in the spatial module. The modules are both an organizational construct and it reduces the total 'jar' weight for those that don't need what's in any one module. I could imagine "sandbox" being an alternative destination if you feel that's appropriate.

        Show
        David Smiley added a comment - Mike, This looks cool; it's been on my backlog of interesting patches to look at. But I noticed it's not in the spatial module and doesn't have "spatial" in the package. Can you please explain to me your rationale why this is so? We have 3 highlighters that have no APIs in common in the highlighter module, we've got lots of suggesters in the suggest module... why wouldn't all spatial capabilities be in the spatial module? It needn't fit any interface or have the same dependencies as anything else in the spatial module. The modules are both an organizational construct and it reduces the total 'jar' weight for those that don't need what's in any one module. I could imagine "sandbox" being an alternative destination if you feel that's appropriate.
        Hide
        Robert Muir added a comment -

        It looks like Mike's patch targets the sandbox entirely already... except for some BitDocIdSet changes that need a little explanation

        Personally, I think this is a good approach when things aren't fully baked. Especially in this case where it has not-fully-baked fileformats, nothing anyone wants to infer backwards compatibility for at the very least.

        I am a bit worried about OfflineSorter, it seems it will use java.io.tmpdir, which a lot of people probably don't configure for "serious" big files like this? And it is not as robust if things can't get cleaned up on windows and so on. But OfflineSorter has some ByteSequenceReader/Writer abstractions, i wonder if long-term we can plug those into our Directory api. Maybe its good to think about a documented way for codecs to freely and easily use scratch files like this, where IndexFileDeleter could help out. For another issue, it shouldn't hold this one up...

        Show
        Robert Muir added a comment - It looks like Mike's patch targets the sandbox entirely already... except for some BitDocIdSet changes that need a little explanation Personally, I think this is a good approach when things aren't fully baked. Especially in this case where it has not-fully-baked fileformats, nothing anyone wants to infer backwards compatibility for at the very least. I am a bit worried about OfflineSorter, it seems it will use java.io.tmpdir, which a lot of people probably don't configure for "serious" big files like this? And it is not as robust if things can't get cleaned up on windows and so on. But OfflineSorter has some ByteSequenceReader/Writer abstractions, i wonder if long-term we can plug those into our Directory api. Maybe its good to think about a documented way for codecs to freely and easily use scratch files like this, where IndexFileDeleter could help out. For another issue, it shouldn't hold this one up...
        Hide
        David Smiley added a comment -

        It looks like Mike's patch targets the sandbox entirely already

        Doh! Why yes it is; I overlooked that. I'll raise my concern at a later time when it's stable and ready to leave the sandbox. If it were me, it might as well be in an appropriate package even in the sandbox but it's not important.

        Show
        David Smiley added a comment - It looks like Mike's patch targets the sandbox entirely already Doh! Why yes it is; I overlooked that. I'll raise my concern at a later time when it's stable and ready to leave the sandbox. If it were me, it might as well be in an appropriate package even in the sandbox but it's not important.
        Hide
        Michael McCandless added a comment -

        I made a fun video animation, to visualize how the BKD tree partitions space and then visits cells at search time, described here: https://plus.google.com/+MichaelMcCandless/posts/6QveLQs7RW6

        Show
        Michael McCandless added a comment - I made a fun video animation, to visualize how the BKD tree partitions space and then visits cells at search time, described here: https://plus.google.com/+MichaelMcCandless/posts/6QveLQs7RW6
        Hide
        Michael McCandless added a comment -

        It looks like Mike's patch targets the sandbox entirely already...

        Right, I think this really must start in sandbox... it's completely new and quite complex, likely has exciting bugs, writes its own index files and there's no guarantee of back compat, etc.

        Longer term, when/if it graduates, I don't think it should go into lucene/spatial: I don't like how "heavy" lucene/spatial is, with external libs bleeding into the public APIs (spatial4j, JTS), too-many-abstractions causing barriers for new contributions, test case base classes strangely forked from external libraries and thus failing to print "reproduce with" lines, etc.

        I think we need competing/alternative approaches that are much more lightweight, and focus on the common case (index points, search shapes), like LUCENE-6450 and this patch... and I think such support really could/should be in Lucene's core, or "spatial2" or "spatiallight" or something. And then I think that leaves lucene/spatial supporting the exotic/difficult cases that require (?) all the abstractions?

        except for some BitDocIdSet changes that need a little explanation

        Woops, that was a leftover from struggling ... I'll revert it.

        Maybe its good to think about a documented way for codecs to freely and easily use scratch files like this, where IndexFileDeleter could help out.

        That's a good idea ... it's true this is a heavy user of OfflineSorter, because it needs to recursively partition all points, separately in lat and lon.

        I'll post a new patch soon, adding missing javadocs, marking things experimental, adding polygon support.

        Show
        Michael McCandless added a comment - It looks like Mike's patch targets the sandbox entirely already... Right, I think this really must start in sandbox... it's completely new and quite complex, likely has exciting bugs, writes its own index files and there's no guarantee of back compat, etc. Longer term, when/if it graduates, I don't think it should go into lucene/spatial: I don't like how "heavy" lucene/spatial is, with external libs bleeding into the public APIs (spatial4j, JTS), too-many-abstractions causing barriers for new contributions, test case base classes strangely forked from external libraries and thus failing to print "reproduce with" lines, etc. I think we need competing/alternative approaches that are much more lightweight, and focus on the common case (index points, search shapes), like LUCENE-6450 and this patch... and I think such support really could/should be in Lucene's core, or "spatial2" or "spatiallight" or something. And then I think that leaves lucene/spatial supporting the exotic/difficult cases that require (?) all the abstractions? except for some BitDocIdSet changes that need a little explanation Woops, that was a leftover from struggling ... I'll revert it. Maybe its good to think about a documented way for codecs to freely and easily use scratch files like this, where IndexFileDeleter could help out. That's a good idea ... it's true this is a heavy user of OfflineSorter, because it needs to recursively partition all points, separately in lat and lon. I'll post a new patch soon, adding missing javadocs, marking things experimental, adding polygon support.
        Hide
        Michael McCandless added a comment -

        I haven't explored this yet, but talking to Karl Wright it sounds like integrating geo3d into the BKD tree would be quite simple, and would "solve" the inefficiency now of visiting leaf cells outside the shape but inside its bounding box, probably giving a good speedup over the simple polygon filtering we do in LUCENE-6450.

        Show
        Michael McCandless added a comment - I haven't explored this yet, but talking to Karl Wright it sounds like integrating geo3d into the BKD tree would be quite simple, and would "solve" the inefficiency now of visiting leaf cells outside the shape but inside its bounding box, probably giving a good speedup over the simple polygon filtering we do in LUCENE-6450 .
        Hide
        David Smiley added a comment -

        Longer term, when/if it graduates, I don't think it should go into lucene/spatial: I don't like how "heavy" lucene/spatial is, with external libs bleeding into the public APIs (spatial4j, JTS), too-many-abstractions causing barriers for new contributions, test case base classes strangely forked from external libraries and thus failing to print "reproduce with" lines, etc.

        I think we need competing/alternative approaches that are much more lightweight, and focus on the common case (index points, search shapes), like LUCENE-6450 and this patch... and I think such support really could/should be in Lucene's core, or "spatial2" or "spatiallight" or something. And then I think that leaves lucene/spatial supporting the exotic/difficult cases that require all the abstractions?

        In a nutshell, you don't like the code in the spatial module for a variety of reasons, but nonetheless that doesn't prevent any other code from being there too that meets your satisfaction.

        Separately from the topic of spatial code belonging in the spatial module, I definitely think it's worth re-looking at the (only) core/base abstraction in the spatial module – SpatialStrategy, to come up with a new design that is not dependent on external libs. p.s. JTS is most definitely not part of the public API of Lucene spatial – preventing that was half the reason for Spatial4j's existence. The other half is about potential re-use for shape/geometry calculations that otherwise has nothing to do with Lucene.

        Show
        David Smiley added a comment - Longer term, when/if it graduates, I don't think it should go into lucene/spatial: I don't like how "heavy" lucene/spatial is, with external libs bleeding into the public APIs (spatial4j, JTS), too-many-abstractions causing barriers for new contributions, test case base classes strangely forked from external libraries and thus failing to print "reproduce with" lines, etc. I think we need competing/alternative approaches that are much more lightweight, and focus on the common case (index points, search shapes), like LUCENE-6450 and this patch... and I think such support really could/should be in Lucene's core, or "spatial2" or "spatiallight" or something. And then I think that leaves lucene/spatial supporting the exotic/difficult cases that require all the abstractions? In a nutshell, you don't like the code in the spatial module for a variety of reasons, but nonetheless that doesn't prevent any other code from being there too that meets your satisfaction. Separately from the topic of spatial code belonging in the spatial module, I definitely think it's worth re-looking at the (only) core/base abstraction in the spatial module – SpatialStrategy, to come up with a new design that is not dependent on external libs. p.s. JTS is most definitely not part of the public API of Lucene spatial – preventing that was half the reason for Spatial4j's existence. The other half is about potential re-use for shape/geometry calculations that otherwise has nothing to do with Lucene.
        Hide
        Michael McCandless added a comment -

        New patch, removing the BitDocIdSet changes, adding javadocs (ant precommit passes).

        I also added BKDPointInPolygonQuery, using same approach as LUCENE-6450, and folded it simplistically into the test by sometimes randomly using it to make the rect bounding box.

        Curiously, I had to use 5 points (first and last point repeated) to fully close the polygon myself ... is this expected (Nicholas Knize?).

        I'll remove IndexAndSearchOpenStreetMaps.java before committing ...

        Show
        Michael McCandless added a comment - New patch, removing the BitDocIdSet changes, adding javadocs (ant precommit passes). I also added BKDPointInPolygonQuery, using same approach as LUCENE-6450 , and folded it simplistically into the test by sometimes randomly using it to make the rect bounding box. Curiously, I had to use 5 points (first and last point repeated) to fully close the polygon myself ... is this expected ( Nicholas Knize ?). I'll remove IndexAndSearchOpenStreetMaps.java before committing ...
        Hide
        David Smiley added a comment -

        Curiously, I had to use 5 points (first and last point repeated) to fully close the polygon myself ... is this expected (Nicholas Knize?).

        Yes.

        Might you try a 360x360 world like I mentioned on your G+ post?

        Show
        David Smiley added a comment - Curiously, I had to use 5 points (first and last point repeated) to fully close the polygon myself ... is this expected (Nicholas Knize?). Yes. Might you try a 360x360 world like I mentioned on your G+ post?
        Hide
        Nicholas Knize added a comment -

        Curiously, I had to use 5 points (first and last point repeated) to fully close the polygon myself ... is this expected

        Yes, in order to create the closed edge. To keep everything simple and fast there is no polygon validation. So that vanilla method obviously doesn't work for OGC compliant polys. It would need to be passed an array of vertex coordinates translated "correctly".

        Show
        Nicholas Knize added a comment - Curiously, I had to use 5 points (first and last point repeated) to fully close the polygon myself ... is this expected Yes, in order to create the closed edge. To keep everything simple and fast there is no polygon validation. So that vanilla method obviously doesn't work for OGC compliant polys. It would need to be passed an array of vertex coordinates translated "correctly".
        Hide
        Michael McCandless added a comment -

        Yes, in order to create the closed edge.

        OK thanks for confirming ... I'll try to add some simple validation.

        Show
        Michael McCandless added a comment - Yes, in order to create the closed edge. OK thanks for confirming ... I'll try to add some simple validation.
        Hide
        Michael McCandless added a comment -

        Another iteration, adding validation for the polygon query (just checking that it's "closed").

        I also stopped redundantly storing the 32 bit lat, 32 bit lon in the BKD tree, and instead look it up from the wrapped DVs ... I had thought the loss of locality would hurt but it turns out it's only ~8% slower on the "bbox around London" test, and it saves 66% of the size of the BKD tree.

        Show
        Michael McCandless added a comment - Another iteration, adding validation for the polygon query (just checking that it's "closed"). I also stopped redundantly storing the 32 bit lat, 32 bit lon in the BKD tree, and instead look it up from the wrapped DVs ... I had thought the loss of locality would hurt but it turns out it's only ~8% slower on the "bbox around London" test, and it saves 66% of the size of the BKD tree.
        Hide
        Michael McCandless added a comment -

        Might you try a 360x360 world like I mentioned on your G+ post?

        Sorry, I haven't tried this yet ... I'd like to instead try doing a better job picking the pivot dimension: right now I just alternate lat and lon on each recursion, but in reading up on KD trees, it's (maybe) better to pick the dimension that has the most variance so you tend to get more square leaf cells ... I'll try exploring this.

        Show
        Michael McCandless added a comment - Might you try a 360x360 world like I mentioned on your G+ post? Sorry, I haven't tried this yet ... I'd like to instead try doing a better job picking the pivot dimension: right now I just alternate lat and lon on each recursion, but in reading up on KD trees, it's (maybe) better to pick the dimension that has the most variance so you tend to get more square leaf cells ... I'll try exploring this.
        Hide
        Michael McCandless added a comment -

        Another iteration, improving the pivot selection on each recursion to try to make closer-to-square-shaped leaf cells. All I do is pick the longer dimension to slice, instead of just alternating lat/lon like before.

        This gives a ~7% speedup on the "bboxes around London over OpenStreetMap points" benchmark.

        The same number of leaf cells must be visited as before (since it's a fixed 512-1024 points per leaf cell), but because the cells are more square shaped, there are fewer cells that cross the boundary and those cells are slower because we must pull the lat/lon for each point and check it against the query shape (vs cells we know are fully enclosed, where we just blindly add all docIDs).

        I made another video, and you can definitely see that the cells are "less slivery": https://plus.google.com/+MichaelMcCandless/posts/Daj9FgYPdtv

        Show
        Michael McCandless added a comment - Another iteration, improving the pivot selection on each recursion to try to make closer-to-square-shaped leaf cells. All I do is pick the longer dimension to slice, instead of just alternating lat/lon like before. This gives a ~7% speedup on the "bboxes around London over OpenStreetMap points" benchmark. The same number of leaf cells must be visited as before (since it's a fixed 512-1024 points per leaf cell), but because the cells are more square shaped, there are fewer cells that cross the boundary and those cells are slower because we must pull the lat/lon for each point and check it against the query shape (vs cells we know are fully enclosed, where we just blindly add all docIDs). I made another video, and you can definitely see that the cells are "less slivery": https://plus.google.com/+MichaelMcCandless/posts/Daj9FgYPdtv
        Hide
        Michael McCandless added a comment -

        Bump fix version to 5.3

        Show
        Michael McCandless added a comment - Bump fix version to 5.3
        Hide
        Michael McCandless added a comment -

        New patch, removing nocommits, adding javadocs. The random test seems to survive beasting ... I think it's ready. I'll commit soon...

        Show
        Michael McCandless added a comment - New patch, removing nocommits, adding javadocs. The random test seems to survive beasting ... I think it's ready. I'll commit soon...
        Hide
        ASF subversion and git services added a comment -

        Commit 1683340 from Michael McCandless in branch 'dev/trunk'
        [ https://svn.apache.org/r1683340 ]

        LUCENE-6477: add experimental but very fast BKD tree for geo-spatial 'bbox/poly contains indexed lat/lon points' queries

        Show
        ASF subversion and git services added a comment - Commit 1683340 from Michael McCandless in branch 'dev/trunk' [ https://svn.apache.org/r1683340 ] LUCENE-6477 : add experimental but very fast BKD tree for geo-spatial 'bbox/poly contains indexed lat/lon points' queries
        Hide
        ASF subversion and git services added a comment -

        Commit 1683348 from Michael McCandless in branch 'dev/branches/branch_5x'
        [ https://svn.apache.org/r1683348 ]

        LUCENE-6477: add experimental but very fast BKD tree for geo-spatial 'bbox/poly contains indexed lat/lon points' queries

        Show
        ASF subversion and git services added a comment - Commit 1683348 from Michael McCandless in branch 'dev/branches/branch_5x' [ https://svn.apache.org/r1683348 ] LUCENE-6477 : add experimental but very fast BKD tree for geo-spatial 'bbox/poly contains indexed lat/lon points' queries
        Hide
        ASF subversion and git services added a comment -

        Commit 1683446 from Michael McCandless in branch 'dev/trunk'
        [ https://svn.apache.org/r1683446 ]

        LUCENE-6477: make nightly test less evil

        Show
        ASF subversion and git services added a comment - Commit 1683446 from Michael McCandless in branch 'dev/trunk' [ https://svn.apache.org/r1683446 ] LUCENE-6477 : make nightly test less evil
        Hide
        ASF subversion and git services added a comment -

        Commit 1683447 from Michael McCandless in branch 'dev/branches/branch_5x'
        [ https://svn.apache.org/r1683447 ]

        LUCENE-6477: make nightly test less evil

        Show
        ASF subversion and git services added a comment - Commit 1683447 from Michael McCandless in branch 'dev/branches/branch_5x' [ https://svn.apache.org/r1683447 ] LUCENE-6477 : make nightly test less evil
        Hide
        ASF subversion and git services added a comment -

        Commit 1683448 from Michael McCandless in branch 'dev/trunk'
        [ https://svn.apache.org/r1683448 ]

        LUCENE-6477: allow boundary tolerance for poly query verify; fail test fast if one thread fails

        Show
        ASF subversion and git services added a comment - Commit 1683448 from Michael McCandless in branch 'dev/trunk' [ https://svn.apache.org/r1683448 ] LUCENE-6477 : allow boundary tolerance for poly query verify; fail test fast if one thread fails
        Hide
        ASF subversion and git services added a comment -

        Commit 1683449 from Michael McCandless in branch 'dev/branches/branch_5x'
        [ https://svn.apache.org/r1683449 ]

        LUCENE-6477: allow boundary tolerance for poly query verify; fail test fast if one thread fails

        Show
        ASF subversion and git services added a comment - Commit 1683449 from Michael McCandless in branch 'dev/branches/branch_5x' [ https://svn.apache.org/r1683449 ] LUCENE-6477 : allow boundary tolerance for poly query verify; fail test fast if one thread fails
        Hide
        ASF subversion and git services added a comment -

        Commit 1684084 from Michael McCandless in branch 'dev/trunk'
        [ https://svn.apache.org/r1684084 ]

        LUCENE-6477: tweak TODOs/javadocs/comments

        Show
        ASF subversion and git services added a comment - Commit 1684084 from Michael McCandless in branch 'dev/trunk' [ https://svn.apache.org/r1684084 ] LUCENE-6477 : tweak TODOs/javadocs/comments
        Hide
        ASF subversion and git services added a comment -

        Commit 1684085 from Michael McCandless in branch 'dev/branches/branch_5x'
        [ https://svn.apache.org/r1684085 ]

        LUCENE-6477: tweak TODOs/javadocs/comments

        Show
        ASF subversion and git services added a comment - Commit 1684085 from Michael McCandless in branch 'dev/branches/branch_5x' [ https://svn.apache.org/r1684085 ] LUCENE-6477 : tweak TODOs/javadocs/comments
        Hide
        ASF subversion and git services added a comment -

        Commit 1684086 from Michael McCandless in branch 'dev/trunk'
        [ https://svn.apache.org/r1684086 ]

        LUCENE-6477: include delegate heap usage in Accountable; fix javadocs

        Show
        ASF subversion and git services added a comment - Commit 1684086 from Michael McCandless in branch 'dev/trunk' [ https://svn.apache.org/r1684086 ] LUCENE-6477 : include delegate heap usage in Accountable; fix javadocs
        Hide
        ASF subversion and git services added a comment -

        Commit 1684087 from Michael McCandless in branch 'dev/branches/branch_5x'
        [ https://svn.apache.org/r1684087 ]

        LUCENE-6477: include delegate heap usage in Accountable; fix javadocs

        Show
        ASF subversion and git services added a comment - Commit 1684087 from Michael McCandless in branch 'dev/branches/branch_5x' [ https://svn.apache.org/r1684087 ] LUCENE-6477 : include delegate heap usage in Accountable; fix javadocs
        Hide
        Shalin Shekhar Mangar added a comment -

        Bulk close for 5.3.0 release

        Show
        Shalin Shekhar Mangar added a comment - Bulk close for 5.3.0 release

          People

          • Assignee:
            Michael McCandless
            Reporter:
            Michael McCandless
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development