Lucene - Core
  1. Lucene - Core
  2. LUCENE-6776

Randomized planet model shows up additional XYZBounds errors

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 5.4, 6.0
    • Component/s: modules/spatial
    • Labels:
      None
    • Lucene Fields:
      New

      Description

      Adding randomized PlanetModel construction causes points to be generated inside a shape that are outside XYZBounds. Michael McCandless please take note.

      1. LUCENE-6776.patch
        40 kB
        Karl Wright
      2. LUCENE-6776.patch
        37 kB
        Karl Wright
      3. LUCENE-6776.patch
        33 kB
        Karl Wright
      4. LUCENE-6776.patch
        30 kB
        Michael McCandless
      5. LUCENE-6776.patch
        30 kB
        Michael McCandless
      6. LUCENE-6776.patch
        27 kB
        Karl Wright
      7. LUCENE-6776.patch
        25 kB
        Karl Wright
      8. LUCENE-6776.patch
        3 kB
        Karl Wright

        Activity

        Hide
        Karl Wright added a comment -

        This patch demonstrates the failure pretty much immediately, with:

        ant test  -Dtestcase=TestGeo3DPointField -Dtests.method=testGeo3DRelations -Dtests.seed=5B916E77D4C84BE5 -Dtests.slow=true -Dtests.locale=sv -Dtests.timezone=Pacific/Norfolk -Dtests.asserts=true -Dtests.file.encoding=US-ASCII
        
        Show
        Karl Wright added a comment - This patch demonstrates the failure pretty much immediately, with: ant test -Dtestcase=TestGeo3DPointField -Dtests.method=testGeo3DRelations -Dtests.seed=5B916E77D4C84BE5 -Dtests.slow= true -Dtests.locale=sv -Dtests.timezone=Pacific/Norfolk -Dtests.asserts= true -Dtests.file.encoding=US-ASCII
        Hide
        Karl Wright added a comment -

        The failure is, once again, a single point that is outside the XYZBounds computed for the shape. Stay tuned for further analysis, probably later this morning. Either there's a mathematical error, or the calculation itself is too problematic in its current form due to error bars.

        Show
        Karl Wright added a comment - The failure is, once again, a single point that is outside the XYZBounds computed for the shape. Stay tuned for further analysis, probably later this morning. Either there's a mathematical error, or the calculation itself is too problematic in its current form due to error bars.
        Hide
        Karl Wright added a comment -

        The analysis: The Z bound is unaffected. The X and Y bounds, however, are being computed improperly due to a wrong assumption I made when deriving the method for computing them. They work for spheres accurately, and they work with increasing inaccuracy for increasingly non-spherical ellipsoids.

        The only fix is to derive an accurate formula for the min/max in X and in Y. I believe this is doable, but I'll have to actually do it before I'm sure. Due to other time constraints, this is not likely to be ready for at least a day.

        Meanwhile, the value of FUDGE_FACTOR chosen is adequate for WGS84, but will not stand up to any planet model that is more oblate than that. When a true fix is in place, FUDGE_FACTOR may be reducible to a very small value.

        Show
        Karl Wright added a comment - The analysis: The Z bound is unaffected. The X and Y bounds, however, are being computed improperly due to a wrong assumption I made when deriving the method for computing them. They work for spheres accurately, and they work with increasing inaccuracy for increasingly non-spherical ellipsoids. The only fix is to derive an accurate formula for the min/max in X and in Y. I believe this is doable, but I'll have to actually do it before I'm sure. Due to other time constraints, this is not likely to be ready for at least a day. Meanwhile, the value of FUDGE_FACTOR chosen is adequate for WGS84, but will not stand up to any planet model that is more oblate than that. When a true fix is in place, FUDGE_FACTOR may be reducible to a very small value.
        Hide
        Michael McCandless added a comment -

        Oh it sounds like good news that this means we can drop FUDGE_FACTOR back down! Thanks for digging Karl Wright.

        Show
        Michael McCandless added a comment - Oh it sounds like good news that this means we can drop FUDGE_FACTOR back down! Thanks for digging Karl Wright .
        Hide
        Karl Wright added a comment -

        Ok, here's the math. This is for the bounds in x. I had to go with the full treatment using lagrange multipliers. Haven't yet debugged it either, but clearly it's doable; you get a quadratic in a parameterization variable, which can then be applied to produce a pair of (x,y,z) values:

        Another approach: Lagrange multipliers
        
        For this, we need grad(F(x,y,z)) = (dF/dx, dF/dy, dF/dz).
        
        Minimize and maximize f(x,y,z) = x, with respect to g(x,y,z) = Ax + By + Cz - D and h(x,y,z) = x^2/ab^2 + y^2/ab^2 + z^2/c^2 - 1
        
        grad(f(x,y,z)) = (1,0,0)
        grad(g(x,y,z)) = (A,B,C)
        grad(h(x,y,z)) = (2x/ab^2,2y/ab^2,2z/c^2)
        
        Equations we need to simultaneously solve:
        
        grad(f(x,y,z)) = l * grad(g(x,y,z)) + m * grad(h(x,y,z))
        g(x,y,z) = 0
        h(x,y,z) = 0
        
        Equations:
        1 = l*A + m*2x/ab^2
        0 = l*B + m*2y/ab^2
        0 = l*C + m*2z/c^2
        Ax + By + Cz + D = 0
        x^2/ab^2 + y^2/ab^2 + z^2/c^2 - 1 = 0
        
        Solve for x in terms of (l, m):
        
        x = ((1 - l*A) * ab^2 ) / (2 * m)
        y = (-l*B * ab^2) / ( 2 * m)
        z = (-l*C * c^2)/ (2 * m)
        
        Two equations, two unknowns:
        
        A * (((1 - l*A) * ab^2 ) / (2 * m)) + B * ((-l*B * ab^2) / ( 2 * m)) + C * ((-l*C * c^2)/ (2 * m)) + D = 0
        
        and
        
        (((1 - l*A) * ab^2 ) / (2 * m))^2/ab^2 + ((-l*B * ab^2) / ( 2 * m))^2/ab^2 + ((-l*C * c^2)/ (2 * m))^2/c^2 - 1 = 0
        
        Simple: solve for l and m, then find x from it.
        
        (a) Use first equation to find l in terms of m.
        
        A * (((1 - l*A) * ab^2 ) / (2 * m)) + B * ((-l*B * ab^2) / ( 2 * m)) + C * ((-l*C * c^2)/ (2 * m)) + D = 0
        A * ((1 - l*A) * ab^2 ) + B * (-l*B * ab^2) + C * (-l*C * c^2) + D * 2 * m = 0
        A * ab^2 - l*A^2* ab^2 - B^2 * l * ab^2 - C^2 * l * c^2 + D * 2 * m = 0
        - l *(A^2* ab^2 + B^2 * ab^2 + C^2 * c^2) + (A * ab^2 + D * 2 * m) = 0
        l = (A * ab^2 + D * 2 * m) / (A^2* ab^2 + B^2 * ab^2 + C^2 * c^2)
        l = m * 2 * (A * ab^2 + D) / (A^2* ab^2 + B^2 * ab^2 + C^2 * c^2)
        
        For convenience:
        
        k = (A^2* ab^2 + B^2 * ab^2 + C^2 * c^2) / (2 * (A * ab^2 + D))
        
        Then:
        
        m = l * k
        
        
        (b) Simplify the second equation before substitution
        
        (((1 - l*A) * ab^2 ) / (2 * m))^2/ab^2 + ((-l*B * ab^2) / ( 2 * m))^2/ab^2 + ((-l*C * c^2)/ (2 * m))^2/c^2 - 1 = 0
        ((1 - l*A) * ab^2 )^2/ab^2 + (-l*B * ab^2)^2/ab^2 + (-l*C * c^2)^2/c^2 = 4 * m^2
        (1 - l*A)^2 * ab^2 + (-l*B)^2 * ab^2 + (-l*C)^2 * c^2 = 4 * m^2
        (1 - 2*l*A + l^2*A^2) * ab^2 + l^2*B^2 * ab^2 + l^2*C^2 * c^2 = 4 * m^2
        ab^2 - 2*l*A*ab^2 + l^2*A^2*ab^2 + l^2*B^2*ab^2 + l^2*C^2*c^2 = 4 * l^2 * k^2
        
        l^2 * (A^2*ab^2 + B^2*ab^2 + C^2*c^2 - 4 * k^2) - l * (2*A*ab^2) + ab^2 = 0
        
        

        There will be similar code for the y min and max.

        Stay tuned for debugging and eventually a patch.

        Show
        Karl Wright added a comment - Ok, here's the math. This is for the bounds in x. I had to go with the full treatment using lagrange multipliers. Haven't yet debugged it either, but clearly it's doable; you get a quadratic in a parameterization variable, which can then be applied to produce a pair of (x,y,z) values: Another approach: Lagrange multipliers For this , we need grad(F(x,y,z)) = (dF/dx, dF/dy, dF/dz). Minimize and maximize f(x,y,z) = x, with respect to g(x,y,z) = Ax + By + Cz - D and h(x,y,z) = x^2/ab^2 + y^2/ab^2 + z^2/c^2 - 1 grad(f(x,y,z)) = (1,0,0) grad(g(x,y,z)) = (A,B,C) grad(h(x,y,z)) = (2x/ab^2,2y/ab^2,2z/c^2) Equations we need to simultaneously solve: grad(f(x,y,z)) = l * grad(g(x,y,z)) + m * grad(h(x,y,z)) g(x,y,z) = 0 h(x,y,z) = 0 Equations: 1 = l*A + m*2x/ab^2 0 = l*B + m*2y/ab^2 0 = l*C + m*2z/c^2 Ax + By + Cz + D = 0 x^2/ab^2 + y^2/ab^2 + z^2/c^2 - 1 = 0 Solve for x in terms of (l, m): x = ((1 - l*A) * ab^2 ) / (2 * m) y = (-l*B * ab^2) / ( 2 * m) z = (-l*C * c^2)/ (2 * m) Two equations, two unknowns: A * (((1 - l*A) * ab^2 ) / (2 * m)) + B * ((-l*B * ab^2) / ( 2 * m)) + C * ((-l*C * c^2)/ (2 * m)) + D = 0 and (((1 - l*A) * ab^2 ) / (2 * m))^2/ab^2 + ((-l*B * ab^2) / ( 2 * m))^2/ab^2 + ((-l*C * c^2)/ (2 * m))^2/c^2 - 1 = 0 Simple: solve for l and m, then find x from it. (a) Use first equation to find l in terms of m. A * (((1 - l*A) * ab^2 ) / (2 * m)) + B * ((-l*B * ab^2) / ( 2 * m)) + C * ((-l*C * c^2)/ (2 * m)) + D = 0 A * ((1 - l*A) * ab^2 ) + B * (-l*B * ab^2) + C * (-l*C * c^2) + D * 2 * m = 0 A * ab^2 - l*A^2* ab^2 - B^2 * l * ab^2 - C^2 * l * c^2 + D * 2 * m = 0 - l *(A^2* ab^2 + B^2 * ab^2 + C^2 * c^2) + (A * ab^2 + D * 2 * m) = 0 l = (A * ab^2 + D * 2 * m) / (A^2* ab^2 + B^2 * ab^2 + C^2 * c^2) l = m * 2 * (A * ab^2 + D) / (A^2* ab^2 + B^2 * ab^2 + C^2 * c^2) For convenience: k = (A^2* ab^2 + B^2 * ab^2 + C^2 * c^2) / (2 * (A * ab^2 + D)) Then: m = l * k (b) Simplify the second equation before substitution (((1 - l*A) * ab^2 ) / (2 * m))^2/ab^2 + ((-l*B * ab^2) / ( 2 * m))^2/ab^2 + ((-l*C * c^2)/ (2 * m))^2/c^2 - 1 = 0 ((1 - l*A) * ab^2 )^2/ab^2 + (-l*B * ab^2)^2/ab^2 + (-l*C * c^2)^2/c^2 = 4 * m^2 (1 - l*A)^2 * ab^2 + (-l*B)^2 * ab^2 + (-l*C)^2 * c^2 = 4 * m^2 (1 - 2*l*A + l^2*A^2) * ab^2 + l^2*B^2 * ab^2 + l^2*C^2 * c^2 = 4 * m^2 ab^2 - 2*l*A*ab^2 + l^2*A^2*ab^2 + l^2*B^2*ab^2 + l^2*C^2*c^2 = 4 * l^2 * k^2 l^2 * (A^2*ab^2 + B^2*ab^2 + C^2*c^2 - 4 * k^2) - l * (2*A*ab^2) + ab^2 = 0 There will be similar code for the y min and max. Stay tuned for debugging and eventually a patch.
        Hide
        Karl Wright added a comment -

        After fixing one problem with the above math, I got the x bounds working well. Then I implemented y bounds, but unfortunately that code fails an assertion. X and Y are perfectly symmetric with one another so I was able to verify the math exactly, so the problem must be implementation related. Still digging...

        Show
        Karl Wright added a comment - After fixing one problem with the above math, I got the x bounds working well. Then I implemented y bounds, but unfortunately that code fails an assertion. X and Y are perfectly symmetric with one another so I was able to verify the math exactly, so the problem must be implementation related. Still digging...
        Hide
        Karl Wright added a comment -

        Michael McCandless This patch calculates x and y bounds using lagrange multipliers. It seems to generally work, but it failed a beasting run. Unfortunately, the failure did not reproduce, so not quite sure what happened there. Am trying to get a reproducible failure.

        Show
        Karl Wright added a comment - Michael McCandless This patch calculates x and y bounds using lagrange multipliers. It seems to generally work, but it failed a beasting run. Unfortunately, the failure did not reproduce, so not quite sure what happened there. Am trying to get a reproducible failure.
        Hide
        Karl Wright added a comment -

        Michael McCandless: It appears that GeoPath's occasionally have some trouble when the planet model gets more extreme than WGS84. It will take some research to get to the bottom of that. Unfortunately I can't reproduce any of the failures that beasting finds!! If there's any way to fix this I'd greatly appreciate it...

        Show
        Karl Wright added a comment - Michael McCandless : It appears that GeoPath's occasionally have some trouble when the planet model gets more extreme than WGS84. It will take some research to get to the bottom of that. Unfortunately I can't reproduce any of the failures that beasting finds!! If there's any way to fix this I'd greatly appreciate it...
        Hide
        Michael McCandless added a comment -

        Unfortunately I can't reproduce any of the failures that beasting finds!! If there's any way to fix this I'd greatly appreciate it...

        Hmm that's very bad.

        Maybe try dropping numThreads to 1 (and put a nocommit comment) and find a failing seed and then see if that fixes the "unable to repro"?

        What exact command line are you using for beasting? Maybe there is a bug in the "Reproduce with: XXX" logic.

        Show
        Michael McCandless added a comment - Unfortunately I can't reproduce any of the failures that beasting finds!! If there's any way to fix this I'd greatly appreciate it... Hmm that's very bad. Maybe try dropping numThreads to 1 (and put a nocommit comment) and find a failing seed and then see if that fixes the "unable to repro"? What exact command line are you using for beasting? Maybe there is a bug in the "Reproduce with: XXX" logic.
        Hide
        Karl Wright added a comment -

        This is what I've been doing:

        ant beast -Dbeast.iters=100 -Dtestcase=TestGeo3DPointField -Dtestmethod=testGeo3DRelations -Dtests.dups=6 -Dtests.iters=10 >capture 2>capture2
        
        Show
        Karl Wright added a comment - This is what I've been doing: ant beast -Dbeast.iters=100 -Dtestcase=TestGeo3DPointField -Dtestmethod=testGeo3DRelations -Dtests.dups=6 -Dtests.iters=10 >capture 2>capture2
        Hide
        Karl Wright added a comment -

        Tried dropping numThreads in verify() to 1. Didn't help with reproducibility, unfortunately.

        Show
        Karl Wright added a comment - Tried dropping numThreads in verify() to 1. Didn't help with reproducibility, unfortunately.
        Hide
        Karl Wright added a comment -

        These turn out to be another series of "the point should be a hit but isn't" failures.

        I tried to reproduce it with the information I could print at the time of the actual failure, and checked whether the XYZBound that was computed was correct. It was. So I really do need repeatability here to make any progress.

        Show
        Karl Wright added a comment - These turn out to be another series of "the point should be a hit but isn't" failures. I tried to reproduce it with the information I could print at the time of the actual failure, and checked whether the XYZBound that was computed was correct. It was. So I really do need repeatability here to make any progress.
        Hide
        Michael McCandless added a comment -

        So I really do need repeatability here to make any progress.

        Argh, we are simply hitting this long ago known issue: LUCENE-6194.

        The workaround (I think?) is to simply add back the -Dtests.iters=10 that you had originally specified to ant beast, but you also must add a wildcard * at the end of the -Dtests.name=XXX. E.g. I just hit a failure with your ant beast line, confirmed the Reproduce with line is buggy (does not in fact reproduce) but then added back the iters and the * and it does reproduce:

        ant test  -Dtestcase=TestGeo3DPointField -Dtests.method=testGeo3DRelations* -Dtests.seed=B8B6B22C45AFEE9F -Dtests.locale=ms_MY -Dtests.timezone=America/Moncton -Dtests.asserts=true -Dtests.file.encoding=US-ASCII -Dtests.iters=10
        

        causes:

           [junit4] FAILURE 0.33s | TestGeo3DPointField.testGeo3DRelations {#7 seed=[B8B6B22C45AFEE9F:7CCEE6732E8A91DE]} <<<
           [junit4]    > Throwable #1: java.lang.AssertionError: invalid hits for shape=GeoPath: {planetmodel=PlanetModel(ab=1.151145876105594 c=0.8488541238944061), width=0.008726646259971648(0.5), points={[[lat=-0.6925658899376476, lon=0.6316613927914589], [lat=0.27828548161836364, lon=0.6785795524104564]]}}
           [junit4]    > 	at __randomizedtesting.SeedInfo.seed([B8B6B22C45AFEE9F:7CCEE6732E8A91DE]:0)
           [junit4]    > 	at org.apache.lucene.bkdtree3d.TestGeo3DPointField.testGeo3DRelations(TestGeo3DPointField.java:708)
           [junit4]    > 	at java.lang.Thread.run(Thread.java:745)
           [junit4] OK      1.64s | TestGeo3DPointField.testGeo3DRelations {#8 seed=[B8B6B22C45AFEE9F:4E62031D59418684]}
           [junit4] OK      0.98s | TestGeo3DPointField.testGeo3DRelations {#9 seed=[B8B6B22C45AFEE9F:99E955A73D16B1D6]}
           [junit4]   2> NOTE: test params are: codec=Asserting(Lucene53): {}, docValues:{}, sim=RandomSimilarityProvider(queryNorm=true,coord=crazy): {}, locale=ms_MY, timezone=America/Moncton
           [junit4]   2> NOTE: Linux 3.13.0-61-generic amd64/Oracle Corporation 1.8.0_40 (64-bit)/cpus=8,threads=1,free=455977424,total=471859200
           [junit4]   2> NOTE: All tests run in this JVM: [TestGeo3DPointField]
           [junit4] Completed [1/1] in 7.14s, 10 tests, 1 failure <<< FAILURES!
        
        Show
        Michael McCandless added a comment - So I really do need repeatability here to make any progress. Argh, we are simply hitting this long ago known issue: LUCENE-6194 . The workaround (I think?) is to simply add back the -Dtests.iters=10 that you had originally specified to ant beast , but you also must add a wildcard * at the end of the -Dtests.name=XXX . E.g. I just hit a failure with your ant beast line, confirmed the Reproduce with line is buggy (does not in fact reproduce) but then added back the iters and the * and it does reproduce: ant test -Dtestcase=TestGeo3DPointField -Dtests.method=testGeo3DRelations* -Dtests.seed=B8B6B22C45AFEE9F -Dtests.locale=ms_MY -Dtests.timezone=America/Moncton -Dtests.asserts=true -Dtests.file.encoding=US-ASCII -Dtests.iters=10 causes: [junit4] FAILURE 0.33s | TestGeo3DPointField.testGeo3DRelations {#7 seed=[B8B6B22C45AFEE9F:7CCEE6732E8A91DE]} <<< [junit4] > Throwable #1: java.lang.AssertionError: invalid hits for shape=GeoPath: {planetmodel=PlanetModel(ab=1.151145876105594 c=0.8488541238944061), width=0.008726646259971648(0.5), points={[[lat=-0.6925658899376476, lon=0.6316613927914589], [lat=0.27828548161836364, lon=0.6785795524104564]]}} [junit4] > at __randomizedtesting.SeedInfo.seed([B8B6B22C45AFEE9F:7CCEE6732E8A91DE]:0) [junit4] > at org.apache.lucene.bkdtree3d.TestGeo3DPointField.testGeo3DRelations(TestGeo3DPointField.java:708) [junit4] > at java.lang.Thread.run(Thread.java:745) [junit4] OK 1.64s | TestGeo3DPointField.testGeo3DRelations {#8 seed=[B8B6B22C45AFEE9F:4E62031D59418684]} [junit4] OK 0.98s | TestGeo3DPointField.testGeo3DRelations {#9 seed=[B8B6B22C45AFEE9F:99E955A73D16B1D6]} [junit4] 2> NOTE: test params are: codec=Asserting(Lucene53): {}, docValues:{}, sim=RandomSimilarityProvider(queryNorm=true,coord=crazy): {}, locale=ms_MY, timezone=America/Moncton [junit4] 2> NOTE: Linux 3.13.0-61-generic amd64/Oracle Corporation 1.8.0_40 (64-bit)/cpus=8,threads=1,free=455977424,total=471859200 [junit4] 2> NOTE: All tests run in this JVM: [TestGeo3DPointField] [junit4] Completed [1/1] in 7.14s, 10 tests, 1 failure <<< FAILURES!
        Hide
        Karl Wright added a comment -

        Thanks-- was able to construct a simple test case that demonstrated the problem. Now, back to debugging...

        Show
        Karl Wright added a comment - Thanks-- was able to construct a simple test case that demonstrated the problem. Now, back to debugging...
        Hide
        Karl Wright added a comment -

        Patch with fix for GeoPath as well. Now it beasts for a good long time without problems. Final acceptance will, of course, require Michael McCandless's light-dimming Big Beaster to wale at it for a day, but...

        Show
        Karl Wright added a comment - Patch with fix for GeoPath as well. Now it beasts for a good long time without problems. Final acceptance will, of course, require Michael McCandless 's light-dimming Big Beaster to wale at it for a day, but...
        Hide
        Michael McCandless added a comment -

        Thanks Karl Wright, I'll start beasting!

        Show
        Michael McCandless added a comment - Thanks Karl Wright , I'll start beasting!
        Hide
        Michael McCandless added a comment -

        I see the FUDGE_FACTOR became small again, but I don't see that we are randomizing the PlanetModel in TestGeo3DPointField in the latest patch (it was in the first patch).

        Should I fold that in from the first patch, for beasting?

        Show
        Michael McCandless added a comment - I see the FUDGE_FACTOR became small again, but I don't see that we are randomizing the PlanetModel in TestGeo3DPointField in the latest patch (it was in the first patch). Should I fold that in from the first patch, for beasting?
        Hide
        Michael McCandless added a comment -

        OK I just started from the last patch, and factored out a randomized getPlanetModel...

        Show
        Michael McCandless added a comment - OK I just started from the last patch, and factored out a randomized getPlanetModel ...
        Hide
        Karl Wright added a comment -

        Sorry! I meant to include the random planet model...

        FWIW, there's nothing special about SPHERE or WGS84. They're just predefined models with specific parameters.

        Show
        Karl Wright added a comment - Sorry! I meant to include the random planet model... FWIW, there's nothing special about SPHERE or WGS84. They're just predefined models with specific parameters.
        Hide
        Michael McCandless added a comment -

        FWIW, there's nothing special about SPHERE or WGS84. They're just predefined models with specific parameters.

        Thanks, that makes sense ... I just wanted to bias the randomness to pick those two specific models more often since users will actually use those ones (presumably) most frequently ...

        Show
        Michael McCandless added a comment - FWIW, there's nothing special about SPHERE or WGS84. They're just predefined models with specific parameters. Thanks, that makes sense ... I just wanted to bias the randomness to pick those two specific models more often since users will actually use those ones (presumably) most frequently ...
        Hide
        Michael McCandless added a comment -

        New patch, just fixing a verbosity issue in the test, and after beasting for quite a while with this patch I hit this failure:

           [junit4] Suite: org.apache.lucene.bkdtree3d.TestGeo3DPointField
           [junit4]   1> doc=4870 did not match but should
           [junit4]   1>   point=[lat=-0.00591253844632244, lon=-0.0020069187259065093]
           [junit4]   1>   quantized=[X=1.001099185736782, Y=-0.0020091272069679327, Z=-0.005919118245803968]
           [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestGeo3DPointField -Dtests.method=testGeo3DRelations -Dtests.seed=3B8A70E29A964D\
        99 -Dtests.multiplier=5 -Dtests.slow=true -Dtests.locale=ar_JO -Dtests.timezone=CST -Dtests.asserts=true -Dtests.file.encoding=UTF-8
           [junit4] FAILURE 1.67s | TestGeo3DPointField.testGeo3DRelations <<<
           [junit4]    > Throwable #1: java.lang.AssertionError: invalid hits for shape=GeoCircle: {planetmodel=PlanetModel.WGS84, center=[lat=-0.005\
        931145568901605, lon=-0.001942031539653079], radius=1.2991918568260272E-4(0.007443821017389608)}
           [junit4]    >        at __randomizedtesting.SeedInfo.seed([3B8A70E29A964D99:8BF50D7615DBE305]:0)
           [junit4]    >        at org.apache.lucene.bkdtree3d.TestGeo3DPointField.testGeo3DRelations(TestGeo3DPointField.java:708)
           [junit4]    >        at java.lang.Thread.run(Thread.java:745)
           [junit4]   2> NOTE: test params are: codec=CheapBastard, sim=RandomSimilarityProvider(queryNorm=true,coord=no): {}, locale=ar_JO, timezone\
        =CST
           [junit4]   2> NOTE: Linux 3.19.0-21-generic amd64/Oracle Corporation 1.8.0_51 (64-bit)/cpus=72,threads=1,free=404816768,total=514850816
           [junit4]   2> NOTE: All tests run in this JVM: [TestGeo3DPointField]
           [junit4] Completed [1/1] in 2.00s, 1 test, 1 failure <<< FAILURES!
        
        Show
        Michael McCandless added a comment - New patch, just fixing a verbosity issue in the test, and after beasting for quite a while with this patch I hit this failure: [junit4] Suite: org.apache.lucene.bkdtree3d.TestGeo3DPointField [junit4] 1> doc=4870 did not match but should [junit4] 1> point=[lat=-0.00591253844632244, lon=-0.0020069187259065093] [junit4] 1> quantized=[X=1.001099185736782, Y=-0.0020091272069679327, Z=-0.005919118245803968] [junit4] 2> NOTE: reproduce with: ant test -Dtestcase=TestGeo3DPointField -Dtests.method=testGeo3DRelations -Dtests.seed=3B8A70E29A964D\ 99 -Dtests.multiplier=5 -Dtests.slow=true -Dtests.locale=ar_JO -Dtests.timezone=CST -Dtests.asserts=true -Dtests.file.encoding=UTF-8 [junit4] FAILURE 1.67s | TestGeo3DPointField.testGeo3DRelations <<< [junit4] > Throwable #1: java.lang.AssertionError: invalid hits for shape=GeoCircle: {planetmodel=PlanetModel.WGS84, center=[lat=-0.005\ 931145568901605, lon=-0.001942031539653079], radius=1.2991918568260272E-4(0.007443821017389608)} [junit4] > at __randomizedtesting.SeedInfo.seed([3B8A70E29A964D99:8BF50D7615DBE305]:0) [junit4] > at org.apache.lucene.bkdtree3d.TestGeo3DPointField.testGeo3DRelations(TestGeo3DPointField.java:708) [junit4] > at java.lang.Thread.run(Thread.java:745) [junit4] 2> NOTE: test params are: codec=CheapBastard, sim=RandomSimilarityProvider(queryNorm=true,coord=no): {}, locale=ar_JO, timezone\ =CST [junit4] 2> NOTE: Linux 3.19.0-21-generic amd64/Oracle Corporation 1.8.0_51 (64-bit)/cpus=72,threads=1,free=404816768,total=514850816 [junit4] 2> NOTE: All tests run in this JVM: [TestGeo3DPointField] [junit4] Completed [1/1] in 2.00s, 1 test, 1 failure <<< FAILURES!
        Hide
        Karl Wright added a comment -

        Fix for beasting failure.

        Doubled MINIMUM_RESOLUTION back to its original value of 1e-12. Also disabled a test case I had inadvertantly included in the previous patch.
        Michael McCandless

        Show
        Karl Wright added a comment - Fix for beasting failure. Doubled MINIMUM_RESOLUTION back to its original value of 1e-12. Also disabled a test case I had inadvertantly included in the previous patch. Michael McCandless
        Hide
        Michael McCandless added a comment -

        Thanks Karl Wright, this patch lasted for 6 hours of heavy (36 JVMs) beasting, then hit this:

        [junit4:pickseed] Seed property 'tests.seed' already defined: C51A81D82CBA7AC0
           [junit4] <JUnit4> says hello! Master seed: C51A81D82CBA7AC0
           [junit4] Executing 1 suite with 1 JVM.
           [junit4] 
           [junit4] Started J0 PID(23324@localhost).
           [junit4] Suite: org.apache.lucene.bkdtree3d.TestGeo3DPointField
           [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestGeo3DPointField -Dtests.method=testGeo3DRelations -Dtests.seed=C51A81D82CBA7AC0 -Dtests.multiplier=5 -Dtests.slow=true -Dtests.linedocsfile=/lucenedata/hudson.enwiki.random.lines.txt.fixed -Dtests.locale=ar_OM -Dtests.timezone=Pacific/Midway -Dtests.asserts=true -Dtests.file.encoding=UTF-8
           [junit4] ERROR   1.17s | TestGeo3DPointField.testGeo3DRelations <<<
           [junit4]    > Throwable #1: java.lang.NullPointerException
           [junit4]    > 	at __randomizedtesting.SeedInfo.seed([C51A81D82CBA7AC0:7565FC4CA3F7D45C]:0)
           [junit4]    > 	at org.apache.lucene.geo3d.GeoCompositeMembershipShape.isWithin(GeoCompositeMembershipShape.java:47)
           [junit4]    > 	at org.apache.lucene.geo3d.BaseXYZSolid.isAreaInsideShape(BaseXYZSolid.java:131)
           [junit4]    > 	at org.apache.lucene.geo3d.XYZSolid.getRelationship(XYZSolid.java:312)
           [junit4]    > 	at org.apache.lucene.bkdtree3d.TestGeo3DPointField.testGeo3DRelations(TestGeo3DPointField.java:546)
           [junit4]    > 	at java.lang.Thread.run(Thread.java:745)
           [junit4]   2> NOTE: test params are: codec=Asserting(Lucene53): {}, docValues:{}, sim=RandomSimilarityProvider(queryNorm=false,coord=no): {}, locale=ar_OM, timezone=Pacific/Midway
           [junit4]   2> NOTE: Linux 3.19.0-21-generic amd64/Oracle Corporation 1.8.0_51 (64-bit)/cpus=72,threads=1,free=413503048,total=514850816
           [junit4]   2> NOTE: All tests run in this JVM: [TestGeo3DPointField]
           [junit4] Completed [1/1] in 1.48s, 1 test, 1 error <<< FAILURES!
        
        Show
        Michael McCandless added a comment - Thanks Karl Wright , this patch lasted for 6 hours of heavy (36 JVMs) beasting, then hit this: [junit4:pickseed] Seed property 'tests.seed' already defined: C51A81D82CBA7AC0 [junit4] <JUnit4> says hello! Master seed: C51A81D82CBA7AC0 [junit4] Executing 1 suite with 1 JVM. [junit4] [junit4] Started J0 PID(23324@localhost). [junit4] Suite: org.apache.lucene.bkdtree3d.TestGeo3DPointField [junit4] 2> NOTE: reproduce with: ant test -Dtestcase=TestGeo3DPointField -Dtests.method=testGeo3DRelations -Dtests.seed=C51A81D82CBA7AC0 -Dtests.multiplier=5 -Dtests.slow=true -Dtests.linedocsfile=/lucenedata/hudson.enwiki.random.lines.txt.fixed -Dtests.locale=ar_OM -Dtests.timezone=Pacific/Midway -Dtests.asserts=true -Dtests.file.encoding=UTF-8 [junit4] ERROR 1.17s | TestGeo3DPointField.testGeo3DRelations <<< [junit4] > Throwable #1: java.lang.NullPointerException [junit4] > at __randomizedtesting.SeedInfo.seed([C51A81D82CBA7AC0:7565FC4CA3F7D45C]:0) [junit4] > at org.apache.lucene.geo3d.GeoCompositeMembershipShape.isWithin(GeoCompositeMembershipShape.java:47) [junit4] > at org.apache.lucene.geo3d.BaseXYZSolid.isAreaInsideShape(BaseXYZSolid.java:131) [junit4] > at org.apache.lucene.geo3d.XYZSolid.getRelationship(XYZSolid.java:312) [junit4] > at org.apache.lucene.bkdtree3d.TestGeo3DPointField.testGeo3DRelations(TestGeo3DPointField.java:546) [junit4] > at java.lang.Thread.run(Thread.java:745) [junit4] 2> NOTE: test params are: codec=Asserting(Lucene53): {}, docValues:{}, sim=RandomSimilarityProvider(queryNorm=false,coord=no): {}, locale=ar_OM, timezone=Pacific/Midway [junit4] 2> NOTE: Linux 3.19.0-21-generic amd64/Oracle Corporation 1.8.0_51 (64-bit)/cpus=72,threads=1,free=413503048,total=514850816 [junit4] 2> NOTE: All tests run in this JVM: [TestGeo3DPointField] [junit4] Completed [1/1] in 1.48s, 1 test, 1 error <<< FAILURES!
        Hide
        Karl Wright added a comment -

        This is unrelated to the fixes done.
        Basically, some shape somewhere is computing an edge point and finding that it can't do it, but trying to use it anyway. Looks like it's a polygon. It should throw IllegalArgumentException in that case, but doesn't. I will code a fix and attach the whole patch to the ticket.

        Show
        Karl Wright added a comment - This is unrelated to the fixes done. Basically, some shape somewhere is computing an edge point and finding that it can't do it, but trying to use it anyway. Looks like it's a polygon. It should throw IllegalArgumentException in that case, but doesn't. I will code a fix and attach the whole patch to the ticket.
        Hide
        Karl Wright added a comment -

        Patch that fixes latest beasting failure.

        Show
        Karl Wright added a comment - Patch that fixes latest beasting failure.
        Hide
        Karl Wright added a comment -

        Found three other places in the XYZSolid family of shapes where the same problems occur.

        Show
        Karl Wright added a comment - Found three other places in the XYZSolid family of shapes where the same problems occur.
        Hide
        Michael McCandless added a comment -

        I've been beasting this last patch for ~6 hours ... no failures! I think it's a keeper ... I'll commit soon.

        Show
        Michael McCandless added a comment - I've been beasting this last patch for ~6 hours ... no failures! I think it's a keeper ... I'll commit soon.
        Hide
        ASF subversion and git services added a comment -

        Commit 1701835 from Michael McCandless in branch 'dev/trunk'
        [ https://svn.apache.org/r1701835 ]

        LUCENE-6776: Fix geo3d math to handle randomly squashed planets

        Show
        ASF subversion and git services added a comment - Commit 1701835 from Michael McCandless in branch 'dev/trunk' [ https://svn.apache.org/r1701835 ] LUCENE-6776 : Fix geo3d math to handle randomly squashed planets
        Hide
        ASF subversion and git services added a comment -

        Commit 1701837 from Michael McCandless in branch 'dev/branches/branch_5x'
        [ https://svn.apache.org/r1701837 ]

        LUCENE-6776: Fix geo3d math to handle randomly squashed planets

        Show
        ASF subversion and git services added a comment - Commit 1701837 from Michael McCandless in branch 'dev/branches/branch_5x' [ https://svn.apache.org/r1701837 ] LUCENE-6776 : Fix geo3d math to handle randomly squashed planets
        Hide
        Michael McCandless added a comment -

        Thanks Karl Wright!

        Show
        Michael McCandless added a comment - Thanks Karl Wright !

          People

          • Assignee:
            Unassigned
            Reporter:
            Karl Wright
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development