Lucene - Core
  1. Lucene - Core
  2. LUCENE-4739

FST cannot be loaded if it's larger than Integer.MAX_VALUE / 2 bytes

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 4.2, 6.0
    • Component/s: None
    • Labels:
      None
    • Lucene Fields:
      New

      Description

      This is really quite awful, but the test I created for > 2.1 GB FSTs never tested save/load ... and ... it doesn't work.

      1. LUCENE-4739.patch
        15 kB
        Michael McCandless
      2. LUCENE-4739.patch
        19 kB
        Michael McCandless

        Activity

        Hide
        Michael McCandless added a comment -

        Initial dirty patch ... I fixed Test2BFSTs to save/load but haven't run yet (it takes ~ 3 hours I think).

        I also modified NodeHash to use long[] because I think I'm hitting a PackedInts bug in GrowableWriter ... not sure yet.

        Show
        Michael McCandless added a comment - Initial dirty patch ... I fixed Test2BFSTs to save/load but haven't run yet (it takes ~ 3 hours I think). I also modified NodeHash to use long[] because I think I'm hitting a PackedInts bug in GrowableWriter ... not sure yet.
        Hide
        Commit Tag Bot added a comment -

        [trunk commit] Adrien Grand
        http://svn.apache.org/viewvc?view=revision&revision=1440909

        LUCENE-4739: Fix integer overflow in Packed64.get/set.

        Show
        Commit Tag Bot added a comment - [trunk commit] Adrien Grand http://svn.apache.org/viewvc?view=revision&revision=1440909 LUCENE-4739 : Fix integer overflow in Packed64.get/set.
        Hide
        Adrien Grand added a comment -

        Fixed. This was due to an int overflow in Packed64's bulk get and set methods. Thanks Mike!

        Show
        Adrien Grand added a comment - Fixed. This was due to an int overflow in Packed64's bulk get and set methods. Thanks Mike!
        Hide
        Commit Tag Bot added a comment -

        [branch_4x commit] Adrien Grand
        http://svn.apache.org/viewvc?view=revision&revision=1440949

        LUCENE-4739: Fix integer overflow in Packed64.get/set (merged from r1440909).

        Show
        Commit Tag Bot added a comment - [branch_4x commit] Adrien Grand http://svn.apache.org/viewvc?view=revision&revision=1440949 LUCENE-4739 : Fix integer overflow in Packed64.get/set (merged from r1440909).
        Hide
        Michael McCandless added a comment -

        Woops, not quite fixed ... there are FST ints that need to be fixed to long still ... I'm working on it.

        Show
        Michael McCandless added a comment - Woops, not quite fixed ... there are FST ints that need to be fixed to long still ... I'm working on it.
        Hide
        Michael McCandless added a comment -

        Patch, I think it's ready.

        Test2BFST passed ... (took 4 hours).

        Show
        Michael McCandless added a comment - Patch, I think it's ready. Test2BFST passed ... (took 4 hours).
        Hide
        Commit Tag Bot added a comment -

        [trunk commit] Michael McCandless
        http://svn.apache.org/viewvc?view=revision&revision=1441213

        LUCENE-4739: fix FST.save/load to work with > 1.1 GB FSTs

        Show
        Commit Tag Bot added a comment - [trunk commit] Michael McCandless http://svn.apache.org/viewvc?view=revision&revision=1441213 LUCENE-4739 : fix FST.save/load to work with > 1.1 GB FSTs
        Hide
        Robert Muir added a comment -

        Thanks guys!

        Show
        Robert Muir added a comment - Thanks guys!
        Hide
        Commit Tag Bot added a comment -

        [branch_4x commit] Michael McCandless
        http://svn.apache.org/viewvc?view=revision&revision=1441214

        LUCENE-4739: fix FST.save/load to work with > 1.1 GB FSTs

        Show
        Commit Tag Bot added a comment - [branch_4x commit] Michael McCandless http://svn.apache.org/viewvc?view=revision&revision=1441214 LUCENE-4739 : fix FST.save/load to work with > 1.1 GB FSTs
        Hide
        Uwe Schindler added a comment -

        Closed after release.

        Show
        Uwe Schindler added a comment - Closed after release.

          People

          • Assignee:
            Adrien Grand
            Reporter:
            Michael McCandless
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development