Lucene - Core
  1. Lucene - Core
  2. LUCENE-5443

DocValuesProducer#ramBytesUsed throws ConcurrentModificationException

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 4.6, 4.6.1, 4.7, 6.0
    • Fix Version/s: 4.7, 6.0
    • Component/s: core/codecs
    • Labels:
      None
    • Lucene Fields:
      New, Patch Available

      Description

      this came up in an elasticsearch issue that if you pull #ramBytesUsed() while docvalues are loaded in a seperate thread you see a ConcurrentModificationException here is an example:

      Caused by: java.util.ConcurrentModificationException
              at java.util.HashMap$HashIterator.nextEntry(HashMap.java:926)
              at java.util.HashMap$ValueIterator.next(HashMap.java:954)
              at org.apache.lucene.codecs.lucene45.Lucene45DocValuesProducer.ramBytesUsed(Lucene45DocValuesProducer.java:291)
              at org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat$FieldsReader.ramBytesUsed(PerFieldDocValuesFormat.java:308)
              at org.apache.lucene.index.SegmentDocValues.ramBytesUsed(SegmentDocValues.java:103)
              at org.apache.lucene.index.SegmentReader.ramBytesUsed(SegmentReader.java:555)
      

        Activity

        Hide
        Shai Erera added a comment -

        I see that in Lucene45DVP, all access to addressInstances is guarded, so I think we should guard ramBytesUsed too. Likewise for ordIndexInstances. But when I look at Lucene4DVP, ramBytesUsed is an AtomicLong, and updated whenever a new DV is added ... can't we do the same? Also, looks like the computation of ramBytesUsed is slightly wrong, as it uses Integer.SIZE which is the number of bits in an int, not bytes.

        Show
        Shai Erera added a comment - I see that in Lucene45DVP, all access to addressInstances is guarded, so I think we should guard ramBytesUsed too. Likewise for ordIndexInstances. But when I look at Lucene4DVP, ramBytesUsed is an AtomicLong, and updated whenever a new DV is added ... can't we do the same? Also, looks like the computation of ramBytesUsed is slightly wrong, as it uses Integer.SIZE which is the number of bits in an int, not bytes.
        Hide
        Uwe Schindler added a comment -

        Also, looks like the computation of ramBytesUsed is slightly wrong, as it uses Integer.SIZE which is the number of bits in an int, not bytes.

        It should use RamUsageEstimator#NUM_BYTES_INT - this constant is 4

        Show
        Uwe Schindler added a comment - Also, looks like the computation of ramBytesUsed is slightly wrong, as it uses Integer.SIZE which is the number of bits in an int, not bytes. It should use RamUsageEstimator#NUM_BYTES_INT - this constant is 4
        Hide
        Shai Erera added a comment -

        Right. I can create a patch a bit later, unless someone beats me to it...

        Show
        Shai Erera added a comment - Right. I can create a patch a bit later, unless someone beats me to it...
        Hide
        Shai Erera added a comment -

        Add AtomicLong ramBytesUsed and compute it whenever a new buffer is added to the maps. I think it's ready!

        Show
        Shai Erera added a comment - Add AtomicLong ramBytesUsed and compute it whenever a new buffer is added to the maps. I think it's ready!
        Hide
        Adrien Grand added a comment -

        +1 to commit

        Show
        Adrien Grand added a comment - +1 to commit
        Hide
        ASF subversion and git services added a comment -

        Commit 1567954 from Shai Erera in branch 'dev/trunk'
        [ https://svn.apache.org/r1567954 ]

        LUCENE-5443: DocValuesProducer.ramBytesUsed throws ConcurrentModificationException

        Show
        ASF subversion and git services added a comment - Commit 1567954 from Shai Erera in branch 'dev/trunk' [ https://svn.apache.org/r1567954 ] LUCENE-5443 : DocValuesProducer.ramBytesUsed throws ConcurrentModificationException
        Hide
        ASF subversion and git services added a comment -

        Commit 1567960 from Shai Erera in branch 'dev/branches/branch_4x'
        [ https://svn.apache.org/r1567960 ]

        LUCENE-5443: DocValuesProducer.ramBytesUsed throws ConcurrentModificationException

        Show
        ASF subversion and git services added a comment - Commit 1567960 from Shai Erera in branch 'dev/branches/branch_4x' [ https://svn.apache.org/r1567960 ] LUCENE-5443 : DocValuesProducer.ramBytesUsed throws ConcurrentModificationException
        Hide
        Shai Erera added a comment -

        Committed to trunk and 4x.

        Show
        Shai Erera added a comment - Committed to trunk and 4x.

          People

          • Assignee:
            Shai Erera
            Reporter:
            Simon Willnauer
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development