Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 4.9, 6.0
    • Component/s: None
    • Labels:
      None
    • Lucene Fields:
      New

      Description

      I see users using this, i think they are unaware of the horrible tradeoffs it makes.

      We don't e.g. have codecs that have the term dictionary entirely on disk or other stupid things in lucene, so we shouldnt be stupid here either.

        Activity

        Hide
        Michael McCandless added a comment -

        +1

        Show
        Michael McCandless added a comment - +1
        Hide
        Adrien Grand added a comment -

        +1

        Show
        Adrien Grand added a comment - +1
        Hide
        ASF subversion and git services added a comment -

        Commit 1602862 from Robert Muir in branch 'dev/trunk'
        [ https://svn.apache.org/r1602862 ]

        LUCENE-5761: Remove DiskDocValuesFormat

        Show
        ASF subversion and git services added a comment - Commit 1602862 from Robert Muir in branch 'dev/trunk' [ https://svn.apache.org/r1602862 ] LUCENE-5761 : Remove DiskDocValuesFormat
        Hide
        ASF subversion and git services added a comment -

        Commit 1602864 from Robert Muir in branch 'dev/branches/branch_4x'
        [ https://svn.apache.org/r1602864 ]

        LUCENE-5761: Remove DiskDocValuesFormat

        Show
        ASF subversion and git services added a comment - Commit 1602864 from Robert Muir in branch 'dev/branches/branch_4x' [ https://svn.apache.org/r1602864 ] LUCENE-5761 : Remove DiskDocValuesFormat
        Hide
        ASF subversion and git services added a comment -

        Commit 1603227 from hossman@apache.org in branch 'dev/trunk'
        [ https://svn.apache.org/r1603227 ]

        LUCENE-5761: upgrade note for solr

        Show
        ASF subversion and git services added a comment - Commit 1603227 from hossman@apache.org in branch 'dev/trunk' [ https://svn.apache.org/r1603227 ] LUCENE-5761 : upgrade note for solr
        Hide
        ASF subversion and git services added a comment -

        Commit 1603228 from hossman@apache.org in branch 'dev/branches/branch_4x'
        [ https://svn.apache.org/r1603228 ]

        LUCENE-5761: upgrade note for solr (merge r1603227)

        Show
        ASF subversion and git services added a comment - Commit 1603228 from hossman@apache.org in branch 'dev/branches/branch_4x' [ https://svn.apache.org/r1603228 ] LUCENE-5761 : upgrade note for solr (merge r1603227)
        Hide
        ASF subversion and git services added a comment -

        Commit 1603229 from hossman@apache.org in branch 'dev/branches/lucene_solr_4_9'
        [ https://svn.apache.org/r1603229 ]

        LUCENE-5761: upgrade note for solr (merge r1603227)

        Show
        ASF subversion and git services added a comment - Commit 1603229 from hossman@apache.org in branch 'dev/branches/lucene_solr_4_9' [ https://svn.apache.org/r1603229 ] LUCENE-5761 : upgrade note for solr (merge r1603227)
        Hide
        Per Steffensen added a comment - - edited

        What is the alternative for those of us that have disk-doc-values in order to be able to sort/facet on certain fields?

        If we do not doc-value those fields the FieldCache will be used when sorting/faceting, which will make us go OOM - we have so much data that we will never be able to buy enough RAM to be able to hold all values of certain field(s) in memory (what the FieldCache does AFAIK). We have seen disk-doc-value as our only option, it actually works, and we can live with it being "slow"

        Show
        Per Steffensen added a comment - - edited What is the alternative for those of us that have disk-doc-values in order to be able to sort/facet on certain fields? If we do not doc-value those fields the FieldCache will be used when sorting/faceting, which will make us go OOM - we have so much data that we will never be able to buy enough RAM to be able to hold all values of certain field(s) in memory (what the FieldCache does AFAIK). We have seen disk-doc-value as our only option, it actually works, and we can live with it being "slow"
        Hide
        Robert Muir added a comment -

        Just use the default, officially supported docvalues format with the release, rather than a confusing, slow, experimental one named "Disk".

        Thats the whole point of this issue. There is no need for this experiment to stay around anymore: we tuned everything about the default codec to keep all the datastructures on disk but still be fast.

        Show
        Robert Muir added a comment - Just use the default, officially supported docvalues format with the release, rather than a confusing, slow, experimental one named "Disk". Thats the whole point of this issue. There is no need for this experiment to stay around anymore: we tuned everything about the default codec to keep all the datastructures on disk but still be fast.
        Hide
        Per Steffensen added a comment -

        Ok, thanks, we will try that when upgrading to a version that does not support "Disk" doc-values anymore.

        I just imagined that the non-"Disk" doc-values required all the data in memory while working with it (just as FieldCache does). E.g. that it memory mapped the entire doc-values file. But I really do not know anything about how it works. We will try the non-"Disk" doc-values when we get that far.

        Thanks again!

        Show
        Per Steffensen added a comment - Ok, thanks, we will try that when upgrading to a version that does not support "Disk" doc-values anymore. I just imagined that the non-"Disk" doc-values required all the data in memory while working with it (just as FieldCache does). E.g. that it memory mapped the entire doc-values file. But I really do not know anything about how it works. We will try the non-"Disk" doc-values when we get that far. Thanks again!

          People

          • Assignee:
            Unassigned
            Reporter:
            Robert Muir
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development