Uploaded image for project: 'Solr'
  1. Solr
  2. SOLR-8220

Read field from docValues for non stored fields

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 5.5, 6.0
    • Component/s: None
    • Labels:
      None

      Description

      useDocValuesAsStored

      Many times a value will be both stored="true" and docValues="true" which requires redundant data to be stored on disk. Since reading from docValues is both efficient and a common practice (facets, analytics, streaming, etc), reading values from docValues when a stored version of the field does not exist would be a valuable disk usage optimization.

      The only caveat with this that I can see would be for multiValued fields as they would always be returned sorted in the docValues approach. I believe this is a fair compromise.

      I've done a rough implementation for this as a field transform, but I think it should live closer to where stored fields are loaded in the SolrIndexSearcher.

      Two open questions/observations:

      1) There doesn't seem to be a standard way to read values for docValues, facets, analytics, streaming, etc, all seem to be doing their own ways, perhaps some of this logic should be centralized.

      2) What will the API behavior be? (Below is my proposed implementation)
      Parameters for fl:

      • fl="docValueField"
        • return field from docValue if the field is not stored and in docValues, if the field is stored return it from stored fields
      • fl="*"
        • return only stored fields
      • fl="+"
        • return stored fields and docValue fields

      2a - would be easiest implementation and might be sufficient for a first pass. 2b - is current behavior

        Attachments

        1. SOLR-8220.patch
          110 kB
          Shalin Shekhar Mangar
        2. SOLR-8220.patch
          111 kB
          Shalin Shekhar Mangar
        3. SOLR-8220.patch
          104 kB
          Ishan Chattopadhyaya
        4. SOLR-8220.patch
          104 kB
          Ishan Chattopadhyaya
        5. SOLR-8220.patch
          107 kB
          Shalin Shekhar Mangar
        6. SOLR-8220.patch
          100 kB
          Ishan Chattopadhyaya
        7. SOLR-8220.patch
          97 kB
          Shalin Shekhar Mangar
        8. SOLR-8220.patch
          97 kB
          Shalin Shekhar Mangar
        9. SOLR-8220.patch
          97 kB
          Shalin Shekhar Mangar
        10. SOLR-8220.patch
          82 kB
          Ishan Chattopadhyaya
        11. SOLR-8220.patch
          81 kB
          Ishan Chattopadhyaya
        12. SOLR-8220.patch
          82 kB
          Ishan Chattopadhyaya
        13. SOLR-8220.patch
          75 kB
          Ishan Chattopadhyaya
        14. SOLR-8220.patch
          73 kB
          Ishan Chattopadhyaya
        15. SOLR-8220.patch
          23 kB
          Ishan Chattopadhyaya
        16. SOLR-8220.patch
          23 kB
          Ishan Chattopadhyaya
        17. SOLR-8220.patch
          23 kB
          Keith Laban
        18. SOLR-8220.patch
          23 kB
          Keith Laban
        19. SOLR-8220.patch
          23 kB
          Keith Laban
        20. SOLR-8220.patch
          18 kB
          Ishan Chattopadhyaya
        21. SOLR-8220.patch
          18 kB
          Ishan Chattopadhyaya
        22. SOLR-8220.patch
          18 kB
          Ishan Chattopadhyaya
        23. SOLR-8220.patch
          18 kB
          Keith Laban
        24. SOLR-8220.patch
          20 kB
          Ishan Chattopadhyaya
        25. SOLR-8220.patch
          21 kB
          Keith Laban
        26. SOLR-8220.patch
          21 kB
          Keith Laban
        27. SOLR-8220.patch
          8 kB
          Ishan Chattopadhyaya
        28. SOLR-8220.patch
          9 kB
          Keith Laban
        29. SOLR-8220-5x.patch
          70 kB
          Ishan Chattopadhyaya
        30. SOLR-8220-branch_5x.patch
          110 kB
          Shalin Shekhar Mangar
        31. SOLR-8220-ishan.patch
          18 kB
          Ishan Chattopadhyaya
        32. SOLR-8220-ishan.patch
          18 kB
          Ishan Chattopadhyaya
        33. SOLR-8220-ishan.patch
          17 kB
          Ishan Chattopadhyaya
        34. SOLR-8220-ishan.patch
          16 kB
          Ishan Chattopadhyaya

          Issue Links

            Activity

              People

              • Assignee:
                shalinmangar Shalin Shekhar Mangar
                Reporter:
                k317h Keith Laban
              • Votes:
                2 Vote for this issue
                Watchers:
                12 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: