Uploaded image for project: 'Solr'
  1. Solr
  2. SOLR-4016

Deduplication is broken by partial update

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 4.0
    • Fix Version/s: 4.1, 6.0
    • Component/s: update
    • Labels:
    • Environment:

      Tomcat6 / Catalina on Ubuntu 12.04 LTS

      Description

      The SignatureUpdateProcessorFactory used (primarily?) for deduplication does not consider partial update semantics.

      The below uses the following solrconfig.xml excerpt:

           <updateRequestProcessorChain name="text_hash">
             <processor class="solr.processor.SignatureUpdateProcessorFactory">
               <bool name="enabled">true</bool>
               <str name="signatureField">text_hash</str>
               <bool name="overwriteDupes">false</bool>
               <str name="fields">text</str>
               <str name="signatureClass">solr.processor.TextProfileSignature</str>
             </processor>
             <processor class="solr.LogUpdateProcessorFactory" />
             <processor class="solr.RunUpdateProcessorFactory" />
           </updateRequestProcessorChain>
      

      Firstly, the processor treats

      {"set": "value"}

      as a string and hashes it, instead of the value alone:

      $ curl '$URL/update?commit=true' -H 'Content-type:application/json' -d '{"add":{"doc":{"id": "abcde", "text": {"set": "hello world"}}}}' && curl '$URL/select?q=id:abcde'
      {"responseHeader":{"status":0,"QTime":30}}
      <?xml version="1.0" encoding="UTF-8"?><response><lst name="responseHeader"><int name="status">0</int><int name="QTime">1</int><lst name="params"><str name="q">id:abcde</str></lst></lst><result name="response" numFound="1" start="0"><doc><str name="id">abcde</str><str name="text">hello world</str><str name="text_hash">ad48c7ad60ac22cc</str><long name="_version_">1417247434224959488</long></doc></result>
      </response>
      $
      $ curl '$URL/update?commit=true' -H 'Content-type:application/json' -d '{"add":{"doc":{"id": "abcde", "text": "hello world"}}}' && curl '$URL/select?q=id:abcde'
      {"responseHeader":{"status":0,"QTime":27}}
      <?xml version="1.0" encoding="UTF-8"?>
      <response>
      <lst name="responseHeader"><int name="status">0</int><int name="QTime">1</int><lst name="params"><str name="q">id:abcde</str></lst></lst><result name="response" numFound="1" start="0"><doc><str name="id">abcde</str><str name="text">hello world</str><str name="text_hash">b169c743d220da8d</str><long name="_version_">1417248022215000064</long></doc></result>
      </response>
      

      Note the different text_hash value.

      Secondly, when updating a field other than those used to create the signature (which I imagine is a more common use-case), the signature is recalculated from no values:

      $ curl '$URL/update?commit=true' -H 'Content-type:application/json' -d '{"add":{"doc":{"id": "abcde", "title": {"set": "new title"}}}}' && curl '$URL/select?q=id:abcde'
      {"responseHeader":{"status":0,"QTime":39}}
      <?xml version="1.0" encoding="UTF-8"?>
      <response>
      <lst name="responseHeader"><int name="status">0</int><int name="QTime">1</int><lst name="params"><str name="q">id:abcde</str></lst></lst><result name="response" numFound="1" start="0"><doc><str name="id">abcde</str><str name="text">hello world</str><str name="text_hash">0000000000000000</str><str name="title">new title</str><long name="_version_">1417248120480202752</long></doc></result>
      </response>
      

        Attachments

        1. SOLR-4016-disallow-partial-update.patch
          8 kB
          Shalin Shekhar Mangar
        2. SOLR-4016-disallow-partial-update.patch
          3 kB
          Shalin Shekhar Mangar
        3. SOLR-4016.patch
          11 kB
          Shalin Shekhar Mangar

          Issue Links

            Activity

              People

              • Assignee:
                shalinmangar Shalin Shekhar Mangar
                Reporter:
                jnothman Joel Nothman
              • Votes:
                1 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: