Solr
  1. Solr
  2. SOLR-3791

SortedMapBackedCache.java throws NullPointerException

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 3.6.1, 4.0-BETA
    • Fix Version/s: 3.6.2, 4.0
    • Labels:
      None
    • Environment:

      Debian

      Description

      As datasource is a mysql via jdbc configured in the DIH. There are some sql statements in entities, that delivers NULL values in some fields.
      When this entities do have processor="CachedSqlEntityProcessor", a NullPointerException will be thrown. I Tried it on different machines, with different copies of the index.

      e.g.:
      <entity processor="CachedSqlEntityProcessor"
      name="locator_ids"
      query="
      SELECT l1.id AS l1id, l2.id AS l2id, l3.id AS l3id, l4.id AS l4id, l5.id AS l5id, l6.id AS l6id, l7.id AS l7id FROM locators l1
      LEFT JOIN locators l2 ON l1.parent_id = l2.id

      .... and so on delivers a result like:

      -------------------------------

      l1id l2id l3id l4id l5id l6id l7id

      -------------------------------

      21843 855 223 66 12 1 NULL

      -------------------------------

      The SortedMapBackedCache throws the NullPointer.
      Staktrace:
      java.lang.NullPointerException
      at java.util.TreeMap.getEntry(TreeMap.java:341)
      at java.util.TreeMap.get(TreeMap.java:272)
      at org.apache.solr.handler.dataimport.SortedMapBackedCache.add(SortedMapBackedCache.java:57)
      at org.apache.solr.handler.dataimport.DIHCacheSupport.populateCache(DIHCacheSupport.java:124)
      at org.apache.solr.handler.dataimport.DIHCacheSupport.getSimpleCacheData(DIHCacheSupport.java:199)
      at org.apache.solr.handler.dataimport.DIHCacheSupport.getCacheData(DIHCacheSupport.java:147)
      at org.apache.solr.handler.dataimport.EntityProcessorBase.getNext(EntityProcessorBase.java:132)
      at org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:75)
      at org.apache.solr.handler.dataimport.EntityProcessorWrapper.pullRow(EntityProcessorWrapper.java:330)
      at org.apache.solr.handler.dataimport.EntityProcessorWrapper.nextRow(EntityProcessorWrapper.java:296)
      at org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:683)

      I had to give an onError="continue" to the entity, in order to get the proper stacktrace. Leaving out the onError property (default=Abort) suppresses the stacktrace in EntityProcessorWrapper.java line 332

      The DIH version 3.5.0 is not affected, works fine for me

      1. SOLR-3791.patch
        3 kB
        James Dyer
      2. SOLR-3791.patch
        0.8 kB
        James Dyer

        Activity

        Hide
        James Dyer added a comment -

        Here is a patch that prevents a NPE in the case you try to enter a row into the cache with a NULL primary key. The solution is to silently ignore the row and discard the data. Steffen, Does this fix your issue?

        A better patch would add logging to this class and log a warning when a row when a NULL key is added. Also, a unit test is needed.

        Show
        James Dyer added a comment - Here is a patch that prevents a NPE in the case you try to enter a row into the cache with a NULL primary key . The solution is to silently ignore the row and discard the data. Steffen, Does this fix your issue? A better patch would add logging to this class and log a warning when a row when a NULL key is added. Also, a unit test is needed.
        Hide
        Steffen Moelter added a comment -

        Hi James,
        your patch didn't work in this way. I have changed it to:

        (starting at line 57 in SortedMapBackedCache)

        //silently ignore if this row has a null key...
        if(pk==null)

        { return; }

        List<Map<String,Object>> thisKeysRecs = theMap.get(pk);
        if (thisKeysRecs == null) {

        this seems to work fine on my machines.
        I don't know how to generate a patch, so that I could attach it here.
        The solr 4.0.0-BETA seems to be affected as well, started the tests yesterday

        Thanks for your patch, that helped me a lot

        Show
        Steffen Moelter added a comment - Hi James, your patch didn't work in this way. I have changed it to: (starting at line 57 in SortedMapBackedCache) //silently ignore if this row has a null key... if(pk==null) { return; } List<Map<String,Object>> thisKeysRecs = theMap.get(pk); if (thisKeysRecs == null) { this seems to work fine on my machines. I don't know how to generate a patch, so that I could attach it here. The solr 4.0.0-BETA seems to be affected as well, started the tests yesterday Thanks for your patch, that helped me a lot
        Hide
        James Dyer added a comment -

        Here is a more-complete patch to allow SortedMapBackedCache handle null keys (ignoring them). A unit test is included.

        I will commit shortly as this needs to be fixed in 4.0

        Show
        James Dyer added a comment - Here is a more-complete patch to allow SortedMapBackedCache handle null keys (ignoring them). A unit test is included. I will commit shortly as this needs to be fixed in 4.0
        Hide
        James Dyer added a comment -

        Problem was introduced in 3.6 so fixing in the 3.6 branch also.

        Show
        James Dyer added a comment - Problem was introduced in 3.6 so fixing in the 3.6 branch also.
        Hide
        James Dyer added a comment -

        committed. Thank you, Steffen.

        Trunk: r1384819
        4x: r1384828
        3x: r1384834

        Show
        James Dyer added a comment - committed. Thank you, Steffen. Trunk: r1384819 4x: r1384828 3x: r1384834
        Hide
        Commit Tag Bot added a comment -

        [branch_4x commit] James Dyer
        http://svn.apache.org/viewvc?view=revision&revision=1384828

        SOLR-3779/SOLR-3791: fix for DIH LineEntityProcessor & CachedSqlEntityProdessor

        Show
        Commit Tag Bot added a comment - [branch_4x commit] James Dyer http://svn.apache.org/viewvc?view=revision&revision=1384828 SOLR-3779 / SOLR-3791 : fix for DIH LineEntityProcessor & CachedSqlEntityProdessor
        Hide
        Uwe Schindler added a comment -

        Closed after release.

        Show
        Uwe Schindler added a comment - Closed after release.

          People

          • Assignee:
            James Dyer
            Reporter:
            Steffen Moelter
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development