Solr
  1. Solr
  2. SOLR-1191

NullPointerException in delta import

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 1.3, 1.4
    • Fix Version/s: 3.1
    • Labels:
      None
    • Environment:

      OS: Windows & Linux.
      Java: 1.6
      DB: MySQL & SQL Server

      Description

      Seeing few of these NullPointerException during delta imports. Once this happens delta import stops working and keeps giving the same error.
      java.lang.NullPointerException
      at org.apache.solr.handler.dataimport.DocBuilder.collectDelta(DocBuilder.java:622)
      at org.apache.solr.handler.dataimport.DocBuilder.doDelta(DocBuilder.java:240)
      at org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:159)
      at org.apache.solr.handler.dataimport.DataImporter.doDeltaImport(DataImporter.java:337)
      at org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:376)
      at org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:355)

      Running delta import for a particular entity fixes the problem and delta import start working again.

      Here is the log just before & after the exception
      05/27 11:59:29 86987686 INFO btpool0-538 org.apache.solr.core.SolrCore - [localhost] webapp=/solr path=/dataimport params=

      {command=delta-import&optimize=false}

      status=0 QTime=0
      05/27 11:59:29 86987687 INFO Thread-4162 org.apache.solr.handler.dataimport.SolrWriter - Read dataimport.properties
      05/27 11:59:29 86987687 INFO Thread-4162 org.apache.solr.handler.dataimport.DataImporter - Starting Delta Import
      05/27 11:59:29 86987687 INFO Thread-4162 org.apache.solr.handler.dataimport.SolrWriter - Read dataimport.properties
      05/27 11:59:29 86987687 INFO Thread-4162 org.apache.solr.handler.dataimport.DocBuilder - Starting delta collection.
      05/27 11:59:29 86987690 INFO Thread-4162 org.apache.solr.handler.dataimport.DocBuilder - Running ModifiedRowKey() for Entity: content
      05/27 11:59:29 86987690 INFO Thread-4162 org.apache.solr.handler.dataimport.DocBuilder - Completed ModifiedRowKey for Entity: content rows obtained : 0
      05/27 11:59:29 86987690 INFO Thread-4162 org.apache.solr.handler.dataimport.DocBuilder - Completed DeletedRowKey for Entity: content rows obtained : 0
      05/27 11:59:29 86987692 INFO Thread-4162 org.apache.solr.handler.dataimport.DocBuilder - Completed parentDeltaQuery for Entity: content
      05/27 11:59:29 86987692 INFO Thread-4162 org.apache.solr.handler.dataimport.DocBuilder - Running ModifiedRowKey() for Entity: job
      05/27 11:59:29 86987692 INFO Thread-4162 org.apache.solr.handler.dataimport.JdbcDataSource - Creating a connection for entity job with URL: jdbc:sqlserver://localhost;databaseName=TestDB
      05/27 11:59:29 86987704 INFO Thread-4162 org.apache.solr.handler.dataimport.JdbcDataSource - Time taken for getConnection(): 12
      05/27 11:59:29 86987707 INFO Thread-4162 org.apache.solr.handler.dataimport.DocBuilder - Completed ModifiedRowKey for Entity: job rows obtained : 0
      05/27 11:59:29 86987707 INFO Thread-4162 org.apache.solr.handler.dataimport.DocBuilder - Completed DeletedRowKey for Entity: job rows obtained : 0
      05/27 11:59:29 86987707 INFO Thread-4162 org.apache.solr.handler.dataimport.DocBuilder - Completed parentDeltaQuery for Entity: job
      05/27 11:59:29 86987707 INFO Thread-4162 org.apache.solr.handler.dataimport.DocBuilder - Delta Import completed successfully
      05/27 11:59:29 86987707 INFO Thread-4162 org.apache.solr.handler.dataimport.DocBuilder - Starting delta collection.
      05/27 11:59:29 86987709 INFO Thread-4162 org.apache.solr.handler.dataimport.DocBuilder - Running ModifiedRowKey() for Entity: user
      05/27 11:59:29 86987709 INFO Thread-4162 org.apache.solr.handler.dataimport.JdbcDataSource - Creating a connection for entity user with URL: jdbc:sqlserver://localhost;databaseName=TestDB
      05/27 11:59:29 86987716 INFO Thread-4162 org.apache.solr.handler.dataimport.JdbcDataSource - Time taken for getConnection(): 7
      05/27 11:59:29 86987873 INFO Thread-4162 org.apache.solr.handler.dataimport.DocBuilder - Completed ModifiedRowKey for Entity: user rows obtained : 46
      05/27 11:59:29 86987873 INFO Thread-4162 org.apache.solr.handler.dataimport.DocBuilder - Completed DeletedRowKey for Entity: user rows obtained : 0
      05/27 11:59:29 86987873 INFO Thread-4162 org.apache.solr.handler.dataimport.DocBuilder - Completed parentDeltaQuery for Entity: user
      05/27 11:59:29 86987882 INFO Thread-4162 org.apache.solr.core.SolrCore - SolrDeletionPolicy.onInit: commits:num=1
      commit{dir=/usr/share/solr/work/localhost/data/index,segFN=segments_m0,version=1242658668035,generation=792,filenames=[wv.fdt, _vf.prx, _z2.frq, _z5.frq, _z2.tis, _wv.fdx, _z6.nrm, _z3.tis, segments
      m0, _uh_1.del, _uj.fnm, _yd.fnm, _yz.fdt, _wa.fnm, _ui.prx, _yz.tis, _z0.nrm, _z7.fdt, _vf_6.del, _cy.fnm, _z5.nrm, _yz.fdx, _cy.tii, _z6.fdx, _uh.prx, _z6.fdt, _z7.fdx, _ui.nrm, _uh.fnm, _yz.tii, _uu.fdt, _x
      g.fnm, _z3.fdt, _uu.fdx, _z3.fdx, _z3_1.del, _wa.tis, _uj.tis, _ui.frq, _wa.tii, _z0.frq, _wv.nrm, _cy.tis, _xr.prx, _z3.tii, _z4.prx, _z7.prx, _xr.fdt, _z4.fdt, _xr.fdx, _z4.fdx, _uh.nrm, _y2.frq, _y2_1.del,
      _uj.tii, _z2.tii, _z2.nrm, _yo.prx, _z5.fnm, _y2.prx, _uu.nrm, _z0_1.del, _cy.prx, _yd.frq, _yo.fnm, _cy.nrm, _z0.fdx, _xg.tis, _z0.fdt, _z2_1.del, _xg.tii, _cy.frq, _xg.frq, _yd.prx, _z7.tis, _z0.tis, _z7.t
      ii, _yd.nrm, _vf.fnm, _z6.frq, _z2.fdx, _ui.fnm, _z5.tis, _z6_1.del, _z5.tii, _z1.fdt, _z0.tii, _z5_1.del, _uu.tis, _uu.tii, _z1.prx, _z1.fdx, _z6.tii, _y2.fnm, _xr.frq, _z4.frq, _z2.fdt, _z6.tis, _vf.frq, _y
      o.frq, _uj.fdx, _wv.fnm, _cy.fdt, _z6.prx, _xr.tis, _z4.tis, _yz.fnm, _uu.fnm, _z2.prx, _yo.tii, _z0.prx, _yd_1.del, _z1.frq, _yo.tis, _z7.nrm, _z5.prx, _yd.fdx, _yz.nrm, _cy.fdx, _yd.fdt, _wa_6.del, _z3.prx,
      _uj_h.del, _ui_e.del, _z7.fnm, _z3.fnm, _uu.prx, _uh.tii, _z6.fnm, _xg.prx, _xg.fdx, _xg.nrm, _wv.prx, _xg.fdt, _uu_b.del, _xr.nrm, _z4.nrm, _uj.prx, _xr.fnm, _z4.fnm, _uj.nrm, _yz_2.del, _uh.fdx, _vf.fdt, _
      uh.fdt, _yo.nrm, _xr.tii, _z4.tii, _vf.fdx, _uh.tis, _uj.fdt, _uj.frq, _yo.fdx, _wa.fdt, _ui.fdt, _z1.tii, _xg_1.del, _wa.fdx, _ui.fdx, _yz.frq, _ui.tii, _z5.fdt, _z5.fdx, _wv.frq, _wa.nrm, _yd.tii, _yd.tis,
      _yo_2.del, _y2.tii, _uu.frq, _cy_jd.del, _uh.frq, _y2.tis, _z1_1.del, _wa.prx, _z1.fnm, _ui.tis, _z2.fnm, _wa.frq, _wv.tii, _z0.fnm, _y2.fdx, _y2.fdt, _z7.frq, _wv.tis, _z3.frq, _y2.nrm, _yz.prx, _xr_2.del, _
      z1.nrm, _vf.tis, _vf.tii, _vf.nrm, _z3.nrm, _z1.tis, _yo.fdt]
      05/27 11:59:29 86987882 INFO Thread-4162 org.apache.solr.core.SolrCore - last commit = 1242658668035
      05/27 11:59:29 86987944 INFO Thread-4162 org.apache.solr.handler.dataimport.DocBuilder - Delta Import completed successfully
      05/27 11:59:29 86987944 INFO Thread-4162 org.apache.solr.handler.dataimport.DocBuilder - Starting delta collection.
      05/27 11:59:29 86987946 INFO Thread-4162 org.apache.solr.handler.dataimport.DocBuilder - Running ModifiedRowKey() for Entity: candidateContent
      05/27 11:59:29 86987946 INFO Thread-4162 org.apache.solr.handler.dataimport.DocBuilder - Completed ModifiedRowKey for Entity: candidateContent rows obtained : 0
      05/27 11:59:29 86987946 INFO Thread-4162 org.apache.solr.handler.dataimport.DocBuilder - Completed DeletedRowKey for Entity: candidateContent rows obtained : 0
      05/27 11:59:29 86987948 INFO Thread-4162 org.apache.solr.handler.dataimport.DocBuilder - Completed parentDeltaQuery for Entity: candidateContent
      05/27 11:59:29 86987950 INFO Thread-4162 org.apache.solr.handler.dataimport.DocBuilder - Running ModifiedRowKey() for Entity: candidateMessage
      05/27 11:59:29 86987950 INFO Thread-4162 org.apache.solr.handler.dataimport.DocBuilder - Completed ModifiedRowKey for Entity: candidateMessage rows obtained : 0
      05/27 11:59:29 86987950 INFO Thread-4162 org.apache.solr.handler.dataimport.DocBuilder - Completed DeletedRowKey for Entity: candidateMessage rows obtained : 0
      05/27 11:59:29 86987950 INFO Thread-4162 org.apache.solr.handler.dataimport.DocBuilder - Completed parentDeltaQuery for Entity: candidateMessage
      05/27 11:59:29 86987952 INFO Thread-4162 org.apache.solr.handler.dataimport.DocBuilder - Running ModifiedRowKey() for Entity: candidateCustom
      05/27 11:59:29 86987953 INFO Thread-4162 org.apache.solr.handler.dataimport.DocBuilder - Completed ModifiedRowKey for Entity: candidateCustom rows obtained : 0
      05/27 11:59:29 86987953 INFO Thread-4162 org.apache.solr.handler.dataimport.DocBuilder - Completed DeletedRowKey for Entity: candidateCustom rows obtained : 0
      05/27 11:59:29 86987953 INFO Thread-4162 org.apache.solr.handler.dataimport.DocBuilder - Completed parentDeltaQuery for Entity: candidateCustom
      05/27 11:59:29 86987953 INFO Thread-4162 org.apache.solr.handler.dataimport.DocBuilder - Running ModifiedRowKey() for Entity: candidate
      05/27 11:59:29 86987953 INFO Thread-4162 org.apache.solr.handler.dataimport.JdbcDataSource - Creating a connection for entity candidate with URL: jdbc:sqlserver://localhost;databaseName=TestDB
      05/27 11:59:29 86987960 INFO Thread-4162 org.apache.solr.handler.dataimport.JdbcDataSource - Time taken for getConnection(): 7
      05/27 11:59:29 86988066 ERROR Thread-4162 org.apache.solr.handler.dataimport.DataImporter - Delta Import Failed
      java.lang.NullPointerException
      at org.apache.solr.handler.dataimport.DocBuilder.collectDelta(DocBuilder.java:622)
      at org.apache.solr.handler.dataimport.DocBuilder.doDelta(DocBuilder.java:240)
      at org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:159)
      at org.apache.solr.handler.dataimport.DataImporter.doDeltaImport(DataImporter.java:337)
      at org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:376)
      at org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:355)
      05/27 11:59:29 86988066 INFO Thread-4162 org.apache.solr.update.UpdateHandler - start rollback
      05/27 11:59:29 86988071 INFO Thread-4162 org.apache.solr.update.UpdateHandler - end_rollback

      1. SOLR-1191.patch
        10 kB
        Gunnlaugur Thor Briem
      2. SOLR-1191.patch
        3 kB
        Gunnlaugur Thor Briem

        Activity

        Hide
        Ali Syed added a comment -

        Problem was in the db-data-config.xml the pk="a.Id" and deltaQuery & deletedPkQuery were "SELECT a.Id ..." .

        Changed the delta queries as follows "SELECT a.Id as 'a.Id' ..."

        It might make sense to fail with illegal argument exception instead of a null pointer.

        Could change following code..
        //Check to see if this delete is in the current delta set
        for (Map<String, Object> modifiedRow : deltaSet) {
        if (modifiedRow.get(entity.pk).equals(row.get(entity.pk)))

        { deltaRemoveSet.add(modifiedRow); }
        }

        as follows...
        //Check to see if this delete is in the current delta set
        for (Map<String, Object> modifiedRow : deltaSet) {
        if (modifiedRow.get(entity.pk) == null) throw new IllegalArgumentException(entity.pk+" doesn't exist in row: " + modifiedRow);
        if (modifiedRow.get(entity.pk).equals(row.get(entity.pk))) { deltaRemoveSet.add(modifiedRow); }

        }

        Show
        Ali Syed added a comment - Problem was in the db-data-config.xml the pk="a.Id" and deltaQuery & deletedPkQuery were "SELECT a.Id ..." . Changed the delta queries as follows "SELECT a.Id as 'a.Id' ..." It might make sense to fail with illegal argument exception instead of a null pointer. Could change following code.. //Check to see if this delete is in the current delta set for (Map<String, Object> modifiedRow : deltaSet) { if (modifiedRow.get(entity.pk).equals(row.get(entity.pk))) { deltaRemoveSet.add(modifiedRow); } } as follows... //Check to see if this delete is in the current delta set for (Map<String, Object> modifiedRow : deltaSet) { if (modifiedRow.get(entity.pk) == null) throw new IllegalArgumentException(entity.pk+" doesn't exist in row: " + modifiedRow); if (modifiedRow.get(entity.pk).equals(row.get(entity.pk))) { deltaRemoveSet.add(modifiedRow); } }
        Hide
        Lance Norskog added a comment -

        This error should appear in the user-visible output as a "User Error".

        Please mark this "to be fixed in Solr 1.4".

        Show
        Lance Norskog added a comment - This error should appear in the user-visible output as a "User Error". Please mark this "to be fixed in Solr 1.4".
        Hide
        Noble Paul added a comment -

        The stacktrace does not refer to trunk. can you just let me know which version you were using? I fixed another NPE recently w/ delta-import. I am not too sure if it is the same

        Show
        Noble Paul added a comment - The stacktrace does not refer to trunk. can you just let me know which version you were using? I fixed another NPE recently w/ delta-import. I am not too sure if it is the same
        Hide
        Noble Paul added a comment -

        this should have been fixed in the trunk as a result of SOLR-1228 we can reopen this if it is reported again

        Show
        Noble Paul added a comment - this should have been fixed in the trunk as a result of SOLR-1228 we can reopen this if it is reported again
        Hide
        Grant Ingersoll added a comment -

        Bulk close for Solr 1.4

        Show
        Grant Ingersoll added a comment - Bulk close for Solr 1.4
        Hide
        Gunnlaugur Thor Briem added a comment -

        I see this still happening on the tip of the 3.1 branch:

        Jan 15, 2011 9:47:39 PM org.apache.solr.handler.dataimport.DataImporter doDeltaImport
        SEVERE: Delta Import Failed
        java.lang.NullPointerException
        at org.apache.solr.handler.dataimport.DocBuilder.collectDelta(DocBuilder.java:860)
        at org.apache.solr.handler.dataimport.DocBuilder.doDelta(DocBuilder.java:282)
        at org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:176)
        at org.apache.solr.handler.dataimport.DataImporter.doDeltaImport(DataImporter.java:356)
        at org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:395)
        at org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:374)

        Same kind of problem:

        pk="ds.id"
        deltaQuery="SELECT id FROM [...]

        and same kind of workaround:

        pk="ds.id"
        deltaQuery="SELECT id AS "ds.id"FROM [...]

        Show
        Gunnlaugur Thor Briem added a comment - I see this still happening on the tip of the 3.1 branch: Jan 15, 2011 9:47:39 PM org.apache.solr.handler.dataimport.DataImporter doDeltaImport SEVERE: Delta Import Failed java.lang.NullPointerException at org.apache.solr.handler.dataimport.DocBuilder.collectDelta(DocBuilder.java:860) at org.apache.solr.handler.dataimport.DocBuilder.doDelta(DocBuilder.java:282) at org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:176) at org.apache.solr.handler.dataimport.DataImporter.doDeltaImport(DataImporter.java:356) at org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:395) at org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:374) Same kind of problem: pk="ds.id" deltaQuery="SELECT id FROM [...] and same kind of workaround: pk="ds.id" deltaQuery="SELECT id AS "ds.id"FROM [...]
        Hide
        Gunnlaugur Thor Briem added a comment - - edited

        I added a patch against branch_3x to resolve this. It resolves deltaQuery columns against pk when they differ by prefix (and reports the error more helpfully when no column matches, or more than one column matches).

        No unit test, sorry (but there's not much deltaQuery coverage anyway). All existing unit tests pass, and this is working fine for me in production.

        Show
        Gunnlaugur Thor Briem added a comment - - edited I added a patch against branch_3x to resolve this. It resolves deltaQuery columns against pk when they differ by prefix (and reports the error more helpfully when no column matches, or more than one column matches). No unit test, sorry (but there's not much deltaQuery coverage anyway). All existing unit tests pass, and this is working fine for me in production.
        Hide
        Simon Willnauer added a comment -

        seems like this is not fixed yet

        Show
        Simon Willnauer added a comment - seems like this is not fixed yet
        Hide
        Koji Sekiguchi added a comment -

        No unit test, sorry (but there's not much deltaQuery coverage anyway).

        There seems to be TestSqlEntityProcessorDelta*.java, no?

        Show
        Koji Sekiguchi added a comment - No unit test, sorry (but there's not much deltaQuery coverage anyway). There seems to be TestSqlEntityProcessorDelta*.java, no?
        Hide
        Gunnlaugur Thor Briem added a comment -

        There seems to be TestSqlEntityProcessorDelta*.java, no?

        Indeed there are, and they do seem to cover delta imports to a fair degree. I must have been underslept. : ) The Hudson coverage report doesn't include the contrib stuff though.

        Show
        Gunnlaugur Thor Briem added a comment - There seems to be TestSqlEntityProcessorDelta*.java, no? Indeed there are, and they do seem to cover delta imports to a fair degree. I must have been underslept. : ) The Hudson coverage report doesn't include the contrib stuff though.
        Hide
        Yonik Seeley added a comment -

        If someone could whip up a test for this, we could get this fix into the upcoming 3.1 release.

        Show
        Yonik Seeley added a comment - If someone could whip up a test for this, we could get this fix into the upcoming 3.1 release.
        Hide
        Gunnlaugur Thor Briem added a comment -

        I'll make one later today or tomorrow.

        Show
        Gunnlaugur Thor Briem added a comment - I'll make one later today or tomorrow.
        Hide
        Gunnlaugur Thor Briem added a comment - - edited

        Updated patch with unit test, against current branch_3x.

        Show
        Gunnlaugur Thor Briem added a comment - - edited Updated patch with unit test, against current branch_3x.
        Hide
        Yonik Seeley added a comment -

        Thanks Gunnlaugur, I committed to trunk and 3x.

        Show
        Yonik Seeley added a comment - Thanks Gunnlaugur, I committed to trunk and 3x.
        Hide
        Grant Ingersoll added a comment -

        Bulk close for 3.1.0 release

        Show
        Grant Ingersoll added a comment - Bulk close for 3.1.0 release

          People

          • Assignee:
            Unassigned
            Reporter:
            Ali Syed
          • Votes:
            1 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development