Lucene - Core
  1. Lucene - Core
  2. LUCENE-2423

NullPointerException when attemping to add a new document to an instance of IndexWriter

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Invalid
    • Affects Version/s: 3.0.1
    • Fix Version/s: 3.0.1
    • Component/s: core/index
    • Labels:
      None
    • Environment:

      Windows Vista Home Premium, Lucene version 3.0.1, JRE6, org.apache.commons.digester.Digester, org.apache.lucene.index.IndexWriter

    • Lucene Fields:
      New

      Description

      I'm attempting to run Listing #4 from "Parsing, Indexing, and Searching XML with Digester and Lucene" <https://www.ibm.com/developerworks/library/j-lucene/> . Using this code (with minor modifications), I am able to read and parse an XML input file and create a document to be indexed. However, attempting to index the document with an instance of IndexWriter (using the statement: writer.addDocument(contactDocument) causes a NullPointerException with the following messages:

      Apr 29, 2010 2:57:32 PM org.apache.commons.digester.Digester endElement
      SEVERE: End event threw exception
      java.lang.reflect.InvocationTargetException
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
      at java.lang.reflect.Method.invoke(Unknown Source)
      at org.apache.commons.beanutils.MethodUtils.invokeMethod(MethodUtils.jav
      a:282)
      at org.apache.commons.digester.SetNextRule.end(SetNextRule.java:217)
      at org.apache.commons.digester.Rule.end(Rule.java:253)
      at org.apache.commons.digester.Digester.endElement(Digester.java:1332)
      at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.endEleme
      nt(Unknown Source)
      at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImp
      l.scanEndElement(Unknown Source)
      at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImp
      l$FragmentContentDriver.next(Unknown Source)
      at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(U
      nknown Source)
      at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImp
      l.scanDocument(Unknown Source)
      at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(U
      nknown Source)
      at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(U
      nknown Source)
      at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(Unknown So
      urce)
      at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(Un
      known Source)
      at com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.p
      arse(Unknown Source)
      at org.apache.commons.digester.Digester.parse(Digester.java:1842)
      at DigesterMarriesLucene.main(DigesterMarriesLucene.java:94)
      Caused by: java.lang.NullPointerException
      at DigesterMarriesLucene.addContact(DigesterMarriesLucene.java:46)
      ... 20 more
      Exception in thread "main" java.lang.NullPointerException
      at org.apache.commons.digester.Digester.createSAXException(Digester.java
      :3333)
      at org.apache.commons.digester.Digester.createSAXException(Digester.java
      :3359)
      at org.apache.commons.digester.Digester.endElement(Digester.java:1335)
      at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.endEleme
      nt(Unknown Source)
      at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImp
      l.scanEndElement(Unknown Source)
      at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImp
      l$FragmentContentDriver.next(Unknown Source)
      at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(U
      nknown Source)
      at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImp
      l.scanDocument(Unknown Source)
      at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(U
      nknown Source)
      at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(U
      nknown Source)
      at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(Unknown So
      urce)
      at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(Un
      known Source)
      at com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.p
      arse(Unknown Source)
      at org.apache.commons.digester.Digester.parse(Digester.java:1842)
      at DigesterMarriesLucene.main(DigesterMarriesLucene.java:94)
      Caused by: java.lang.NullPointerException
      at DigesterMarriesLucene.addContact(DigesterMarriesLucene.java:46)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
      at java.lang.reflect.Method.invoke(Unknown Source)
      at org.apache.commons.beanutils.MethodUtils.invokeMethod(MethodUtils.jav
      a:282)
      at org.apache.commons.digester.SetNextRule.end(SetNextRule.java:217)
      at org.apache.commons.digester.Rule.end(Rule.java:253)
      at org.apache.commons.digester.Digester.endElement(Digester.java:1332)
      ... 12 more

      1. address-book.xml
        0.7 kB
        Bill Herbert
      2. DigesterMarriesLucene.java
        8 kB
        Bill Herbert

        Activity

        Bill Herbert created issue -
        Hide
        Shai Erera added a comment -

        I don't see any Lucene related classes in the stacktrace (org.apache.lucene.*). This does not look like a Lucene problem. What makes you think it's related to Lucene?

        Show
        Shai Erera added a comment - I don't see any Lucene related classes in the stacktrace (org.apache.lucene.*). This does not look like a Lucene problem. What makes you think it's related to Lucene?
        Hide
        Bill Herbert added a comment -

        Since I was able to read and parse the XML input file and create a document to be input to IndexWriter, and since the NullPointerException was not generated until IndexWriter was called (with the new document as the input parameter), it appeared that IndexWriter (which is part of Lucene) was the cause. I am a novice at interpreting the stacktrace and can't comment on why no Lucene rrelated classes are mentioned. However, there is obviously a problem here and I will greatly appreciate any suggestions as to a fix.

        Thanks, Bill

        Show
        Bill Herbert added a comment - Since I was able to read and parse the XML input file and create a document to be input to IndexWriter, and since the NullPointerException was not generated until IndexWriter was called (with the new document as the input parameter), it appeared that IndexWriter (which is part of Lucene) was the cause. I am a novice at interpreting the stacktrace and can't comment on why no Lucene rrelated classes are mentioned. However, there is obviously a problem here and I will greatly appreciate any suggestions as to a fix. Thanks, Bill
        Hide
        Shai Erera added a comment -

        The fact that there is no Lucene related code in the stacktrace means there was no Lucene code invoked. From what I gather, the exception is thrown here:
        Caused by: java.lang.NullPointerException
        at DigesterMarriesLucene.addContact(DigesterMarriesLucene.java:46)
        ... 20 more

        Can you paste line 46 of DigesterMarriesLucene? Few lines above and below would help as well.

        Show
        Shai Erera added a comment - The fact that there is no Lucene related code in the stacktrace means there was no Lucene code invoked. From what I gather, the exception is thrown here: Caused by: java.lang.NullPointerException at DigesterMarriesLucene.addContact(DigesterMarriesLucene.java:46) ... 20 more Can you paste line 46 of DigesterMarriesLucene? Few lines above and below would help as well.
        Hide
        Bill Herbert added a comment -

        The last statement in addConntact(), below, is line #46

        public void addContact(Contact contact) throws IOException

        { System.out.println("Adding " + contact.getName()); Document contactDocument = new Document(); contactDocument.add(new Field("type", contact.getType(), Field.Store.YES, Field.Index.ANALYZED)); contactDocument.add(new Field("name", contact.getName(), Field.Store.YES, Field.Index.ANALYZED)); contactDocument.add(new Field("address", contact.getAddress(), Field.Store.YES, Field.Index.ANALYZED)); contactDocument.add(new Field("city", contact.getCity(), Field.Store.YES, Field.Index.ANALYZED)); contactDocument.add(new Field("province", contact.getProvince(), Field.Store.YES, Field.Index.ANALYZED)); contactDocument.add(new Field("postalcode", contact.getPostalcode(), Field.Store.YES, Field.Index.ANALYZED)); contactDocument.add(new Field("country", contact.getCountry(), Field.Store.YES, Field.Index.ANALYZED)); contactDocument.add(new Field("telephone", contact.getTelephone(), Field.Store.YES, Field.Index.ANALYZED)); System.out.println("Added Telephone to contactDocument: " + contact.getTelephone()); System.out.println("Printed Telephone from contactDocument: " + contactDocument.getFields("telephone")); System.out.println("Printed entire contactDocument: " + contactDocument.getFields()); writer.addDocument(contactDocument); //line #46 }

        If it is any help, here is the code surrounding line #94:

        // set different properties of Contact instance using specified methods
        digester.addCallMethod("address-book/contact/name", "setName", 0);
        digester.addCallMethod("address-book/contact/address", "setAddress", 0);
        digester.addCallMethod("address-book/contact/city", "setCity", 0);
        digester.addCallMethod("address-book/contact/province", "setProvince", 0);
        digester.addCallMethod("address-book/contact/postalcode", "setPostalcode", 0);
        digester.addCallMethod("address-book/contact/country", "setCountry", 0);
        digester.addCallMethod("address-book/contact/telephone", "setTelephone", 0);

        // call 'addContact' method when the next 'address-book/contact' pattern is seen
        digester.addSetNext("address-book/contact", "addContact" );

        // now that rules and actions are configured, start the parsing process
        DigesterMarriesLucene dml = (DigesterMarriesLucene) digester.parse(new File(args[0])); // line #94

        // optimize and close the index
        writer.optimize();
        writer.close();
        } // end of public static void main

        Thanks again, Bill

        Show
        Bill Herbert added a comment - The last statement in addConntact(), below, is line #46 public void addContact(Contact contact) throws IOException { System.out.println("Adding " + contact.getName()); Document contactDocument = new Document(); contactDocument.add(new Field("type", contact.getType(), Field.Store.YES, Field.Index.ANALYZED)); contactDocument.add(new Field("name", contact.getName(), Field.Store.YES, Field.Index.ANALYZED)); contactDocument.add(new Field("address", contact.getAddress(), Field.Store.YES, Field.Index.ANALYZED)); contactDocument.add(new Field("city", contact.getCity(), Field.Store.YES, Field.Index.ANALYZED)); contactDocument.add(new Field("province", contact.getProvince(), Field.Store.YES, Field.Index.ANALYZED)); contactDocument.add(new Field("postalcode", contact.getPostalcode(), Field.Store.YES, Field.Index.ANALYZED)); contactDocument.add(new Field("country", contact.getCountry(), Field.Store.YES, Field.Index.ANALYZED)); contactDocument.add(new Field("telephone", contact.getTelephone(), Field.Store.YES, Field.Index.ANALYZED)); System.out.println("Added Telephone to contactDocument: " + contact.getTelephone()); System.out.println("Printed Telephone from contactDocument: " + contactDocument.getFields("telephone")); System.out.println("Printed entire contactDocument: " + contactDocument.getFields()); writer.addDocument(contactDocument); //line #46 } If it is any help, here is the code surrounding line #94: // set different properties of Contact instance using specified methods digester.addCallMethod("address-book/contact/name", "setName", 0); digester.addCallMethod("address-book/contact/address", "setAddress", 0); digester.addCallMethod("address-book/contact/city", "setCity", 0); digester.addCallMethod("address-book/contact/province", "setProvince", 0); digester.addCallMethod("address-book/contact/postalcode", "setPostalcode", 0); digester.addCallMethod("address-book/contact/country", "setCountry", 0); digester.addCallMethod("address-book/contact/telephone", "setTelephone", 0); // call 'addContact' method when the next 'address-book/contact' pattern is seen digester.addSetNext("address-book/contact", "addContact" ); // now that rules and actions are configured, start the parsing process DigesterMarriesLucene dml = (DigesterMarriesLucene) digester.parse(new File(args [0] )); // line #94 // optimize and close the index writer.optimize(); writer.close(); } // end of public static void main Thanks again, Bill
        Hide
        Jeremy Volkman added a comment -

        If that is truly line 46, it seems that the only possible cause would be that writer == null. Can you verify that you've actually initialized the writer field with an IndexWriter instance?

        Show
        Jeremy Volkman added a comment - If that is truly line 46, it seems that the only possible cause would be that writer == null. Can you verify that you've actually initialized the writer field with an IndexWriter instance?
        Hide
        Shai Erera added a comment -

        Yup, that seems the only possible cause for the NPE. If any NPE was thrown from Lucene, you'd see an IndexWriter line in the stacktrace.

        Show
        Shai Erera added a comment - Yup, that seems the only possible cause for the NPE. If any NPE was thrown from Lucene, you'd see an IndexWriter line in the stacktrace.
        Hide
        Bill Herbert added a comment -

        Here's the code I used to create IndexWriter instance:

        // IndexWriter to use for adding contacts to the index
        final File INDEX_DIR = new File(indexDir);
        IndexWriter writer = new IndexWriter(FSDirectory.open(INDEX_DIR), analyzer, createFlag, IndexWriter.MaxFieldLength.LIMITED);

        But I'm not sure how to verify the initialization.

        Show
        Bill Herbert added a comment - Here's the code I used to create IndexWriter instance: // IndexWriter to use for adding contacts to the index final File INDEX_DIR = new File(indexDir); IndexWriter writer = new IndexWriter(FSDirectory.open(INDEX_DIR), analyzer, createFlag, IndexWriter.MaxFieldLength.LIMITED); But I'm not sure how to verify the initialization.
        Hide
        Jeremy Volkman added a comment -

        Is this contained within a single class? If so, maybe you could attach the entire thing?

        Show
        Jeremy Volkman added a comment - Is this contained within a single class? If so, maybe you could attach the entire thing?
        Bill Herbert made changes -
        Field Original Value New Value
        Attachment DigesterMarriesLucene.java [ 12443397 ]
        Attachment address-book.xml [ 12443398 ]
        Hide
        Bill Herbert added a comment -

        Thanks for your follow-up on this

        Following your observation that I might have writer==NULL, I did some checking and found:
        1) An instance of writer is created in main()
        2) A directory is created for its use
        3) Initially, there are no documents in this directory
        4) Within addContact(), a contactDocument object is created for the type "Individual"
        5) After adding name, address, etc. fields to contactDocument, there is no output from the statements:
        System.out.println("Within addContact(), Print name of Directory used by writer: " + writer.getDirectory());
        System.out.println("Statement in addContact() just prior to 'writer.adddocument', number of documents in contactDocument: " + writer.numDocs());
        6) Finally, an NPE is caused by the statement: writer.addDocument(contactDocument);

        The complete class listing is attached along with a small input file.
        FYI, included in my classpath are:
        lucene-core-3.0.1.jar
        commons-digester-2.0.jar
        commons-beanutils-1.8.3.jar
        commons-logging-1.1.1.jar

        Bill

        Show
        Bill Herbert added a comment - Thanks for your follow-up on this Following your observation that I might have writer==NULL, I did some checking and found: 1) An instance of writer is created in main() 2) A directory is created for its use 3) Initially, there are no documents in this directory 4) Within addContact(), a contactDocument object is created for the type "Individual" 5) After adding name, address, etc. fields to contactDocument, there is no output from the statements: System.out.println("Within addContact(), Print name of Directory used by writer: " + writer.getDirectory()); System.out.println("Statement in addContact() just prior to 'writer.adddocument', number of documents in contactDocument: " + writer.numDocs()); 6) Finally, an NPE is caused by the statement: writer.addDocument(contactDocument); The complete class listing is attached along with a small input file. FYI, included in my classpath are: lucene-core-3.0.1.jar commons-digester-2.0.jar commons-beanutils-1.8.3.jar commons-logging-1.1.1.jar Bill
        Hide
        Uwe Schindler added a comment -

        On the first look, I see writer is null and never gets initialized. This can be easily verified:

        • The global static IndexWriter is not initialized in its declaration
        • The main method has a local IndexWriter writer declaration, the static one is untouched and stays null.

        So this is not a bug of Lucene, but your code.

        Show
        Uwe Schindler added a comment - On the first look, I see writer is null and never gets initialized. This can be easily verified: The global static IndexWriter is not initialized in its declaration The main method has a local IndexWriter writer declaration, the static one is untouched and stays null. So this is not a bug of Lucene, but your code.
        Uwe Schindler made changes -
        Status Open [ 1 ] Resolved [ 5 ]
        Resolution Invalid [ 6 ]
        Uwe Schindler made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        Hide
        Jeremy Volkman added a comment -

        Bill, your static member "writer" is never being initialized. You intend to do this in your main method with the line

        IndexWriter writer = new IndexWriter(FSDirectory.open(INDEX_DIR), analyzer, createFlag, IndexWriter.MaxFieldLength.LIMITED);

        However, that line is creating a local variable within the main method called "writer", which is not visible to the addContact method. Change that line to

        writer = new IndexWriter(FSDirectory.open(INDEX_DIR), analyzer, createFlag, IndexWriter.MaxFieldLength.LIMITED);

        Show
        Jeremy Volkman added a comment - Bill, your static member "writer" is never being initialized. You intend to do this in your main method with the line IndexWriter writer = new IndexWriter(FSDirectory.open(INDEX_DIR), analyzer, createFlag, IndexWriter.MaxFieldLength.LIMITED); However, that line is creating a local variable within the main method called "writer", which is not visible to the addContact method. Change that line to writer = new IndexWriter(FSDirectory.open(INDEX_DIR), analyzer, createFlag, IndexWriter.MaxFieldLength.LIMITED);
        Hide
        Bill Herbert added a comment -

        Jeremy,
        Thanks so much for your help.
        The code is now running just fine.
        Bill

        Show
        Bill Herbert added a comment - Jeremy, Thanks so much for your help. The code is now running just fine. Bill
        Mark Thomas made changes -
        Workflow jira [ 12509722 ] Default workflow, editable Closed status [ 12563574 ]
        Mark Thomas made changes -
        Workflow Default workflow, editable Closed status [ 12563574 ] jira [ 12584311 ]

          People

          • Assignee:
            Unassigned
            Reporter:
            Bill Herbert
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Time Tracking

              Estimated:
              Original Estimate - 168h
              168h
              Remaining:
              Remaining Estimate - 168h
              168h
              Logged:
              Time Spent - Not Specified
              Not Specified

                Development