Solr
  1. Solr
  2. SOLR-2644

DIH handler - when using threads=2 the default logging is set too high

    Details

      Description

      Setting threads parameter in DIH handler, every add outputs to the log in INFO level.
      The only current solution is to set the following in log4j.properties:

      log4j.rootCategory=INFO, logfile
      log4j.logger.org.apache.solr.handler.dataimport.DocBuilder=FATAL
      log4j.logger.org.apache.solr.handler.dataimport.ThreadedEntityProcessorWrapper=FATAL

      These 2 log messages need to be changed to DEBUG.

      1. SOLR-2644.patch
        2 kB
        Shalin Shekhar Mangar
      2. SOLR-2644.patch
        0.6 kB
        Shalin Shekhar Mangar
      3. SOLR-2644-2.patch
        2 kB
        Bill Bell

        Activity

        Hide
        Bill Bell added a comment -

        Example of threads="2".
        <document>
        <entity name="main" threads="2" transformer="RegexTransformer"
        query="select * from table">
        </entity>
        </document>

        Show
        Bill Bell added a comment - Example of threads="2". <document> <entity name="main" threads="2" transformer="RegexTransformer" query="select * from table"> </entity> </document>
        Hide
        Bill Bell added a comment -

        Example logging:

        Jul 10, 2011 5:21:50 PM org.apache.solr.handler.dataimport.ThreadedEntityProcessorWrapper nextRow
        INFO: arow :

        {id=635, priority=1, prof_type=1, user_query=Clinical Biochemical Geneticist}

        Jul 10, 2011 5:21:50 PM org.apache.solr.handler.dataimport.ThreadedEntityProcessorWrapper nextRow
        INFO: arow :

        {id=636, priority=1, prof_type=1, user_query=Clinical Cytogeneticist}

        Jul 10, 2011 5:21:50 PM org.apache.solr.handler.dataimport.DocBuilder$EntityRunner runAThread
        INFO: a row on docrootSolrInputDocument[{id=id(1.0)=

        {635}

        , priority=priority(1.0)=

        {1}, prof_type=prof_type(1.0)={1}

        , user_query=user_query(1.0)={Clinical Biochemical Geneticist}}]

        Show
        Bill Bell added a comment - Example logging: Jul 10, 2011 5:21:50 PM org.apache.solr.handler.dataimport.ThreadedEntityProcessorWrapper nextRow INFO: arow : {id=635, priority=1, prof_type=1, user_query=Clinical Biochemical Geneticist} Jul 10, 2011 5:21:50 PM org.apache.solr.handler.dataimport.ThreadedEntityProcessorWrapper nextRow INFO: arow : {id=636, priority=1, prof_type=1, user_query=Clinical Cytogeneticist} Jul 10, 2011 5:21:50 PM org.apache.solr.handler.dataimport.DocBuilder$EntityRunner runAThread INFO: a row on docrootSolrInputDocument[{id=id(1.0)= {635} , priority=priority(1.0)= {1}, prof_type=prof_type(1.0)={1} , user_query=user_query(1.0)={Clinical Biochemical Geneticist}}]
        Hide
        Bill Bell added a comment - - edited

        Example:

        Jul 10, 2011 10:17:54 PM org.apache.solr.handler.dataimport.ThreadedEntityProcessorWrapper nextRow
        INFO: arow :

        {id=622, priority=1, prof_type=3, user_query=Naturopath}

        Jul 10, 2011 10:17:54 PM org.apache.solr.handler.dataimport.DocBuilder$EntityRunner runAThread
        INFO: a row on docrootSolrInputDocument[{id=id(1.0)=

        {622}

        , priority=priority(1.0)=

        {1}

        , prof_type=prof_type(1.0)=

        {3}

        , user_query=user_query(1.0)={Naturopath}}]

        logging.properties blocks these 2 messages:

        # Default global logging level:
        .level=INFO
        
        # Write to a file:
        handlers=java.util.logging.FileHandler
        
        # Write log messages in XML format (SimpleFormatter or XMLFormatter)
        java.util.logging.FileHandler.formatter=java.util.logging.SimpleFormatter
        
        # Log to the current working directory, with log files named solrxxx.log
        java.util.logging.FileHandler.pattern=C:/solr/jetty-6.1.25/jetty-6.1.25/solr/logs/Solr%u.log.txt
        
        # Limiting size of output file in bytes: 1GB
        java.util.logging.FileHandler.limit=1024000000
        
        # Number of output files to cycle through, by appending an 
        # integer to the base file name: (faster performance when set to > 1)
        java.util.logging.FileHandler.count=1
        
        java.util.logging.FileHandler.append=1
        
        org.apache.solr.handler.dataimport.DocBuilder.level = SEVERE
        org.apache.solr.handler.dataimport.ThreadedEntityProcessorWrapper.level = SEVERE
        
        Show
        Bill Bell added a comment - - edited Example: Jul 10, 2011 10:17:54 PM org.apache.solr.handler.dataimport.ThreadedEntityProcessorWrapper nextRow INFO: arow : {id=622, priority=1, prof_type=3, user_query=Naturopath} Jul 10, 2011 10:17:54 PM org.apache.solr.handler.dataimport.DocBuilder$EntityRunner runAThread INFO: a row on docrootSolrInputDocument[{id=id(1.0)= {622} , priority=priority(1.0)= {1} , prof_type=prof_type(1.0)= {3} , user_query=user_query(1.0)={Naturopath}}] logging.properties blocks these 2 messages: # Default global logging level: .level=INFO # Write to a file: handlers=java.util.logging.FileHandler # Write log messages in XML format (SimpleFormatter or XMLFormatter) java.util.logging.FileHandler.formatter=java.util.logging.SimpleFormatter # Log to the current working directory, with log files named solrxxx.log java.util.logging.FileHandler.pattern=C:/solr/jetty-6.1.25/jetty-6.1.25/solr/logs/Solr%u.log.txt # Limiting size of output file in bytes: 1GB java.util.logging.FileHandler.limit=1024000000 # Number of output files to cycle through, by appending an # integer to the base file name: (faster performance when set to > 1) java.util.logging.FileHandler.count=1 java.util.logging.FileHandler.append=1 org.apache.solr.handler.dataimport.DocBuilder.level = SEVERE org.apache.solr.handler.dataimport.ThreadedEntityProcessorWrapper.level = SEVERE
        Hide
        Shalin Shekhar Mangar added a comment -

        This was probably added for debugging. Attached patch to remove the extra logging.

        I'll commit shortly.

        Show
        Shalin Shekhar Mangar added a comment - This was probably added for debugging. Attached patch to remove the extra logging. I'll commit shortly.
        Hide
        Bill Bell added a comment -

        New patch you forgot solr/contrib/dataimporthandler/src/java/org/apache/solr/handler/dataimport/DocBuilder.java.

        Also, I would rather change to debug and leave it.

        Show
        Bill Bell added a comment - New patch you forgot solr/contrib/dataimporthandler/src/java/org/apache/solr/handler/dataimport/DocBuilder.java. Also, I would rather change to debug and leave it.
        Hide
        Shalin Shekhar Mangar added a comment -

        Thanks Bill.

        I've added checked logging at both the places otherwise we pay the cost of Map.toString for nothing.

        I'll commit this shortly.

        Show
        Shalin Shekhar Mangar added a comment - Thanks Bill. I've added checked logging at both the places otherwise we pay the cost of Map.toString for nothing. I'll commit this shortly.
        Hide
        Shalin Shekhar Mangar added a comment -

        Committed revision 1145502 on trunk and 1145509 on branch_3x.

        Show
        Shalin Shekhar Mangar added a comment - Committed revision 1145502 on trunk and 1145509 on branch_3x.
        Hide
        Robert Muir added a comment -

        bulk close for 3.4

        Show
        Robert Muir added a comment - bulk close for 3.4

          People

          • Assignee:
            Shalin Shekhar Mangar
            Reporter:
            Bill Bell
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development