[SOLR-2644] DIH handler - when using threads=2 the default logging is set too high - ASF JIRA

Details

Type: Bug
Status: Closed
Priority: Major
Resolution: Fixed
Affects Version/s: 3.3
Fix Version/s: 3.4, 4.0-ALPHA
Component/s: contrib - DataImportHandler
Labels:
None

Description

Setting threads parameter in DIH handler, every add outputs to the log in INFO level.
The only current solution is to set the following in log4j.properties:

log4j.rootCategory=INFO, logfile
log4j.logger.org.apache.solr.handler.dataimport.DocBuilder=FATAL
log4j.logger.org.apache.solr.handler.dataimport.ThreadedEntityProcessorWrapper=FATAL

These 2 log messages need to be changed to DEBUG.

Options
- Sort By Name
- Sort By Date
- Ascending
- Descending

Attachments

SOLR-2644.patch

12/Jul/11 09:10

2 kB

Shalin Shekhar Mangar
SOLR-2644.patch

11/Jul/11 13:42

0.6 kB

Shalin Shekhar Mangar
SOLR-2644-2.patch

12/Jul/11 02:10

2 kB

Bill Bell

Activity

Ascending order - Click to sort in descending order

Hide

Permalink

Bill Bell added a comment - 10/Jul/11 23:21

Example of threads="2".
<document>
<entity name="main" threads="2" transformer="RegexTransformer"
query="select * from table">
</entity>
</document>

Show

Bill Bell added a comment - 10/Jul/11 23:21 Example of threads="2". <document> <entity name="main" threads="2" transformer="RegexTransformer" query="select * from table"> </entity> </document>

Hide

Permalink

Bill Bell added a comment - 10/Jul/11 23:24

Example logging:

Jul 10, 2011 5:21:50 PM org.apache.solr.handler.dataimport.ThreadedEntityProcessorWrapper nextRow
INFO: arow :

{id=635, priority=1, prof_type=1, user_query=Clinical Biochemical Geneticist}

Jul 10, 2011 5:21:50 PM org.apache.solr.handler.dataimport.ThreadedEntityProcessorWrapper nextRow
INFO: arow :

{id=636, priority=1, prof_type=1, user_query=Clinical Cytogeneticist}

Jul 10, 2011 5:21:50 PM org.apache.solr.handler.dataimport.DocBuilder$EntityRunner runAThread
INFO: a row on docrootSolrInputDocument[{id=id(1.0)=

{635}

, priority=priority(1.0)=

{1}, prof_type=prof_type(1.0)={1}

, user_query=user_query(1.0)={Clinical Biochemical Geneticist}}]

Show

Bill Bell added a comment - 10/Jul/11 23:24 Example logging: Jul 10, 2011 5:21:50 PM org.apache.solr.handler.dataimport.ThreadedEntityProcessorWrapper nextRow INFO: arow : {id=635, priority=1, prof_type=1, user_query=Clinical Biochemical Geneticist} Jul 10, 2011 5:21:50 PM org.apache.solr.handler.dataimport.ThreadedEntityProcessorWrapper nextRow INFO: arow : {id=636, priority=1, prof_type=1, user_query=Clinical Cytogeneticist} Jul 10, 2011 5:21:50 PM org.apache.solr.handler.dataimport.DocBuilder$EntityRunner runAThread INFO: a row on docrootSolrInputDocument[{id=id(1.0)= {635} , priority=priority(1.0)= {1}, prof_type=prof_type(1.0)={1} , user_query=user_query(1.0)={Clinical Biochemical Geneticist}}]

Hide

Permalink

Bill Bell added a comment - 11/Jul/11 04:19 - edited

Example:

Jul 10, 2011 10:17:54 PM org.apache.solr.handler.dataimport.ThreadedEntityProcessorWrapper nextRow
INFO: arow :

{id=622, priority=1, prof_type=3, user_query=Naturopath}

Jul 10, 2011 10:17:54 PM org.apache.solr.handler.dataimport.DocBuilder$EntityRunner runAThread
INFO: a row on docrootSolrInputDocument[{id=id(1.0)=

{622}

, priority=priority(1.0)=

{1}

, prof_type=prof_type(1.0)=

{3}

, user_query=user_query(1.0)={Naturopath}}]

logging.properties blocks these 2 messages:

# Default global logging level:
.level=INFO

# Write to a file:
handlers=java.util.logging.FileHandler

# Write log messages in XML format (SimpleFormatter or XMLFormatter)
java.util.logging.FileHandler.formatter=java.util.logging.SimpleFormatter

# Log to the current working directory, with log files named solrxxx.log
java.util.logging.FileHandler.pattern=C:/solr/jetty-6.1.25/jetty-6.1.25/solr/logs/Solr%u.log.txt

# Limiting size of output file in bytes: 1GB
java.util.logging.FileHandler.limit=1024000000

# Number of output files to cycle through, by appending an 
# integer to the base file name: (faster performance when set to > 1)
java.util.logging.FileHandler.count=1

java.util.logging.FileHandler.append=1

org.apache.solr.handler.dataimport.DocBuilder.level = SEVERE
org.apache.solr.handler.dataimport.ThreadedEntityProcessorWrapper.level = SEVERE

Show

Bill Bell added a comment - 11/Jul/11 04:19 - edited Example: Jul 10, 2011 10:17:54 PM org.apache.solr.handler.dataimport.ThreadedEntityProcessorWrapper nextRow INFO: arow : {id=622, priority=1, prof_type=3, user_query=Naturopath} Jul 10, 2011 10:17:54 PM org.apache.solr.handler.dataimport.DocBuilder$EntityRunner runAThread INFO: a row on docrootSolrInputDocument[{id=id(1.0)= {622} , priority=priority(1.0)= {1} , prof_type=prof_type(1.0)= {3} , user_query=user_query(1.0)={Naturopath}}] logging.properties blocks these 2 messages: # Default global logging level: .level=INFO # Write to a file: handlers=java.util.logging.FileHandler # Write log messages in XML format (SimpleFormatter or XMLFormatter) java.util.logging.FileHandler.formatter=java.util.logging.SimpleFormatter # Log to the current working directory, with log files named solrxxx.log java.util.logging.FileHandler.pattern=C:/solr/jetty-6.1.25/jetty-6.1.25/solr/logs/Solr%u.log.txt # Limiting size of output file in bytes: 1GB java.util.logging.FileHandler.limit=1024000000 # Number of output files to cycle through, by appending an # integer to the base file name: (faster performance when set to > 1) java.util.logging.FileHandler.count=1 java.util.logging.FileHandler.append=1 org.apache.solr.handler.dataimport.DocBuilder.level = SEVERE org.apache.solr.handler.dataimport.ThreadedEntityProcessorWrapper.level = SEVERE

Hide

Permalink

Shalin Shekhar Mangar added a comment - 11/Jul/11 13:42

This was probably added for debugging. Attached patch to remove the extra logging.

I'll commit shortly.

Show

Shalin Shekhar Mangar added a comment - 11/Jul/11 13:42 This was probably added for debugging. Attached patch to remove the extra logging. I'll commit shortly.

Hide

Permalink

Bill Bell added a comment - 12/Jul/11 02:11

New patch you forgot solr/contrib/dataimporthandler/src/java/org/apache/solr/handler/dataimport/DocBuilder.java.

Also, I would rather change to debug and leave it.

Show

Bill Bell added a comment - 12/Jul/11 02:11 New patch you forgot solr/contrib/dataimporthandler/src/java/org/apache/solr/handler/dataimport/DocBuilder.java. Also, I would rather change to debug and leave it.

Hide

Permalink

Shalin Shekhar Mangar added a comment - 12/Jul/11 09:10

Thanks Bill.

I've added checked logging at both the places otherwise we pay the cost of Map.toString for nothing.

I'll commit this shortly.

Show

Shalin Shekhar Mangar added a comment - 12/Jul/11 09:10 Thanks Bill. I've added checked logging at both the places otherwise we pay the cost of Map.toString for nothing. I'll commit this shortly.

Hide

Permalink

Shalin Shekhar Mangar added a comment - 12/Jul/11 09:27

Committed revision 1145502 on trunk and 1145509 on branch_3x.

Show

Shalin Shekhar Mangar added a comment - 12/Jul/11 09:27 Committed revision 1145502 on trunk and 1145509 on branch_3x.

Hide

Permalink

Robert Muir added a comment - 16/Sep/11 14:48

bulk close for 3.4

Show

Robert Muir added a comment - 16/Sep/11 14:48 bulk close for 3.4

People

Assignee:

Shalin Shekhar Mangar

Reporter:

Bill Bell

Votes:

0 Vote for this issue

Watchers:

0 Start watching this issue

Dates

Created:

10/Jul/11 05:33

Updated:

16/Sep/11 14:48

Resolved:

12/Jul/11 09:27

Development

Agile

View on Board