Solr
  1. Solr
  2. SOLR-1916

investigate DIH use of default locale

    Details

    • Type: Task Task
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 3.1, 4.0-ALPHA
    • Fix Version/s: 4.1, 5.0
    • Labels:
      None

      Description

      This is a spinoff from LUCENE-2466.

      In this issue I changed my locale to various locales and found some problems in Lucene/Solr triggered by use of the default Locale.

      I noticed some use of the default-locale for Date operations in DIH (TimeZone.getDefault/Locale.getDefault) and, while no tests fail, I think it might be better to support a locale parameter for this.

      The wiki documents that numeric parsing can support localized numerics formats: http://wiki.apache.org/solr/DataImportHandler#NumberFormatTransformer

      In both cases, I don't think we should ever use the default Locale. If no Locale is provided, I find that new Locale("") <-- Unicode Root Locale, is a better default for a server situation in a lot of cases, as it won't change depending on the computer, or perhaps we just make Locale params mandatory for this.

      Finally, in both cases, if localized numbers/dates are explicitly supported, I think we should come up with a test strategy to ensure everything is working. One idea is to do something similar to or make use of Lucene's LocalizedTestCase.

      1. SOLR-1916.patch
        0.8 kB
        Robert Muir
      2. SOLR-1916.patch
        6 kB
        James Dyer

        Issue Links

          Activity

          Hide
          Commit Tag Bot added a comment -

          [branch_4x commit] James Dyer
          http://svn.apache.org/viewvc?view=revision&revision=1411825

          SOLR-1916: DIH - fix remaining forbidden apis & remove build-time exclusion

          Show
          Commit Tag Bot added a comment - [branch_4x commit] James Dyer http://svn.apache.org/viewvc?view=revision&revision=1411825 SOLR-1916 : DIH - fix remaining forbidden apis & remove build-time exclusion
          Hide
          Commit Tag Bot added a comment -

          [branch_4x commit] James Dyer
          http://svn.apache.org/viewvc?view=revision&revision=1413692

          SOLR-1916: reinstate @Ignore'ed test

          Show
          Commit Tag Bot added a comment - [branch_4x commit] James Dyer http://svn.apache.org/viewvc?view=revision&revision=1413692 SOLR-1916 : reinstate @Ignore'ed test
          Hide
          Commit Tag Bot added a comment -

          [trunk commit] James Dyer
          http://svn.apache.org/viewvc?view=revision&revision=1413684

          SOLR-1916: reinstate @Ignore'ed test

          Show
          Commit Tag Bot added a comment - [trunk commit] James Dyer http://svn.apache.org/viewvc?view=revision&revision=1413684 SOLR-1916 : reinstate @Ignore'ed test
          Hide
          Commit Tag Bot added a comment -

          [branch_4x commit] James Dyer
          http://svn.apache.org/viewvc?view=revision&revision=1413692

          SOLR-1916: reinstate @Ignore'ed test

          Show
          Commit Tag Bot added a comment - [branch_4x commit] James Dyer http://svn.apache.org/viewvc?view=revision&revision=1413692 SOLR-1916 : reinstate @Ignore'ed test
          Hide
          Commit Tag Bot added a comment -

          [branch_4x commit] James Dyer
          http://svn.apache.org/viewvc?view=revision&revision=1411825

          SOLR-1916: DIH - fix remaining forbidden apis & remove build-time exclusion

          Show
          Commit Tag Bot added a comment - [branch_4x commit] James Dyer http://svn.apache.org/viewvc?view=revision&revision=1411825 SOLR-1916 : DIH - fix remaining forbidden apis & remove build-time exclusion
          Hide
          Commit Tag Bot added a comment -

          [trunk commit] James Dyer
          http://svn.apache.org/viewvc?view=revision&revision=1411823

          SOLR-1916: entry in CHANGES.txt

          Show
          Commit Tag Bot added a comment - [trunk commit] James Dyer http://svn.apache.org/viewvc?view=revision&revision=1411823 SOLR-1916 : entry in CHANGES.txt
          Hide
          Commit Tag Bot added a comment -

          [trunk commit] James Dyer
          http://svn.apache.org/viewvc?view=revision&revision=1411820

          SOLR-1916: DIH - fix remaining forbidden apis & remove build-time exclusion

          Show
          Commit Tag Bot added a comment - [trunk commit] James Dyer http://svn.apache.org/viewvc?view=revision&revision=1411820 SOLR-1916 : DIH - fix remaining forbidden apis & remove build-time exclusion
          Hide
          James Dyer added a comment -

          Patch that fixes remaining forbidden apis (mostly in tests). Also removes the DIH exception to "check-forbidden-apis".

          I will commit shortly and (finally) close this issue.

          Show
          James Dyer added a comment - Patch that fixes remaining forbidden apis (mostly in tests). Also removes the DIH exception to "check-forbidden-apis". I will commit shortly and (finally) close this issue.
          Hide
          Robert Muir added a comment -

          James, thanks for looking at this!!!

          It may not be a locale issue, instead a time zone issue (or both). But this test definitely failed intermittently in the past.

          For example, it failed during a Daylight savings time window (but only for developers in Europe!) and Chris Male addressed
          some of the issues in SOLR-1821.

          Fortunately, Uwe Schindler has made it dead easy to identify most of these issues: we not longer have to solely rely upon unit tests alone.

          http://blog.thetaphi.de/2012/07/default-locales-default-charsets-and.html

          DIH currently has 40 violations!

          Try this:

          Index: build.xml
          ===================================================================
          --- build.xml	(revision 1406757)
          +++ build.xml	(working copy)
          @@ -250,8 +250,6 @@
                 </apiFileSet>
                 <fileset dir="${basedir}/build">
                   <include name="**/*.class" />
          -        <!-- exclude DIH for now as it is broken with Locales and Encodings: SOLR-1916 -->
          -        <exclude name="contrib/solr-dataimporthandler*/**" />
                 </fileset>
               </forbidden-apis>
             </target>
          

          Then run

          rmuir@beast:~/workspace/lucene-trunk/solr$ ant check-forbidden-apis
          ...
          -check-forbidden-java-apis:
          [forbidden-apis] Reading API signatures: /home/rmuir/workspace/lucene-trunk/lucene/tools/forbiddenApis/commons-io.txt
          [forbidden-apis] Reading API signatures: /home/rmuir/workspace/lucene-trunk/lucene/tools/forbiddenApis/executors.txt
          [forbidden-apis] Reading API signatures: /home/rmuir/workspace/lucene-trunk/lucene/tools/forbiddenApis/jdk-deprecated.txt
          [forbidden-apis] Reading API signatures: /home/rmuir/workspace/lucene-trunk/lucene/tools/forbiddenApis/jdk.txt
          [forbidden-apis] Loading classes to check...
          [forbidden-apis] Scanning for API signatures and dependencies...
          [forbidden-apis] Forbidden method invocation: java.text.DecimalFormatSymbols#<init>()
          [forbidden-apis]   in org.apache.solr.handler.dataimport.TestNumberFormatTransformer (TestNumberFormatTransformer.java:36)
          [forbidden-apis] Forbidden method invocation: java.text.DecimalFormatSymbols#<init>()
          [forbidden-apis]   in org.apache.solr.handler.dataimport.TestNumberFormatTransformer (TestNumberFormatTransformer.java:37)
          [forbidden-apis] Forbidden method invocation: java.text.MessageFormat#<init>(java.lang.String)
          [forbidden-apis]   in org.apache.solr.handler.dataimport.DebugLogger (DebugLogger.java:52)
          [forbidden-apis] Forbidden method invocation: java.text.SimpleDateFormat#<init>(java.lang.String)
          [forbidden-apis]   in org.apache.solr.handler.dataimport.TestDateFormatTransformer (TestDateFormatTransformer.java:43)
          [forbidden-apis] Forbidden method invocation: java.text.SimpleDateFormat#<init>(java.lang.String)
          [forbidden-apis]   in org.apache.solr.handler.dataimport.TestDateFormatTransformer (TestDateFormatTransformer.java:66)
          [forbidden-apis] Forbidden method invocation: java.text.SimpleDateFormat#<init>(java.lang.String)
          [forbidden-apis]   in org.apache.solr.handler.dataimport.MailEntityProcessor (MailEntityProcessor.java:88)
          [forbidden-apis] Forbidden method invocation: java.lang.String#getBytes()
          [forbidden-apis]   in org.apache.solr.handler.dataimport.TestDocBuilder2 (TestDocBuilder2.java:250)
          [forbidden-apis] Forbidden method invocation: java.lang.String#getBytes()
          [forbidden-apis]   in org.apache.solr.handler.dataimport.TestDocBuilder2 (TestDocBuilder2.java:251)
          [forbidden-apis] Forbidden method invocation: java.lang.String#getBytes()
          [forbidden-apis]   in org.apache.solr.handler.dataimport.TestDocBuilder2 (TestDocBuilder2.java:252)
          [forbidden-apis] Forbidden method invocation: java.lang.String#getBytes()
          [forbidden-apis]   in org.apache.solr.handler.dataimport.TestDocBuilder2 (TestDocBuilder2.java:257)
          [forbidden-apis] Forbidden method invocation: java.text.SimpleDateFormat#<init>(java.lang.String)
          [forbidden-apis]   in org.apache.solr.handler.dataimport.DataImporter$3 (DataImporter.java:490)
          [forbidden-apis] Forbidden method invocation: java.lang.String#format(java.lang.String,java.lang.Object[])
          [forbidden-apis]   in org.apache.solr.handler.dataimport.DocBuilder (DocBuilder.java:711)
          [forbidden-apis] Forbidden method invocation: java.lang.String#format(java.lang.String,java.lang.Object[])
          [forbidden-apis]   in org.apache.solr.handler.dataimport.DocBuilder (DocBuilder.java:717)
          [forbidden-apis] Forbidden method invocation: java.lang.String#format(java.lang.String,java.lang.Object[])
          [forbidden-apis]   in org.apache.solr.handler.dataimport.DocBuilder (DocBuilder.java:725)
          [forbidden-apis] Forbidden method invocation: java.lang.String#format(java.lang.String,java.lang.Object[])
          [forbidden-apis]   in org.apache.solr.handler.dataimport.DocBuilder (DocBuilder.java:727)
          [forbidden-apis] Forbidden method invocation: java.io.InputStreamReader#<init>(java.io.InputStream)
          [forbidden-apis]   in org.apache.solr.handler.dataimport.FieldReaderDataSource (FieldReaderDataSource.java:109)
          [forbidden-apis] Forbidden method invocation: java.lang.String#getBytes()
          [forbidden-apis]   in org.apache.solr.handler.dataimport.TestFileListWithLineEntityProcessor (TestFileListWithLineEntityProcessor.java:36)
          [forbidden-apis] Forbidden method invocation: java.lang.String#getBytes()
          [forbidden-apis]   in org.apache.solr.handler.dataimport.TestFileListWithLineEntityProcessor (TestFileListWithLineEntityProcessor.java:37)
          [forbidden-apis] Forbidden method invocation: java.lang.String#getBytes()
          [forbidden-apis]   in org.apache.solr.handler.dataimport.TestFileListWithLineEntityProcessor (TestFileListWithLineEntityProcessor.java:38)
          [forbidden-apis] Forbidden method invocation: java.text.SimpleDateFormat#<init>(java.lang.String)
          [forbidden-apis]   in org.apache.solr.handler.dataimport.TestVariableResolver (TestVariableResolver.java:92)
          [forbidden-apis] Forbidden method invocation: java.text.SimpleDateFormat#<init>(java.lang.String)
          [forbidden-apis]   in org.apache.solr.handler.dataimport.TestVariableResolver (TestVariableResolver.java:101)
          [forbidden-apis] Forbidden method invocation: java.text.SimpleDateFormat#<init>(java.lang.String)
          [forbidden-apis]   in org.apache.solr.handler.dataimport.TestVariableResolver (TestVariableResolver.java:107)
          [forbidden-apis] Forbidden method invocation: java.text.SimpleDateFormat#<init>(java.lang.String)
          [forbidden-apis]   in org.apache.solr.handler.dataimport.TestVariableResolver (TestVariableResolver.java:138)
          [forbidden-apis] Forbidden method invocation: java.text.SimpleDateFormat#<init>(java.lang.String)
          [forbidden-apis]   in org.apache.solr.handler.dataimport.TestVariableResolver (TestVariableResolver.java:147)
          [forbidden-apis] Forbidden method invocation: java.text.SimpleDateFormat#<init>(java.lang.String)
          [forbidden-apis]   in org.apache.solr.handler.dataimport.EvaluatorBag$4 (EvaluatorBag.java:164)
          [forbidden-apis] Forbidden method invocation: java.util.GregorianCalendar#<init>()
          [forbidden-apis]   in org.apache.solr.handler.dataimport.TestEvaluatorBag (TestEvaluatorBag.java:122)
          [forbidden-apis] Forbidden method invocation: java.text.SimpleDateFormat#<init>(java.lang.String)
          [forbidden-apis]   in org.apache.solr.handler.dataimport.TestEvaluatorBag (TestEvaluatorBag.java:125)
          [forbidden-apis] Forbidden method invocation: java.util.GregorianCalendar#<init>()
          [forbidden-apis]   in org.apache.solr.handler.dataimport.TestEvaluatorBag (TestEvaluatorBag.java:128)
          [forbidden-apis] Forbidden method invocation: java.text.SimpleDateFormat#<init>(java.lang.String)
          [forbidden-apis]   in org.apache.solr.handler.dataimport.TestEvaluatorBag (TestEvaluatorBag.java:135)
          [forbidden-apis] Forbidden method invocation: java.lang.String#getBytes()
          [forbidden-apis]   in org.apache.solr.handler.dataimport.TestFileListEntityProcessor (TestFileListEntityProcessor.java:43)
          [forbidden-apis] Forbidden method invocation: java.lang.String#getBytes()
          [forbidden-apis]   in org.apache.solr.handler.dataimport.TestFileListEntityProcessor (TestFileListEntityProcessor.java:44)
          [forbidden-apis] Forbidden method invocation: java.lang.String#getBytes()
          [forbidden-apis]   in org.apache.solr.handler.dataimport.TestFileListEntityProcessor (TestFileListEntityProcessor.java:45)
          [forbidden-apis] Forbidden method invocation: java.lang.String#getBytes()
          [forbidden-apis]   in org.apache.solr.handler.dataimport.TestFileListEntityProcessor (TestFileListEntityProcessor.java:140)
          [forbidden-apis] Forbidden method invocation: java.lang.String#getBytes()
          [forbidden-apis]   in org.apache.solr.handler.dataimport.TestFileListEntityProcessor (TestFileListEntityProcessor.java:141)
          [forbidden-apis] Forbidden method invocation: java.lang.String#getBytes()
          [forbidden-apis]   in org.apache.solr.handler.dataimport.TestFileListEntityProcessor (TestFileListEntityProcessor.java:142)
          [forbidden-apis] Forbidden method invocation: java.lang.String#getBytes()
          [forbidden-apis]   in org.apache.solr.handler.dataimport.TestFileListEntityProcessor (TestFileListEntityProcessor.java:164)
          [forbidden-apis] Forbidden method invocation: java.lang.String#getBytes()
          [forbidden-apis]   in org.apache.solr.handler.dataimport.TestFileListEntityProcessor (TestFileListEntityProcessor.java:179)
          [forbidden-apis] Forbidden method invocation: java.lang.String#getBytes()
          [forbidden-apis]   in org.apache.solr.handler.dataimport.TestFileListEntityProcessor (TestFileListEntityProcessor.java:180)
          [forbidden-apis] Forbidden method invocation: java.lang.String#getBytes()
          [forbidden-apis]   in org.apache.solr.handler.dataimport.TestFileListEntityProcessor (TestFileListEntityProcessor.java:181)
          [forbidden-apis] Forbidden method invocation: java.io.InputStreamReader#<init>(java.io.InputStream)
          [forbidden-apis]   in org.apache.solr.handler.dataimport.FileDataSource (FileDataSource.java:128)
          [forbidden-apis] Scanned 2080 (and 648 related) class file(s) for forbidden API invocations (in 1.08s), 40 error(s).
          
          BUILD FAILED
          /home/rmuir/workspace/lucene-trunk/solr/build.xml:243: Check for forbidden API calls failed, see log.
          
          Show
          Robert Muir added a comment - James, thanks for looking at this!!! It may not be a locale issue, instead a time zone issue (or both). But this test definitely failed intermittently in the past. For example, it failed during a Daylight savings time window (but only for developers in Europe!) and Chris Male addressed some of the issues in SOLR-1821 . Fortunately, Uwe Schindler has made it dead easy to identify most of these issues: we not longer have to solely rely upon unit tests alone. http://blog.thetaphi.de/2012/07/default-locales-default-charsets-and.html DIH currently has 40 violations! Try this: Index: build.xml =================================================================== --- build.xml (revision 1406757) +++ build.xml (working copy) @@ -250,8 +250,6 @@ </apiFileSet> <fileset dir="${basedir}/build"> <include name="**/*.class" /> - <!-- exclude DIH for now as it is broken with Locales and Encodings: SOLR-1916 --> - <exclude name="contrib/solr-dataimporthandler*/**" /> </fileset> </forbidden-apis> </target> Then run rmuir@beast:~/workspace/lucene-trunk/solr$ ant check-forbidden-apis ... -check-forbidden-java-apis: [forbidden-apis] Reading API signatures: /home/rmuir/workspace/lucene-trunk/lucene/tools/forbiddenApis/commons-io.txt [forbidden-apis] Reading API signatures: /home/rmuir/workspace/lucene-trunk/lucene/tools/forbiddenApis/executors.txt [forbidden-apis] Reading API signatures: /home/rmuir/workspace/lucene-trunk/lucene/tools/forbiddenApis/jdk-deprecated.txt [forbidden-apis] Reading API signatures: /home/rmuir/workspace/lucene-trunk/lucene/tools/forbiddenApis/jdk.txt [forbidden-apis] Loading classes to check... [forbidden-apis] Scanning for API signatures and dependencies... [forbidden-apis] Forbidden method invocation: java.text.DecimalFormatSymbols#<init>() [forbidden-apis] in org.apache.solr.handler.dataimport.TestNumberFormatTransformer (TestNumberFormatTransformer.java:36) [forbidden-apis] Forbidden method invocation: java.text.DecimalFormatSymbols#<init>() [forbidden-apis] in org.apache.solr.handler.dataimport.TestNumberFormatTransformer (TestNumberFormatTransformer.java:37) [forbidden-apis] Forbidden method invocation: java.text.MessageFormat#<init>(java.lang.String) [forbidden-apis] in org.apache.solr.handler.dataimport.DebugLogger (DebugLogger.java:52) [forbidden-apis] Forbidden method invocation: java.text.SimpleDateFormat#<init>(java.lang.String) [forbidden-apis] in org.apache.solr.handler.dataimport.TestDateFormatTransformer (TestDateFormatTransformer.java:43) [forbidden-apis] Forbidden method invocation: java.text.SimpleDateFormat#<init>(java.lang.String) [forbidden-apis] in org.apache.solr.handler.dataimport.TestDateFormatTransformer (TestDateFormatTransformer.java:66) [forbidden-apis] Forbidden method invocation: java.text.SimpleDateFormat#<init>(java.lang.String) [forbidden-apis] in org.apache.solr.handler.dataimport.MailEntityProcessor (MailEntityProcessor.java:88) [forbidden-apis] Forbidden method invocation: java.lang.String#getBytes() [forbidden-apis] in org.apache.solr.handler.dataimport.TestDocBuilder2 (TestDocBuilder2.java:250) [forbidden-apis] Forbidden method invocation: java.lang.String#getBytes() [forbidden-apis] in org.apache.solr.handler.dataimport.TestDocBuilder2 (TestDocBuilder2.java:251) [forbidden-apis] Forbidden method invocation: java.lang.String#getBytes() [forbidden-apis] in org.apache.solr.handler.dataimport.TestDocBuilder2 (TestDocBuilder2.java:252) [forbidden-apis] Forbidden method invocation: java.lang.String#getBytes() [forbidden-apis] in org.apache.solr.handler.dataimport.TestDocBuilder2 (TestDocBuilder2.java:257) [forbidden-apis] Forbidden method invocation: java.text.SimpleDateFormat#<init>(java.lang.String) [forbidden-apis] in org.apache.solr.handler.dataimport.DataImporter$3 (DataImporter.java:490) [forbidden-apis] Forbidden method invocation: java.lang.String#format(java.lang.String,java.lang.Object[]) [forbidden-apis] in org.apache.solr.handler.dataimport.DocBuilder (DocBuilder.java:711) [forbidden-apis] Forbidden method invocation: java.lang.String#format(java.lang.String,java.lang.Object[]) [forbidden-apis] in org.apache.solr.handler.dataimport.DocBuilder (DocBuilder.java:717) [forbidden-apis] Forbidden method invocation: java.lang.String#format(java.lang.String,java.lang.Object[]) [forbidden-apis] in org.apache.solr.handler.dataimport.DocBuilder (DocBuilder.java:725) [forbidden-apis] Forbidden method invocation: java.lang.String#format(java.lang.String,java.lang.Object[]) [forbidden-apis] in org.apache.solr.handler.dataimport.DocBuilder (DocBuilder.java:727) [forbidden-apis] Forbidden method invocation: java.io.InputStreamReader#<init>(java.io.InputStream) [forbidden-apis] in org.apache.solr.handler.dataimport.FieldReaderDataSource (FieldReaderDataSource.java:109) [forbidden-apis] Forbidden method invocation: java.lang.String#getBytes() [forbidden-apis] in org.apache.solr.handler.dataimport.TestFileListWithLineEntityProcessor (TestFileListWithLineEntityProcessor.java:36) [forbidden-apis] Forbidden method invocation: java.lang.String#getBytes() [forbidden-apis] in org.apache.solr.handler.dataimport.TestFileListWithLineEntityProcessor (TestFileListWithLineEntityProcessor.java:37) [forbidden-apis] Forbidden method invocation: java.lang.String#getBytes() [forbidden-apis] in org.apache.solr.handler.dataimport.TestFileListWithLineEntityProcessor (TestFileListWithLineEntityProcessor.java:38) [forbidden-apis] Forbidden method invocation: java.text.SimpleDateFormat#<init>(java.lang.String) [forbidden-apis] in org.apache.solr.handler.dataimport.TestVariableResolver (TestVariableResolver.java:92) [forbidden-apis] Forbidden method invocation: java.text.SimpleDateFormat#<init>(java.lang.String) [forbidden-apis] in org.apache.solr.handler.dataimport.TestVariableResolver (TestVariableResolver.java:101) [forbidden-apis] Forbidden method invocation: java.text.SimpleDateFormat#<init>(java.lang.String) [forbidden-apis] in org.apache.solr.handler.dataimport.TestVariableResolver (TestVariableResolver.java:107) [forbidden-apis] Forbidden method invocation: java.text.SimpleDateFormat#<init>(java.lang.String) [forbidden-apis] in org.apache.solr.handler.dataimport.TestVariableResolver (TestVariableResolver.java:138) [forbidden-apis] Forbidden method invocation: java.text.SimpleDateFormat#<init>(java.lang.String) [forbidden-apis] in org.apache.solr.handler.dataimport.TestVariableResolver (TestVariableResolver.java:147) [forbidden-apis] Forbidden method invocation: java.text.SimpleDateFormat#<init>(java.lang.String) [forbidden-apis] in org.apache.solr.handler.dataimport.EvaluatorBag$4 (EvaluatorBag.java:164) [forbidden-apis] Forbidden method invocation: java.util.GregorianCalendar#<init>() [forbidden-apis] in org.apache.solr.handler.dataimport.TestEvaluatorBag (TestEvaluatorBag.java:122) [forbidden-apis] Forbidden method invocation: java.text.SimpleDateFormat#<init>(java.lang.String) [forbidden-apis] in org.apache.solr.handler.dataimport.TestEvaluatorBag (TestEvaluatorBag.java:125) [forbidden-apis] Forbidden method invocation: java.util.GregorianCalendar#<init>() [forbidden-apis] in org.apache.solr.handler.dataimport.TestEvaluatorBag (TestEvaluatorBag.java:128) [forbidden-apis] Forbidden method invocation: java.text.SimpleDateFormat#<init>(java.lang.String) [forbidden-apis] in org.apache.solr.handler.dataimport.TestEvaluatorBag (TestEvaluatorBag.java:135) [forbidden-apis] Forbidden method invocation: java.lang.String#getBytes() [forbidden-apis] in org.apache.solr.handler.dataimport.TestFileListEntityProcessor (TestFileListEntityProcessor.java:43) [forbidden-apis] Forbidden method invocation: java.lang.String#getBytes() [forbidden-apis] in org.apache.solr.handler.dataimport.TestFileListEntityProcessor (TestFileListEntityProcessor.java:44) [forbidden-apis] Forbidden method invocation: java.lang.String#getBytes() [forbidden-apis] in org.apache.solr.handler.dataimport.TestFileListEntityProcessor (TestFileListEntityProcessor.java:45) [forbidden-apis] Forbidden method invocation: java.lang.String#getBytes() [forbidden-apis] in org.apache.solr.handler.dataimport.TestFileListEntityProcessor (TestFileListEntityProcessor.java:140) [forbidden-apis] Forbidden method invocation: java.lang.String#getBytes() [forbidden-apis] in org.apache.solr.handler.dataimport.TestFileListEntityProcessor (TestFileListEntityProcessor.java:141) [forbidden-apis] Forbidden method invocation: java.lang.String#getBytes() [forbidden-apis] in org.apache.solr.handler.dataimport.TestFileListEntityProcessor (TestFileListEntityProcessor.java:142) [forbidden-apis] Forbidden method invocation: java.lang.String#getBytes() [forbidden-apis] in org.apache.solr.handler.dataimport.TestFileListEntityProcessor (TestFileListEntityProcessor.java:164) [forbidden-apis] Forbidden method invocation: java.lang.String#getBytes() [forbidden-apis] in org.apache.solr.handler.dataimport.TestFileListEntityProcessor (TestFileListEntityProcessor.java:179) [forbidden-apis] Forbidden method invocation: java.lang.String#getBytes() [forbidden-apis] in org.apache.solr.handler.dataimport.TestFileListEntityProcessor (TestFileListEntityProcessor.java:180) [forbidden-apis] Forbidden method invocation: java.lang.String#getBytes() [forbidden-apis] in org.apache.solr.handler.dataimport.TestFileListEntityProcessor (TestFileListEntityProcessor.java:181) [forbidden-apis] Forbidden method invocation: java.io.InputStreamReader#<init>(java.io.InputStream) [forbidden-apis] in org.apache.solr.handler.dataimport.FileDataSource (FileDataSource.java:128) [forbidden-apis] Scanned 2080 (and 648 related) class file(s) for forbidden API invocations (in 1.08s), 40 error(s). BUILD FAILED /home/rmuir/workspace/lucene-trunk/solr/build.xml:243: Check for forbidden API calls failed, see log.
          Hide
          James Dyer added a comment -

          Robert,

          I'm having a hard time finding a seed, locale, or timezone for which TestEvaluatorBag#testGetDateFormatEvaluator will fail. Can you provide more info? (Maybe my jvm doesn't support enough locales for me to get a failure-prone one?)

          Show
          James Dyer added a comment - Robert, I'm having a hard time finding a seed, locale, or timezone for which TestEvaluatorBag#testGetDateFormatEvaluator will fail. Can you provide more info? (Maybe my jvm doesn't support enough locales for me to get a failure-prone one?)
          Hide
          Robert Muir added a comment -

          It fails sometimes. Remember lucene's tests choose a random locale and timezone each time.

          I disabled only tests that sometimes fail.

          Show
          Robert Muir added a comment - It fails sometimes. Remember lucene's tests choose a random locale and timezone each time. I disabled only tests that sometimes fail.
          Hide
          Mark Waddle added a comment -

          Robert,
          Why is the get testGetDateFormatEvaluator test ignored? The test passes as I would expect and it looks to me like it is testing the right things. I want to re-activate it for related SOLR-2201, but want to be sure it is a good idea.
          Thanks,
          Mark

          Show
          Mark Waddle added a comment - Robert, Why is the get testGetDateFormatEvaluator test ignored? The test passes as I would expect and it looks to me like it is testing the right things. I want to re-activate it for related SOLR-2201 , but want to be sure it is a good idea. Thanks, Mark
          Hide
          Robert Muir added a comment -

          I committed the warning message (1065891, 1065892 in 3x).

          Hoping we can improve this one in the future, at least to re-enable the time-related tests.

          Show
          Robert Muir added a comment - I committed the warning message (1065891, 1065892 in 3x). Hoping we can improve this one in the future, at least to re-enable the time-related tests.
          Hide
          Robert Muir added a comment -

          patch adding a README.txt to contrib/dataimporthandler with a warning
          similar to what was in Solr 1.4.1

          Show
          Robert Muir added a comment - patch adding a README.txt to contrib/dataimporthandler with a warning similar to what was in Solr 1.4.1
          Hide
          Robert Muir added a comment -

          I investigated and dove in... definitely a pretty big/invasive change here.

          My approach was to require locale and timezone to be mandatory parameters for DIH... even then
          I ended up having to modify a huge amount of code to try to prevent any problems.

          My recommendation for 3.1 would be to take the warning that was in Solr 1.4.1 and add it to DIH's CHANGES.txt
          for the release, but additionally recommending that you set -Duser.timezone to match the database tz...
          (I think, I am not a timezone policeman!)

          Show
          Robert Muir added a comment - I investigated and dove in... definitely a pretty big/invasive change here. My approach was to require locale and timezone to be mandatory parameters for DIH... even then I ended up having to modify a huge amount of code to try to prevent any problems. My recommendation for 3.1 would be to take the warning that was in Solr 1.4.1 and add it to DIH's CHANGES.txt for the release, but additionally recommending that you set -Duser.timezone to match the database tz... (I think, I am not a timezone policeman!)
          Hide
          Fuad Efendi added a comment -

          I had similar issue, Microsoft SQL Server, DATETIME type.
          DIH stores Date in a filesystem using default (for SOLR) timezone and locale. Then, Delta Import executed query with WHERE last_update_date > '01.12.2010' (just as a sample).
          Localized string is used instead of real date. And timezone of remote database is not necessarily the same as timezone of SOLR.
          Fortunately, it's easy to fix (without altering code).

          Show
          Fuad Efendi added a comment - I had similar issue, Microsoft SQL Server, DATETIME type. DIH stores Date in a filesystem using default (for SOLR) timezone and locale. Then, Delta Import executed query with WHERE last_update_date > '01.12.2010' (just as a sample). Localized string is used instead of real date. And timezone of remote database is not necessarily the same as timezone of SOLR. Fortunately, it's easy to fix (without altering code).
          Hide
          Hoss Man added a comment -

          Bulk updating 240 Solr issues to set the Fix Version to "next" per the process outlined in this email...

          http://mail-archives.apache.org/mod_mbox/lucene-dev/201005.mbox/%3Calpine.DEB.1.10.1005251052040.24672@radix.cryptio.net%3E

          Selection criteria was "Unresolved" with a Fix Version of 1.5, 1.6, 3.1, or 4.0. email notifications were suppressed.

          A unique token for finding these 240 issues in the future: hossversioncleanup20100527

          Show
          Hoss Man added a comment - Bulk updating 240 Solr issues to set the Fix Version to "next" per the process outlined in this email... http://mail-archives.apache.org/mod_mbox/lucene-dev/201005.mbox/%3Calpine.DEB.1.10.1005251052040.24672@radix.cryptio.net%3E Selection criteria was "Unresolved" with a Fix Version of 1.5, 1.6, 3.1, or 4.0. email notifications were suppressed. A unique token for finding these 240 issues in the future: hossversioncleanup20100527

            People

            • Assignee:
              James Dyer
              Reporter:
              Robert Muir
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development