Uploaded image for project: 'Solr'
  1. Solr
  2. SOLR-9430

Locale in <propertyWriter in DIH is looked up based on Locale displayName (language tag "en-US" or legacy name "en_US" does not work, English works)

    Details

      Description

      having a DIH with DB datasource and propertyWriter such as:

      <propertyWriter dateFormat="yyyy-MM-dd HH:mm:ss.SSSSSS" type="SimplePropertiesWriter" locale="en_US" />

      does not work with locale in form en_US as mentioned in documentation;
      Locale is being looked up by Locale.getDisplayName() which returns human readable representation as opposed to Locale.toLanguageTag, which returns form such as en-US.

      Propertywirter with locale in this form works:
      <propertyWriter dateFormat="yyyy-MM-dd HH:mm:ss.SSSSSS" type="SimplePropertiesWriter" locale="Slovak" />

      Problematic line code:
      https://github.com/apache/lucene-solr/blob/branch_6_1/solr/contrib/dataimporthandler/src/java/org/apache/solr/handler/dataimport/SimplePropertiesWriter.java#L95

      1. SOLR-9430.patch
        3 kB
        Uwe Schindler
      2. SOLR-9430.patch
        2 kB
        Uwe Schindler

        Issue Links

          Activity

          Hide
          thetaphi Uwe Schindler added a comment -

          You should have to use the language tag, the old Display name is just for backwards compatibility and was removed in latest Lucene versions. Unfortunately this part of Solr still uses the old and buggy code.

          There is an other issue where all uses of Solr was transformed to use language tags (LUCENE-6978), but this one seems to be missing. I will take the issue and fix it like the other places.

          Please keep in mind, that using "en_US" is no longer supported in Solr, you have to use "en-US" since Lucene/Solr 5.5.

          Show
          thetaphi Uwe Schindler added a comment - You should have to use the language tag, the old Display name is just for backwards compatibility and was removed in latest Lucene versions. Unfortunately this part of Solr still uses the old and buggy code. There is an other issue where all uses of Solr was transformed to use language tags ( LUCENE-6978 ), but this one seems to be missing. I will take the issue and fix it like the other places. Please keep in mind, that using "en_US" is no longer supported in Solr, you have to use "en-US" since Lucene/Solr 5.5.
          Hide
          bs76 Boris Steiner added a comment -

          Cool, thanks

          Show
          bs76 Boris Steiner added a comment - Cool, thanks
          Hide
          thetaphi Uwe Schindler added a comment -

          Here is a patch, handling this in the same way like elsewhere in DIH. This code primarily accepts language tags, but for backwards compatibility also old-style locale names (toString) and display names.

          Show
          thetaphi Uwe Schindler added a comment - Here is a patch, handling this in the same way like elsewhere in DIH. This code primarily accepts language tags, but for backwards compatibility also old-style locale names (toString) and display names.
          Hide
          thetaphi Uwe Schindler added a comment -

          The documentation team (hi, Cassandra Targett) should review Solr's Confluence Wiki for wrong locales. From 5.5 on, all Locales anywhere in config files should use the standardized "language tag" format (see https://tools.ietf.org/html/bcp47), although the old format is still supported, but "deprecated". We should at least fix the examples.

          Show
          thetaphi Uwe Schindler added a comment - The documentation team (hi, Cassandra Targett ) should review Solr's Confluence Wiki for wrong locales. From 5.5 on, all Locales anywhere in config files should use the standardized "language tag" format (see https://tools.ietf.org/html/bcp47 ), although the old format is still supported, but "deprecated". We should at least fix the examples.
          Hide
          thetaphi Uwe Schindler added a comment -

          Better patch that does not violate forbiddenapis and is consistent with other Solr code.

          Show
          thetaphi Uwe Schindler added a comment - Better patch that does not violate forbiddenapis and is consistent with other Solr code.
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit e99d9706741edb27b5ba373ad6fd84150a076705 in lucene-solr's branch refs/heads/master from Uwe Schindler
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=e99d970 ]

          SOLR-9430: Fix locale lookup in DIH <propertyWriter/> to use BCP47 language tags to be consistent with other places in Solr. Language names still work for backwards compatibility

          Show
          jira-bot ASF subversion and git services added a comment - Commit e99d9706741edb27b5ba373ad6fd84150a076705 in lucene-solr's branch refs/heads/master from Uwe Schindler [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=e99d970 ] SOLR-9430 : Fix locale lookup in DIH <propertyWriter/> to use BCP47 language tags to be consistent with other places in Solr. Language names still work for backwards compatibility
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 545ce387143bec3f598c1f2465afaa9aae12607d in lucene-solr's branch refs/heads/branch_6x from Uwe Schindler
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=545ce38 ]

          SOLR-9430: Fix locale lookup in DIH <propertyWriter/> to use BCP47 language tags to be consistent with other places in Solr. Language names still work for backwards compatibility

          Show
          jira-bot ASF subversion and git services added a comment - Commit 545ce387143bec3f598c1f2465afaa9aae12607d in lucene-solr's branch refs/heads/branch_6x from Uwe Schindler [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=545ce38 ] SOLR-9430 : Fix locale lookup in DIH <propertyWriter/> to use BCP47 language tags to be consistent with other places in Solr. Language names still work for backwards compatibility
          Hide
          shalinmangar Shalin Shekhar Mangar added a comment -

          Re-opened to back-port to 6.2.1

          Show
          shalinmangar Shalin Shekhar Mangar added a comment - Re-opened to back-port to 6.2.1
          Hide
          shalinmangar Shalin Shekhar Mangar added a comment -

          Closing after 6.2.1 release

          Show
          shalinmangar Shalin Shekhar Mangar added a comment - Closing after 6.2.1 release

            People

            • Assignee:
              thetaphi Uwe Schindler
              Reporter:
              bs76 Boris Steiner
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Time Tracking

                Estimated:
                Original Estimate - 24h
                24h
                Remaining:
                Remaining Estimate - 24h
                24h
                Logged:
                Time Spent - Not Specified
                Not Specified

                  Development