Geronimo
  1. Geronimo
  2. GERONIMO-4763

i18n properties files should be converted to ascii at build time.

    Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 2.1.5, 2.2
    • Fix Version/s: 2.2
    • Component/s: usability
    • Security Level: public (Regular issues)
    • Labels:
      None

      Description

      Current i18n properties files are stored in source code repo after they are converted to ascii from native offline. It's very hard to contribute new translations.

      We should keep native characters in source code while convert them to ascii at build time. maven plugin native2ascii-maven-plugin could be used here:

      <plugin>
                      <groupId>org.codehaus.mojo</groupId>
                      <artifactId>native2ascii-maven-plugin</artifactId>
                      <version>1.0-alpha-1</version>
                      <configuration>
                          <dest>target/classes</dest>
                          <src>src/main/resources</src>
                      </configuration>
                      <executions>
                          <execution>
                              <id>native2ascii-utf8</id>
                              <goals>
                                  <goal>native2ascii</goal>
                              </goals>
                              <configuration>
                                  <encoding>UTF8</encoding>
                                  <includes>
                                      ConsoleResources_jp.properties,
                                      ConsoleResources_zh*.properties
                                  </includes>
                              </configuration>
                          </execution>
                      </executions>
                  </plugin>
      
      1. G4763_fix_inactive_profiles.patch
        3 kB
        Shawn Jiang
      2. G4763_fix_IBM_SDK.patch
        1.0 kB
        Shawn Jiang
      3. G4763_mv_i18n_trunk.bat
        2 kB
        Shawn Jiang
      4. G4763_trunk.patch
        547 kB
        Shawn Jiang

        Activity

        Hide
        Rex Wang added a comment -

        I like this
        -Rex

        Show
        Rex Wang added a comment - I like this -Rex
        Hide
        David Jencks added a comment -

        What are the options that will work here? I can think of a few but I don't know which will work:

        – everything is in ascii unicode escape sequences all the time (is this what we have now?)
        – everything is in utf-8 all the time (svn, checked out source code, any compiled property files
        – utf-8 in svn and checkout and unicode escape after compilation

        The last two require that svn handle checkout and checkin of utf-8 files without problem on various line-ending systems. Has anyone verified that this would work? Also we might investigate what these property files would look like in common IDES such as idea, eclipse, emacs, vim.

        Assuming utf-8 in svn and IDEs works ok, why would we change to ascii during compilation?

        Show
        David Jencks added a comment - What are the options that will work here? I can think of a few but I don't know which will work: – everything is in ascii unicode escape sequences all the time (is this what we have now?) – everything is in utf-8 all the time (svn, checked out source code, any compiled property files – utf-8 in svn and checkout and unicode escape after compilation The last two require that svn handle checkout and checkin of utf-8 files without problem on various line-ending systems. Has anyone verified that this would work? Also we might investigate what these property files would look like in common IDES such as idea, eclipse, emacs, vim. Assuming utf-8 in svn and IDEs works ok, why would we change to ascii during compilation?
        Hide
        Shawn Jiang added a comment - - edited
        - everything is in ascii unicode escape sequences all the time (is this what we have now?)
        

        yes, this is what we have. use \uXXXX ascii as the i18n resource storage format.

        - everything is in utf-8 all the time (svn, checked out source code, any compiled property files
        

        not exactly, we can't use utf-8(except 8859-1 char) in compiled properties because resource bundle only happy with \uXXXX escaped format with 8859-1 charset.

        - utf-8 in svn and checkout and unicode escape after compilation
        

        This is what I'm going to do.EOL of svn does not have any relationship with this i18n content encoding. I've tested Eclipse, notepad, gedit, vim. They are all happy with UTF-8 i18n properties. I think if the editor support utf-8, there's no reason it can't read UTF-8 i18n native language properties.

        As for the question "why would we change to ascii during compilation?"

        The problem is with resource bundles. For resouce bundle, Properties files are always read as ISO-8859-1. As a result, To include Unicode characters in i18n properties file, we must use \uXXXX escapes(with native2ascii tool).

        Show
        Shawn Jiang added a comment - - edited - everything is in ascii unicode escape sequences all the time (is this what we have now?) yes, this is what we have. use \uXXXX ascii as the i18n resource storage format. - everything is in utf-8 all the time (svn, checked out source code, any compiled property files not exactly, we can't use utf-8(except 8859-1 char) in compiled properties because resource bundle only happy with \uXXXX escaped format with 8859-1 charset. - utf-8 in svn and checkout and unicode escape after compilation This is what I'm going to do.EOL of svn does not have any relationship with this i18n content encoding. I've tested Eclipse, notepad, gedit, vim. They are all happy with UTF-8 i18n properties. I think if the editor support utf-8, there's no reason it can't read UTF-8 i18n native language properties. As for the question "why would we change to ascii during compilation?" The problem is with resource bundles. For resouce bundle, Properties files are always read as ISO-8859-1. As a result, To include Unicode characters in i18n properties file, we must use \uXXXX escapes(with native2ascii tool).
        Hide
        Shawn Jiang added a comment -

        1, Please apply G4763_trunk.patch with trunk code firstly. You need to verify if the patched i18n file is UTF-8 encoding. Eg. If you are using TortoiseSVN on windows. Please make sure default UTF-8 patch option is on.

        TortoiseMerge-->menu View --->settings--->general -->default to UTF-8 encoding.
        

        2, Use G4763_mv_i18n_trunk.bat to svn mvn all needed i18n resource to i18n-resources.

        Please replace the trunk path in the script to your local trunk path before running the script.

        Show
        Shawn Jiang added a comment - 1, Please apply G4763_trunk.patch with trunk code firstly. You need to verify if the patched i18n file is UTF-8 encoding. Eg. If you are using TortoiseSVN on windows. Please make sure default UTF-8 patch option is on. TortoiseMerge-->menu View --->settings--->general -->default to UTF-8 encoding. 2, Use G4763_mv_i18n_trunk.bat to svn mvn all needed i18n resource to i18n-resources. Please replace the trunk path in the script to your local trunk path before running the script.
        Hide
        Ivan added a comment -

        If no objection, I would like to do this change to 2.2, it should make the i18n work more easilier.

        Show
        Ivan added a comment - If no objection, I would like to do this change to 2.2, it should make the i18n work more easilier.
        Hide
        Kan Ogawa added a comment -

        Ivan,

        In addition, is it possible to change this to 2.1?

        Show
        Kan Ogawa added a comment - Ivan, In addition, is it possible to change this to 2.1?
        Hide
        Shawn Jiang added a comment -

        Kan,

        The reason why I just uploaded patch on trunk is that we are going to branch 2.2. I'll make a 2.1 patch soon.

        Show
        Shawn Jiang added a comment - Kan, The reason why I just uploaded patch on trunk is that we are going to branch 2.2. I'll make a 2.1 patch soon.
        Hide
        Shawn Jiang added a comment - - edited

        Clearing this comments because there's no problem with NON-SUN JDK at all.

        Show
        Shawn Jiang added a comment - - edited Clearing this comments because there's no problem with NON-SUN JDK at all.
        Hide
        Jack Cai added a comment -

        I think it is "sun.tools.native2ascii.*". Most JDKs should contain these classes in their tools.jar.

        Show
        Jack Cai added a comment - I think it is "sun.tools.native2ascii.*". Most JDKs should contain these classes in their tools.jar.
        Hide
        Shawn Jiang added a comment -

        Jack,

        Thanks for your reminder, you are right. IBM JDK also has these classes. Actually I just tried to build the patched geronimo with IBM JDK. There's no problem !

        There was a misleading JIRA on native2ascii maven plugin here: http://jira.codehaus.org/browse/MOJO-682 . I turned out that the reporter is using JRE instead of JDK when he is using the plugin. _____!

        Show
        Shawn Jiang added a comment - Jack, Thanks for your reminder, you are right. IBM JDK also has these classes. Actually I just tried to build the patched geronimo with IBM JDK. There's no problem ! There was a misleading JIRA on native2ascii maven plugin here: http://jira.codehaus.org/browse/MOJO-682 . I turned out that the reporter is using JRE instead of JDK when he is using the plugin. _____ !
        Hide
        David Jencks added a comment -

        Will the native2ascii work with harmony? If not directly do they have an equivalent solution?

        Show
        David Jencks added a comment - Will the native2ascii work with harmony? If not directly do they have an equivalent solution?
        Hide
        Lars Kühne added a comment -

        There are Eclipse plugins like eclipse-rbe that do the ascii conversion on the fly. Idea also supports this out of the box.

        In light of these tools, is it really neccessary to add additional build complexity?

        Show
        Lars Kühne added a comment - There are Eclipse plugins like eclipse-rbe that do the ascii conversion on the fly. Idea also supports this out of the box . In light of these tools, is it really neccessary to add additional build complexity?
        Hide
        Kan Ogawa added a comment -

        Lars,

        When doing checkout and diff i18n properties from svn repository, it is a big merit to read them directly without native2ascii tool.

        Also, if a simple translating work, translator doesn't need to always prepare jdk itself (including native2ascii) and it is possible to edit i18n properties easily and effectively on various editor (e.g. notepad, vim, and emacs, etc.).

        So I voted here.

        Show
        Kan Ogawa added a comment - Lars, When doing checkout and diff i18n properties from svn repository, it is a big merit to read them directly without native2ascii tool. Also, if a simple translating work, translator doesn't need to always prepare jdk itself (including native2ascii) and it is possible to edit i18n properties easily and effectively on various editor (e.g. notepad, vim, and emacs, etc.). So I voted here.
        Hide
        Shawn Jiang added a comment -

        Will the native2ascii work with harmony? If not directly do they have an equivalent solution?
        --------------------------------------
        I don't know, Harmony should have a bunch of build problems with Geronimo besides this. If Harmony does not have the equivalent solution. Maybe they should have a feature JIRA open to add this support.

        Anyway, I agree that if we need to support building geronimo with Harmony, we can't apply the patch of this JIRA for now.

        is it really neccessary to add additional build complexity?
        -------------------------------------
        As Kan said, It's really necessary for i18n translation contributors.

        Show
        Shawn Jiang added a comment - Will the native2ascii work with harmony? If not directly do they have an equivalent solution? -------------------------------------- I don't know, Harmony should have a bunch of build problems with Geronimo besides this. If Harmony does not have the equivalent solution. Maybe they should have a feature JIRA open to add this support. Anyway, I agree that if we need to support building geronimo with Harmony, we can't apply the patch of this JIRA for now. is it really neccessary to add additional build complexity? ------------------------------------- As Kan said, It's really necessary for i18n translation contributors.
        Hide
        David Jencks added a comment -

        I have no objection to using the plugin, I was mostly wondering if a harmony component could be used in the native2ascii plugin to remove the dependency on sun vm code.

        Show
        David Jencks added a comment - I have no objection to using the plugin, I was mostly wondering if a harmony component could be used in the native2ascii plugin to remove the dependency on sun vm code.
        Hide
        Jack Cai added a comment -

        AFAIK, there is no Harmony equivalent yet. A few JDK tools haven't been implemented in Harmony yet.

        Show
        Jack Cai added a comment - AFAIK, there is no Harmony equivalent yet. A few JDK tools haven't been implemented in Harmony yet.
        Hide
        Shawn Jiang added a comment -

        David,

        Can you advise can we apply this patch even when there's no harmony native2ascii component ?

        In my opinion, native2ascii is widely used in JAVA world. If we only use it only for build time. There's no big problem here. As for harmony + geronimo build, We can still workaround it by adding a external native2ascii dependency to pom file.

        Show
        Shawn Jiang added a comment - David, Can you advise can we apply this patch even when there's no harmony native2ascii component ? In my opinion, native2ascii is widely used in JAVA world. If we only use it only for build time. There's no big problem here. As for harmony + geronimo build, We can still workaround it by adding a external native2ascii dependency to pom file.
        Hide
        David Jencks added a comment -

        I have no objections to the patch.

        Show
        David Jencks added a comment - I have no objections to the patch.
        Hide
        Ivan added a comment -

        Commit the patch to trunk at rev 801552. Thanks !

        Show
        Ivan added a comment - Commit the patch to trunk at rev 801552. Thanks !
        Hide
        Jarek Gawor added a comment -

        Trunk builds on IBM SDK are broken by this change: http://people.apache.org/builds/geronimo/server/binaries/trunk/20090807/build-0300.log

        Looks like the native2ascii-maven-plugin-1.0-alpha-1 is missing a build profile for IBM SDK where it adds the tools.jar to its classpath (as it does for Sun JDKs). We need to fix the plugin or change our poms to directly use the Ant task and construct the classpath correctly.

        Show
        Jarek Gawor added a comment - Trunk builds on IBM SDK are broken by this change: http://people.apache.org/builds/geronimo/server/binaries/trunk/20090807/build-0300.log Looks like the native2ascii-maven-plugin-1.0-alpha-1 is missing a build profile for IBM SDK where it adds the tools.jar to its classpath (as it does for Sun JDKs). We need to fix the plugin or change our poms to directly use the Ant task and construct the classpath correctly.
        Hide
        Shawn Jiang added a comment -

        Here is the patch to fix the IBM SDK compilation problem. Thanks.

        Show
        Shawn Jiang added a comment - Here is the patch to fix the IBM SDK compilation problem. Thanks.
        Hide
        David Jencks added a comment -

        patch applied rev 802439, thanks shawn!

        Show
        David Jencks added a comment - patch applied rev 802439, thanks shawn!
        Hide
        David Jencks added a comment -

        This solution prevents building on mac os x where there is no tools.jar file at all.

        I susupect we could solve this with profiles for different jdks.

        Show
        David Jencks added a comment - This solution prevents building on mac os x where there is no tools.jar file at all. I susupect we could solve this with profiles for different jdks.
        Hide
        Shawn Jiang added a comment -

        I don't have a Mac at hand. Can you verify if there's a jar in mac os x JDK that conatins "sun.tools.native2ascii.*" ?

        If there's not, I don't think a simple profile can resolve this. Mac JDK user might need a copy of a jar that contains "sun.tools.native2ascii.*" to start the mac profile.

        Show
        Shawn Jiang added a comment - I don't have a Mac at hand. Can you verify if there's a jar in mac os x JDK that conatins "sun.tools.native2ascii.*" ? If there's not, I don't think a simple profile can resolve this. Mac JDK user might need a copy of a jar that contains "sun.tools.native2ascii.*" to start the mac profile.
        Hide
        David Jencks added a comment -

        The classes are the the classes.jar and the build worked fine before adding this dependency.

        I suspect that having a profile with a pluginmanagerment section that only adds the dependency will work. I'm not sure how to active the profile depending on the jdk.

        Show
        David Jencks added a comment - The classes are the the classes.jar and the build worked fine before adding this dependency. I suspect that having a profile with a pluginmanagerment section that only adds the dependency will work. I'm not sure how to active the profile depending on the jdk.
        Hide
        Jack Cai added a comment -

        A profile can also be activated bt determing the OS. This might be useful to deal with Mac OS.

        Show
        Jack Cai added a comment - A profile can also be activated bt determing the OS. This might be useful to deal with Mac OS.
        Hide
        David Jencks added a comment -

        I added a profile activated by the tools.jar file that includes the dependency. rev 803287. We'll see if it works – at least I think building on osx works.

        Show
        David Jencks added a comment - I added a profile activated by the tools.jar file that includes the dependency. rev 803287. We'll see if it works – at least I think building on osx works.
        Hide
        Jack Cai added a comment -

        I guess the jms-resource-providers.properties in console-base-portlets and activemq-portlets should remain in the "resources" folder instead of moving to i18n-resources.

        Show
        Jack Cai added a comment - I guess the jms-resource-providers.properties in console-base-portlets and activemq-portlets should remain in the "resources" folder instead of moving to i18n-resources.
        Hide
        Shawn Jiang added a comment -

        Maven does not support multiple activated profiles at the same time.

        Show
        Shawn Jiang added a comment - Maven does not support multiple activated profiles at the same time.
        Hide
        David Jencks added a comment -

        How do you know that maven doesn't support multiple activated profiles? this seems like a giant limitation and I couldn't find documentation saying this.

        Show
        David Jencks added a comment - How do you know that maven doesn't support multiple activated profiles? this seems like a giant limitation and I couldn't find documentation saying this.
        Hide
        Shawn Jiang added a comment -

        Yes, there's no document on this. I just searched the google and found this:

        http://n2.nabble.com/Activation-of-multiple-profiles-at-the-same-time-td2496510.html#none

        It's definitely a defect of maven.

        Show
        Shawn Jiang added a comment - Yes, there's no document on this. I just searched the google and found this: http://n2.nabble.com/Activation-of-multiple-profiles-at-the-same-time-td2496510.html#none It's definitely a defect of maven.
        Hide
        Shawn Jiang added a comment -

        Put the ibmjdk profile to pom.xml of plugins since we only depend on native2ascii plugin in there.

        It workaround the limitation of maven that it does not support multiple active profiles in the same pom file.

        Show
        Shawn Jiang added a comment - Put the ibmjdk profile to pom.xml of plugins since we only depend on native2ascii plugin in there. It workaround the limitation of maven that it does not support multiple active profiles in the same pom file.
        Hide
        Ivan added a comment -

        I have tried the patch of Shawn on the Linux platform, seems it works.
        David, could you please help to check it on you Mac machine ?
        Thanks !

        Show
        Ivan added a comment - I have tried the patch of Shawn on the Linux platform, seems it works. David, could you please help to check it on you Mac machine ? Thanks !
        Hide
        David Jencks added a comment -

        Patcch applied rev 803916. Worked fine on my mac. very clever! thanks!

        Show
        David Jencks added a comment - Patcch applied rev 803916. Worked fine on my mac. very clever! thanks!
        Hide
        Shawn Jiang added a comment -

        I filed a JIRA against maven here:

        http://jira.codehaus.org/browse/MNG-4300, Maven does not support multiple activted profiles in the same pom.xml at the same time.

        Hopefully Maven can fix this in the next release.

        Show
        Shawn Jiang added a comment - I filed a JIRA against maven here: http://jira.codehaus.org/browse/MNG-4300 , Maven does not support multiple activted profiles in the same pom.xml at the same time. Hopefully Maven can fix this in the next release.
        Hide
        Kan Ogawa added a comment -

        Ivan,

        English resource bundle files in the PlanCreator portlet aren't "*_en.properties". Please rename them.

        Please open the following link:
        https://svn.apache.org/repos/asf/geronimo/server/trunk/plugins/plancreator/plancreator-portlets/src/main/i18n-resources/

        Show
        Kan Ogawa added a comment - Ivan, English resource bundle files in the PlanCreator portlet aren't "*_en.properties". Please rename them. Please open the following link: https://svn.apache.org/repos/asf/geronimo/server/trunk/plugins/plancreator/plancreator-portlets/src/main/i18n-resources/
        Hide
        Jarek Gawor added a comment -

        I renamed the PlanCreator resource bundles (2.2: 805860, trunk: 805859). Thanks for reporting this!

        Show
        Jarek Gawor added a comment - I renamed the PlanCreator resource bundles (2.2: 805860, trunk: 805859). Thanks for reporting this!

          People

          • Assignee:
            Ivan
            Reporter:
            Shawn Jiang
          • Votes:
            1 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development