Issue 83349

Summary: Patch to Add Myanmar locale support
Product: Internationalization Reporter: devel
Component: i18npoolAssignee: stefan.baltzer
Status: CLOSED FIXED QA Contact: issues <issues.openoffice.org>
Severity: trivial    
Priority: P3 CC: issues, wunnakoko
Version: current   
Target Milestone: ---   
Hardware: All   
OS: All   
URL: http://www.thanlwinsoft.org/ThanLwinSoft/Downloads/OpenOffice/myLocale-20071103.diff
Issue Type: PATCH Latest Confirmation on: ---
Developer Difficulty: ---
Attachments:
Description Flags
myLocale-20071103.diff
none
myLocale-20071214.diff
none
Locale Patch
none
myLocale-20080206.diff against OOH680_m5
none
myLocale-icu-20080206.diff for Myanmar in ICU
none
Minor changes to myLocale-20080206.diff
none
New patch with only one currency element.
none
updated patch for dev300_m21
none
patch for DEV300_m31
none
data for i18npool/source/collator/data/
none
Changes collator and defaults to natural numbers none

Description devel 2007-11-05 20:18:31 UTC
The patch at
http://www.thanlwinsoft.org/ThanLwinSoft/Downloads/OpenOffice/myLocale-20071103.diff
is against OOG680_m7 (2.3.0-7). This enables Myanmar to be chosen as a locale
with a CTL font. 
It also provides Myanmar dates/currency in Calc. 

Myanmar sorting is also possible, provided you have ICU 3.6 with a Myanmar
locale - not included in this patch. I will update my ICU Myanmar patches
shortly. If you are have a Myanmar enabled ICU on the system and you build OOo
to use it, then OOo will provide correct sorting in a Myanmar locale.

I've tested this patch on Ubuntu gutsy built against system libraries. Does any
one else in this project have an OOo build working? If so, please can you test
this patch. Some of the locale data may well need changing e.g. choice of date
formats, separators etc.
Comment 1 devel 2007-11-05 20:20:42 UTC
Created attachment 49441 [details]
myLocale-20071103.diff
Comment 2 stx123 2007-11-08 18:39:14 UTC
I guess the issue is more appropriate for the l10n project. Eike, would you have
a look?
Comment 3 devel 2007-12-14 11:47:47 UTC
Created attachment 50314 [details]
myLocale-20071214.diff
Comment 4 devel 2007-12-14 11:52:34 UTC
The myLocale-20071214.diff patch corrects an error in the
defaultnumberingprovider and has an updated my_MM.xml file.

My previous comment referred to a Myanmar ICU locale. I have an ICU patch to
provide this at http://www.thanlwinsoft.org/ThanLwinSoft/Downloads/#collate but
it requires ICU to be built from SVN rather than a release build. It seems ICU
is unlikely to accept Myanmar as a locale until Myanmar is submitted to the CLDR.
Comment 5 ooo 2008-01-28 10:43:21 UTC
Reassigning to spare time account, considering for OOo3.0
Comment 6 wunnakoko 2008-01-30 10:26:19 UTC
Created attachment 51256 [details]
Locale Patch
Comment 7 erack 2008-02-01 09:52:02 UTC
@kstribley: Keith, please note that to integrate code or data contributed we
need a signed Joint Copyright Assignment form (JCA) filled-out, see
http://contributing.openoffice.org/programming.html#jca

@wunnakoko: Thanks for the updated patch, but I'd like some comments on what it
changes and why. I didn't browse through yet. Same for you: to include
contributions we need the JCA signed.

Thanks
  Eike
Comment 8 wunnakoko 2008-02-06 04:50:41 UTC
Eike: I have already sent the JCA by mail and also a scanned copy to the email.

The difference between myLocale-20080130.diff, which I submit, and the one
mylocale-20071214.diff, which Keith submitted are that 
-I deleted the FEC as currency assignment
-Burmese translation of quarter 1,2,3, approved by Myanmar Language Commission.
Comment 9 devel 2008-02-06 15:27:27 UTC
Created attachment 51402 [details]
myLocale-20080206.diff against OOH680_m5
Comment 10 devel 2008-02-06 15:39:17 UTC
The latest patch was against OOH680_m5 and was built on Windows XP. This showed
up some omissions when --with-lang=my_MM is used, so this patch now includes
extra changes to:

setup_native/source/win32/msi-encodinglist.txt
solenv/bin/modules/installer/download.pm
solenv/inc/postset.mk
scp2/source/ooo/file_ooo.scp
scp2/source/ooo/module_langpack.ulf
scp2/source/ooo/module_langpack.scp
i18npool/source/isolang/mslangid.cxx

I've incorporated WunnaKoKo's changes, though I've left FEC just commented it
out in the XML. FEC are still used in Myanmar (see e.g.
http://www.commerce.gov.mm/pdf/mirt/bulletin_201107.pdf 
page 6), though it is no longer recognized as an international currency.

@erack - Eike, I've emailed off a scan of the signed JCA.

@wunnakoko - please can you use UTF-8 for patches, not UTF-16, it is unlikely to
apply cleanly if you change the encoding.

thanks,
Keith
Comment 11 devel 2008-02-06 15:54:32 UTC
Created attachment 51404 [details]
myLocale-icu-20080206.diff for Myanmar in ICU
Comment 12 devel 2008-02-06 16:01:14 UTC
myLocale-icu-20080206.diff needs to be applied in addition to myLocale to add
Myanmar support to ICU. The actual data needs to be downloaded from 
http://www.thanlwinsoft.org/ThanLwinSoft/Downloads/OpenOffice/icudt36l.dat.bz2
and extracted into the ICU directory. You may need to rebuild from scratch if
your platform uses a different Endianness or version of ICU.
(icudt36l.dat is too big to attach to this issue)
Comment 13 wunnakoko 2008-02-08 10:04:34 UTC
Created attachment 51432 [details]
Minor changes to myLocale-20080206.diff
Comment 14 wunnakoko 2008-02-08 10:07:57 UTC
Minor spelling checking to Keith's myLocale-20080206.diff
Comment 15 erack 2008-02-09 14:57:42 UTC
There seems to be some confusion about the Currency elements. Currently the
BankSymbol (as well as the CurrencyID, which isn't the case yet everywhere, I'm
about to change that) have to be the ISO 4217 code and have to be unique within
a locale data file. There are two Currency elements with BankSymbol MMK, that
doesn't work. A decision has to be made which one to use, and it should reflect
the officially used CurrencySymbol, which according to
http://en.wikipedia.org/wiki/Kyat is 'K', not 'Ks'.

Regarding the ICU patch: I don't want to integrate that. Actually we want to get
rid of all patches applied to ICU, see
http://wiki.services.openoffice.org/wiki/ICU/bugs_and_patches, and for OOo3.0 we
hope to be able to upgrade to ICU 4.0 anyway. That should have the changes, will it?

Btw, patches to external projects that have their own patch file like
icu-3.6.patch should not be submitted as patches to the patch file, these likely
will not apply in another revision as the regenerated patch file may be
different, not that much of a problem in this case here as the patch almost only
adds new files to the patch file. Such patches should be created as diffs
against the unpacked and patched source as present after build in
<outputdir>/misc/build/... instead.

Thanks
  Eike
Comment 16 wunnakoko 2008-02-13 04:13:43 UTC
Created attachment 51486 [details]
New patch with only one currency element.
Comment 17 devel 2008-02-15 09:49:20 UTC
It is unfortunate that we are only allowed one currency. If Myanmar digits are
used, then it is normal to write Kyats in Burmese. If Arabic digits are used,
then Ks is normally used. Using K only is much rarer, so if we have to
standardise on one, then I would prefer Ks, since this would match users
expectations much more closely.
Comment 18 devel 2008-02-15 09:57:35 UTC
ICU 4 will probably not include Myanmar, because they want to wait until Myanmar
is included in the CLDR. We weren't able to submit Myanmar last year, because
Unicode 5.1 was not released even in Beta. Hopefully, it can be submitted when
the CLDR next opens for changes next month, but that may take a year to be
accepted I guess. 

Hopefully, ICU 4 will at least include the Unicode 5.1 changes.
Comment 19 erack 2008-02-28 13:14:10 UTC
Sorry, I don't have time anymore to get this in before UI freeze, it touches too
many modules that are not already part of my CWS and I have to get that ready
for QA. Rescheduling to OOo3.1
Comment 20 devel 2008-08-17 05:22:53 UTC
Created attachment 55803 [details]
updated patch for dev300_m21
Comment 21 devel 2008-09-11 15:33:27 UTC
Created attachment 56420 [details]
patch for DEV300_m31
Comment 22 devel 2008-09-11 15:40:52 UTC
The last attachment patches some additional files as suggested by Javier Sola.

binfilter/bf_svtools/source/config/svt_ctloptions.cxx
cppcanvas/source/mtfrenderer/implrenderer.cxx
svtools/source/config/ctloptions.cxx
svx/source/dialog/optgdlg.cxx
wizards/com/sun/star/wizards/letter/LocaleCodes.java

It also includes a currency fix when using natural numbers so that the currency
is displayed in Myanmar script as well rather than using K.

The collation data still isn't included in the patch, so ICU 4 still needs to be
patched with the my collation data from the CLDR.
Comment 23 erack 2008-09-13 14:14:01 UTC
Thanks. Note that patching files in module binfilter doesn't make sense here,
these are used only to store/load legacy SO5.2 binary file format. Good that
files in svtools and svx were touched, as this shows that those places still do
things on their own instead of using appropriate i18n methods, continuing with
wrong assumptions like Vietnamese being a CTL language..

For the collation I hope we will be able to have a tailored collation in module
i18npool instead of having to patch ICU. We want to get rid of ICU patches
instead of introducing new ones..
Comment 24 erack 2008-09-16 21:04:18 UTC
In locale data there were some defaults duplicated, we can have only one default
format code per usage and type:

(those errors are now detected by LocaleNode.cxx when compiling the locale data)

Error: Duplicated default for usage="FIXED_NUMBER" type="medium": formatindex="50".
Error: Duplicated default for usage="FIXED_NUMBER" type="short": formatindex="51".
Error: Duplicated default for usage="DATE" type="medium": formatindex="60".
Error: Duplicated default for usage="DATE" type="short": formatindex="61".

Those were NatNum1 format codes, I changed them to non-default.

Also reordered FormatElements by usage and formatindex for better overview,
similar to other locale data. In case you need to update the data please submit
diffs against the revision checked in to cvs.
Comment 25 erack 2008-09-16 21:36:15 UTC
In cws locales31:

offapi/com/sun/star/style/NumberingType.idl  1.18.128.1
i18npool/inc/i18npool/mslangid.hxx  1.5.24.1
i18npool/source/defaultnumberingprovider/defaultnumberingprovider.cxx  1.30.24.1
i18npool/source/isolang/mslangid.cxx  1.10.24.1
i18npool/source/localedata/data/Attic/my_MM.xml  1.1.2.2
i18npool/source/localedata/data/localedata_others.map  1.17.28.1
i18npool/source/localedata/data/makefile.mk  1.47.24.1
i18npool/source/localedata/localedata.cxx  1.59.24.1
i18npool/source/localedata/LocaleNode.cxx  1.29.24.1
i18npool/source/localedata/LocaleNode.hxx  1.14.24.1
officecfg/registry/data/org/openoffice/VCL.xcu  1.62.116.1
scp2/source/ooo/file_ooo.scp  1.257.18.1
scp2/source/ooo/module_langpack.ulf  1.10.178.1
solenv/bin/modules/installer/globals.pm  1.103.16.1
svtools/source/config/ctloptions.cxx  1.18.140.1
svx/source/dialog/optgdlg.cxx  1.53.20.1
vcl/source/gdi/outdev3.cxx  1.242.10.1
vcl/source/gdi/sallayout.cxx  1.94.90.1
cppcanvas/source/mtfrenderer/implrenderer.cxx  1.25.6.1
wizards/com/sun/star/wizards/letter/LocaleCodes.java  1.7.46.1

Note that I omitted the change to tools/source/rc/resmgr.cxx as the
ResMgr::GetLang() method is only kept for legacy SDK compatibility, the
ambiguous phone code numbers for resource bundles aren't used anymore.

svtools/source/config/ctloptions.cxx and svx/source/dialog/optgdlg.cxx now use
the new MsLangId::needsSequenceChecking() method, so in future no changes are
necessary there, but only in i18npool/source/isolang/mslangid.cxx
Comment 26 devel 2008-09-17 17:53:34 UTC
Great thanks for committing that.
I've now tested using the collation rules in i18npool/source/collator/data/ in
which case the entry my_MM.xml for the collator needs to be changed to:

<Collator unoid="dictionary" default="true" />

and it needs an additional patch. Do you want this in a separate issue? I'll
attach it here for now.
Comment 27 devel 2008-09-17 17:57:16 UTC
Created attachment 56591 [details]
data for i18npool/source/collator/data/
Comment 28 devel 2008-09-19 08:58:56 UTC
Thanks for questioning the choice of defaults. I had been concentrating on
getting the choices right and overlooked that.
It would be better to default to Myanmar digits for Number, Currency, Date and
Time. The more technical formats can stay as they are. For some reason, setting
the medium FIXED_NUMBER [NatNum1]General to default didn't seem to take effect,
though the others do.

The attached patch (to follow) fixes that and includes the change of collator
from ICU's to the custom collator that the previous patch puts in
i18npool/source/collator/data/my_dictionary.txt. It needs to be applied in
addition to myCollator-i83349.diff.
I used the term dictionary, because the collation follows that used in the
Myanmar Language Commission Dictionary and has been described by John Okell as
dictionary order (Burmese: an introduction to the script, Northern Illinois
University, 1994). Other collations are possible based on Pali for example.
Comment 29 devel 2008-09-19 09:03:16 UTC
Created attachment 56631 [details]
Changes collator and defaults to natural numbers
Comment 30 erack 2008-09-19 15:35:11 UTC
Thanks, and yes, reopening this issue for that purpose is fine. I'll continue to
work on this after vacation.
Comment 31 erack 2008-10-03 21:06:47 UTC
I tried to add the collator data, but got only a "Rule parsering error" from the
gencoll_rule utility, which means the ICU RuleBasedCollator didn't munge the
data provided. May I assume you used ICU 4.0 with that data? I'll wait for
integration of CWS i18n42, which will upgrade our ICU 3.6 to ICU 4.0, and see if
that will do.

Updated locale data
i18npool/source/localedata/data/Attic/my_MM.xml  1.1.2.3
but omitted the Collator dictionary change for now, has to be changed later when
collator data works.

Yes, [NatNum1]General currently doesn't work, that may be related to issue 53633.
Comment 32 erack 2008-10-04 12:05:03 UTC
Implemented handling of [NatNum1]General as primary format. Actually that was
not related to issue 53633.

svtools/source/numbers/zforlist.cxx  1.72.54.1
svtools/source/numbers/zformat.cxx  1.78.138.1
svtools/source/numbers/zforscan.cxx  1.49.140.1
svx/inc/svx/numfmtsh.hxx  1.3.242.1
svx/source/dialog/numfmt.cxx  1.32.128.1
svx/source/items/numfmtsh.cxx  1.20.128.1
i18npool/source/localedata/data/Attic/my_MM.xml  1.1.2.4
i18npool/source/localedata/data/locale.dtd  1.24.34.1

In my_MM.xml format codes I moved the [NatNum1] modifiers from the additional
formats to the primary formats, so all NatNum formats are now nicely grouped on
top of the list in the number formatter dialog. Also moved the default attribute
for the type="short" format.
Comment 33 devel 2008-10-06 05:40:07 UTC
Thanks very much for fixing the NatNum issues. 
Yes, I was building against ICU 4.0, so that might explain the build problems
for collation.
Comment 34 erack 2008-11-25 01:08:43 UTC
After having migrated CWS locales31 to SVN and rebased to m35, ICU 4.0 is in
use. Applied the remaining parts of the patch for the collator without problem:

revision 264270
i18npool/source/collator/data/collator_data.map
i18npool/source/collator/data/my_dictionary.txt
i18npool/source/localedata/data/my_MM.xml
Comment 35 ooo 2008-12-11 15:03:52 UTC
Reassigning to QA for verification.
Comment 36 stefan.baltzer 2008-12-17 13:35:40 UTC
Verified in CWS locales31
Comment 37 thorsten.ziehm 2010-02-22 14:58:29 UTC
This issue is closed automatically. It should be fixed in a version with is
available for longer than half a year (OOo 3.1). If you think this issue isn't
fixed in the current version (OOo 3.2) please reopen it. But then please pay
attention about the field 'target milestone'.
The closure was approved by the Release Status Meeting at 22nd of February 2010
and it is based on the issue handling guideline for fixed/verified issues  :
http://wiki.services.openoffice.org/wiki/Handle_fixed_verified_issues