Chinese translation of core docs

• Key: HADOOP-4164
• Type: Wish
• Status: Resolved
• Resolution: Fixed
• Priority: Major
• Assignee: Xuebing Yan
• Reporter: Xuebing Yan
• Component/s: documentation
• Fix Version/s: 0.17.3

Description

There are many active Chinese users in Hadoop community, a Chinese translation of the core docs will be very helpful to Hadoop newbies and experienced Hadoop users as well. Our team(Distributed computing@Alibaba Search Center) have just worked out a translation of r0.17.x, now we're updating it to r0.18 and r0.19.

The translation covers all the files under xdocs directory in the repos. Thus, we will have a Chinese version for each main doc, and leave the API docs, Wiki, FAQ, Mailing Lists, Release Notes, All Changes as is.

Comments

[Nigel Daley]

This is not a 0.18.1 blocker.

[Doug Cutting]

This will be great to have! Thanks!

[Xuebing Yan]

HADOOP-4164.patch can be applied onto hadoop-0.17.x distribution, i.e. http://svn.apache.org/repos/asf/hadoop/core/tags/release-0.17.x/. after this patch being applied, there will be a cn subdir under release-0.17.x/src/docs/.
To build Chinese docs,
1 follow the README.BUILD in the cn subdir to download the fonts file needed for generating the pdf files,
2 change to release-0.17.x/src/docs/cn, run "forrest"

This patch has been tested through ant test-patch, the following are the command line and the output.

NOTE that all the 4 '@author' occurrence are in the XSLT files used by fop, where it means the 'author' attribute of an xml node, is a kind of XPath syntax, not the same meaning as it is in the the Java Docs. So I don't think it's a problem, or do you have any good idea to avoid this warning?

command line:

ant -Dpatch.file=../release-0.17.0/HADOOP-4164.patch -Dforrest.home=$FORREST_HOME -Dfindbugs.home=$FINDBUGS_HOME test-patch

output:

....

[exec] findbugs:
[exec] [mkdir] Created dir: /home/yanxb/dev/release-0.17.1/build/test/findbugs
[exec] [findbugs] Executing findbugs from ant task
[exec] [findbugs] Running FindBugs...
[exec] [findbugs] The following classes needed for analysis were missing:
[exec] [findbugs] org.apache.tools.ant.Task
[exec] [findbugs] org.apache.tools.ant.DirectoryScanner
[exec] [findbugs] org.apache.tools.ant.types.FileSet
[exec] [findbugs] org.apache.tools.ant.Project
[exec] [findbugs] org.apache.tools.ant.BuildException
[exec] [findbugs] Warnings generated: 219
[exec] [findbugs] Missing classes: 6
[exec] [findbugs] Classes needed for analysis were missing
[exec] [findbugs] Output saved to /home/yanxb/dev/release-0.17.1/build/test/findbugs/hadoop-findbugs-report.xml
[exec] [xslt] Processing /home/yanxb/dev/release-0.17.1/build/test/findbugs/hadoop-findbugs-report.xml to /home/yanxb/dev/release-0.17.1/build/test/findbugs/hadoop-findbugs-report.html
[exec] [xslt] Loading stylesheet /home/yanxb/findbugs-1.3.5/src/xsl/default.xsl
[exec]
[exec] BUILD SUCCESSFUL
[exec] Total time: 3 minutes 33 seconds
[exec] Starting with /home/yanxb/tmp/trunkFindbugsWarnings.xml
[exec] Merging /home/yanxb/tmp/patchFindbugsWarnings.xml
[exec]
[exec]
[exec]
[exec]
[exec] -1 overall.
[exec]
[exec] @author -1. The patch appears to contain 4 @author tags which the Hadoop community has agreed to not allow in code contributions.
[exec]
[exec] tests included +1. The patch appears to include 9 new or modified tests.
[exec]
[exec] javadoc +1. The javadoc tool did not generate any warning messages.
[exec]
[exec] javac +1. The applied patch does not generate any new javac compiler warnings.
[exec]
[exec] findbugs +1. The patch does not introduce any new Findbugs warnings.
[exec]
[exec]
[exec]
[exec]
[exec] ======================================================================
[exec] ======================================================================
[exec] Finished build.
[exec] ======================================================================
[exec] ======================================================================
[exec]
[exec]

BUILD FAILED
/home/yanxb/dev/release-0.17.1/build.xml:1135: exec returned: 1

Total time: 10 minutes 37 seconds

[Xuebing Yan]

made a little modification, now it passes through the test-patch completely.

[Owen O'Malley]

I tried this and it doesn't build for me. I get:

validate-xdocs:
/Users/oom/work/eclipse/hadoop-git/src/docs/cn/src/documentation/content/xdocs/index.xml:44:10: The content of element type "body" must match "(section|p|source|note|warning|fixme|table|ol|ul|dl|figure|anchor)+".

Maybe it is a problem with my locale. What is your locale set to when you run forrest? Which version of forrest were you running? I'm running forrest 0.8.

Also, can you please remove the link to alibaba.com? In Hadoop, we like to encourage community ownership of the project and so don't include author tags in our code or documentation.

[Xuebing Yan]

The problem was caused by the <br/> tag before the alibaba.com link paragraph.

I've removed the link to alibaba.com and the tag. Now it passes through the tests

ant -Dpatch.file=../HADOOP-4164.patch -Dforrest.home=$FORREST_HOME -Dfindbugs.home=$FINDBUGS_HOME test-patch

and

ant clean test javadoc checkstyle

successfully in my dev env where

JAVA_HOME=$HOME/dev/jdk1.5.0_16
FORREST_HOME=$HOME/dev/apache-forrest-0.8
FINDBUGS_HOME=$HOME/dev/findbugs-1.3.5
ANT_HOME=$HOME/dev/apache-ant-1.7.1

Please give another try, it should work for you.

[Edward J. Yoon]

Latest patch is good for me. +1

But IMO, The linking to each local community is better such as a "related project" category.
For example,

Related Project
 Hbase
 Mahout
--
Local Community
 China
 Japan
 Korea

[Hadoop QA]

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12392059/HADOOP-4164.patch
against trunk revision 704310.

+1 @author. The patch does not contain any @author tags.

+1 tests included. The patch appears to include 9 new or modified tests.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 javac. The applied patch does not increase the total number of javac compiler warnings.

+1 findbugs. The patch does not introduce any new Findbugs warnings.

+1 Eclipse classpath. The patch retains Eclipse classpath integrity.

-1 core tests. The patch failed core unit tests.

+1 contrib tests. The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3456/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3456/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3456/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3456/console

This message is automatically generated.

[Xuebing Yan]

This patch is only for hadoop-0.17.x, we have also finished 0.18 translation, and have 0.19 in progress. Once 0.17.x is pushed online, I will be more familiar with this procedure, and can start to make patch for 0.18.

[Edward J. Yoon]

BTW, how many chinese committers have we got? I think they should review this document.

[Tsz Wo (Nicholas), SZE]

> BTW, how many chinese committers have we got? I think they should review this document.

I would review this in these few days. 謝謝! (Thanks you!)

[Tsz Wo (Nicholas), SZE]

I followed the instructions in README.BUILD and built the Chinese doc successfully.

Question: The patch requires AR PL UMing fonts with Arphic Public License. Is it okay?

[Xuebing Yan]

I checked the license before I chose AR PL UMing fonts. The license says, "You may copy and distribute verbatim copies of this Font in any medium, without restriction, provided that you retain this license file (ARPHICPL.TXT) unaltered in all copies". I think it's just fine the way we use it.

[Tsz Wo (Nicholas), SZE]

Talked to Hairong and Owen about this. We should do the following:

• Suppose the font license is good. The build process should not require a fixed location (/usr/local/share/fopfonts/). We should put this directory under the trunk, probably trunk/src/doc/fonts/fopfonts.
• The Chinese doc should be built by "ant docs".

[Hairong Kuang]

After talking with Nicholas, we decided to split the task of reviewing the patch. I have finished reviewing the sections "Quickstart", "Cluster Setup", and "HDFS Architecture". One comment that's not related to the Chinese translation is that HDFS Architecture section needs to be updated. It still talks a lot of future features like permissions, rebalanicing, quotas, and periodic checkpoint that have already implemented.

Since I am familiar with the area of Chinese translation of the computer technique documents, I am not sure if the Chinese translation of some computer terminologies like "rack", "cluster", "replica", and "persistence" etc. are standard or not. It would be nice if those are the standard translations. My following comments are related to the accuracy of the understanding.

1. In "Hadoop Quickstart"->purpose: "run examples/simple jobs" should better be "运行示例程序或简单作业"；
2. In "Cluster setup">"Logging">"History Logging": "failed and killed tip details" should better be "失败和终止的任务细节". A tip is a map/reduce task.
3. In "HDFS Architecture"->"Introduction": "commodity hardware" should better be "通用硬件"。
4. In "HDFS Architecture"->"The file system namespace": user quota includes user space quota & disk quota.

[Xuebing Yan]

We always chose the mainland strandard computer terminologies during the translation, I will check it over and try to get a list of them. After all of the reviews have been done I will merge all the sugguestions into the next patch.

With regard to fonts location,

• I have tried to use a relative path earlier, but failed to find a resolution. Maybe we should consult guys who know forrest well to make sure this is possible.
• If we can use a relative location, we should incorporate the fonts file in the source, thus the source will be kind of heavier, because the fonts file itself is about 20M. This might be a problem.
• If we don't include the fonts file, there is still another problem. We should have all users to remember that before they compile the doc they should get the fonts file downloaded, put into a fixed directory. that's not too automated.

What do you think?

[Tsz Wo (Nicholas), SZE]

I have read Hadoop DFS User Guide, HDFS Permissions Guide and part of Map-Reduce Tutorial (till Job Input). Most of the doc look good. Here are two points I found:

• The term 数据块 is referring to both data block and data chunk. It may be confusing.

• "Speculative" should not be translated to 预防性. It should be something like 推測性.

BTW, I think this sentence about fsck is wrong in the English doc.

• Unlike a traditional fsck utility for native filesystems, this command does not correct the errors it detects.

[Owen O'Malley]

I think 20mb is ok for now. Let's go ahead and check it in and we can fix it later if someone comes up with a better solution. Nicholas, how did you get relative paths working? I think that is much better than absolute paths.

[Tsz Wo (Nicholas), SZE]

> Nicholas, how did you get relative paths working?

I had not use relative paths when I built the Chinese doc. I just have replaced all fopfonts path /usr/local/share/fopfonts/... to some other absolute path, so that I didn't need admin access to create a share directory.

Tried to find out how to make relative path works for Cocoon. However, the result is negative. It seems that this is a known missing feature. See

• http://mail-archives.apache.org/mod_mbox/forrest-dev/200808.mbox/[e-mail]%3E
• https://issues.apache.org/jira/browse/COCOON-1542

We could define a variable and replace it in runtime by a script, i.e.:

1. Define a variable, say FOPFONT_HOME, and use it in the xml and other source files.
2. In "ant docs", run a script to replace FOPFONT_HOME with something like `pwd`/src/docs/cn/fopfont/

BTW, I have uploaded the generated Chinese doc from the current patch to http://people.apache.org/~szetszwo/4164build/site/ for other reviewers' convenient.

[Xuebing Yan]

We found a workaround to the relative path problem. Since forrest will find all needed files in $FORREST_HOME/main/webapp, we can just copy the fonts related files into the webapp directory before we start to compile, remove them later. Although it's not so good that we must have write permission to $FORREST_HOME/main/webapp directory, it's pretty convenient right now as we wrap all those into an ant target cn-docs.

The new patch also fixed all the translation problems Hairong and Nicholas pointed out couples of days ago, except the "data block" one. We didn't find any other better words for "data block" in Chinese, fortunately, there is only three "trunk" occured in the core docs.

hdfs_design.xml: "Thus, an HDFS file is chopped up into 64 MB chunks, and if possible, each chunk will reside on a different Datanode."
mapred_tutorial.xml: "independent chunks which are processed by the <em>map tasks</em>"

The patch passed both

ant -Dpatch.file=~/dev/HADOOP-4164.patch -Dforrest.home=$FORREST_HOME -Dfindbugs.home=$FINDBUGS_HOME test-patch

and

ant clean test javadoc checkstyle

successfully on my dev-box.

[Xuebing Yan]

I'm sorry, but I found that the last patch does not work for ant cn-docs, because 'svn diff' did not have uming.ttc(a binary file) included into the differences by default.

I've created a new patch, which works well for:

• ant -Dpatch.file=~/Devel/hadoop/HADOOP-4164.patch -Dforrest.home=$FORREST_HOME -Dfindbugs.home=$FINDBUGS_HOME test-patch
• ant docs -Dforrest.home=$FORREST_HOME (Chinese docs can be made by 'ant docs' in this version)
• ant clean
• ant clean test javadoc checkstyle

Since the original patch is 23M, far larger than 10M, I just 7zipped it to make it below the upload limit.

[Tsz Wo (Nicholas), SZE]

Created HADOOP-4723 for the English doc problems. Will check the new patch soon.

[Tsz Wo (Nicholas), SZE]

Since forrest will find all needed files in $FORREST_HOME/main/webapp, we can just copy the fonts related files into the webapp directory before we start to compile, remove them later. Although it's not so good that we must have write permission to $FORREST_HOME/main/webapp directory, it's pretty convenient right now as we wrap all those into an ant target cn-docs.

Requiring w permission to $FORREST_HOME/main/webapp sounds a little bit too much. Owen, what do you think?

[Tsz Wo (Nicholas), SZE]

Failed to build HADOOP-4164.patch.7z this time.

1. Should use Java 5. Otherwise, build failed.
2. "ant clean" should not remove the generated cn doc.
3. After fixed build.xml, successfully built the doc but all the Chinese characters in the generated doc became '?'. This is what I got: http://people.apache.org/~szetszwo/4164_20081126/cn/

build.xml.patch: this is the fixes for 1. and 2. The ant command should be

 ant docs -Dforrest.home=$FORREST_HOME -Djava5.home=$JAVA5

[Xuebing Yan]

Nicholas, I reproduced the problem by changing my locale settings from zh_CN.utf8 to either POSIX or en_US.iso85991.

So, I made a new patch, which will set locale to en_US.utf8 before forrest invoked:

   <exec dir="${cndocs.src}" executable="${forrest.home}/bin/forrest" failonerror="true">
      <env key="JAVA_HOME" value="${java5.home}"/>
      <env key="LANG" value="en_US.utf8"/>
    </exec>

Now it's okay in three locale settings: zh_CN.utf8, POSIX and en_US.iso85991 on my dev-box, hope it works for you.

[Tsz Wo (Nicholas), SZE]

I have successfully built the latest patch: http://people.apache.org/~szetszwo/4164_20081202/

I was wrong about my previous comment. Since this patch is going to 0.17, we don't need -Djava5.home=$JAVA5 in the ant command. Could you remove the java5.home related stuffs from build.xml, Xuebing? I am sorry about that.

[Xuebing Yan]

So glad that it finally works.
Java5 stuff is removed, please have another try.

[Tsz Wo (Nicholas), SZE]

BUILD SUCCESSFUL: http://people.apache.org/~szetszwo/4164_20081203/

If the workaround to the relative path problem is fine, we could commit this.

[Owen O'Malley]

We need to make this patch apply to trunk first and maybe 18 and 19. I can't see us making any more 17 releases. Then we can make much smaller patches to update the documentation from 17 to trunk.

I also don't think it is reasonable to force installation of the fonts in the forrest installation. Could we write the path name (inside the hadoop source) to the xml file as part of the build? Does that make sense? Look at how we invoke src/saveVersion from build.xml to generate a file in the build directory that we then compile.

[Xuebing Yan]

The way like what save/saveVersion does sounds great, I will try to work a new patch out based on trunk.

I have all versions of Chinese translations prior to 0.19, should we just publish them at at http://hadoop.apache.org/core/docs/, like r0.17.0.cn or something else? If it's worth we doing that, what should I do to make it happen?

[Zheng Shao]

@Xuebing: If it's not easy to set that up to hadoop.apache.org, let me know and I can help you make it happen on http://hadooper.cn/ The website is hosted by Chinese Academy of Science.

[Owen O'Malley]

Uploading it to the Apache site is not a problem. Yes, we can include it in http://hadoop.apache.org/core/docs/r0.17.2/. I'm sorry that this has gone through so many iterations, but clearly this has been a learning experience, since no other translations have been done yet. I think it is very exciting that the Hadoop documentation was translated into Chinese, and I'm looking forward to getting it published.

[Doug Cutting]

It would be better to host the translation on hadoop.apache.org, so it is clear that it is the official translation. We should treat these translation contributions much like code contributions, as a route to becoming a committer. Once we commit the first version, and then you submit more high-quality patches updating and improving them, then we can add you as a committer to maintain this documentation at hadoop.apache.org as the official version that would be included in Hadoop releases, etc.

[Zheng Shao]

@Owen. What would be the next steps to get this published on http://hadoop.apache.org/core/docs/r0.17.2/ ?

[Xuebing Yan]

Owen, I've made two patches for 0.17.2 and 0.18.2. As you know, Chinese docs can be made this way:

7z e HADOOP-0.18.2.patch.7z
svn co http://svn.apache.org/repos/asf/hadoop/core/tags/release-0.18.2 release-0.18.2
cd release-0.18.2
patch -p0 < HADOOP-0.18.2.patch
ant docs -Dforrest.home=$FORREST_HOME

then you will see Chinese docs under docs/cn, please get them published, thanks.

Doug Cutting, I'm going to make a trunk-based patch, in which the main Chinese docs will be the same as 0.18.2 at first. After that patch have been applied, I can submit patches to make it update to date. Am I right? And which one should I based on, trunk or branch-0.19?

Thanks Zheng, I agree that it's better we host them on hadoop.apache.org first, and I believe Owen will publish them soon after seeing those patches.

[Doug Cutting]

> After that patch have been applied, I can submit patches to make it update to date. Am I right?

Yes, that sounds good. Thanks!

[Xuebing Yan]

HADOOP-r724961.patch.7z is the one for trunk, although it's made from revision 724961, it can be applied to any post revision if there is no modification to build.xml.
Chinese docs in this patch remains as it is in HADOOP-0.18.2.patch for now, updates will be submitted soon.

[Tsz Wo (Nicholas), SZE]

Tried HADOOP-r724961.patch.7z. Need to add <env key="JAVA_HOME" value="${java5.home}"/> in build.xml. Then, build successful with current trunk: http://people.apache.org/~szetszwo/4164_20081210/