[HADOOP-14604] Fix the bug of aggregation log have one replication in HDFS Federation environment

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 2.6.0
• Fix Version/s: 2.9.0, 2.7.4, 3.0.0-alpha4, 2.8.2
• Component/s: fs, viewfs

Environment

Hadoop 2.6.0 with HDFS Federation

• Tags: HDFS Federation, replication, aggregation log
• Assignee: Doris Gu
• Reporter: dingguotao
• Created: 28/Jun/17 10:24
• Updated: 06/Dec/18 13:13
• Resolved: 06/Sep/18 03:03

Description

With HDFS Federation configured, the aggregation log have a default replication count of 1, rather than the value of dfs.replication in hdfs-site.xml configuration file.

Before:

[op@NM-304-SA5212M4-BIGDATA-640 ~]$ hdfs dfs -ls /yarn/apps/op/logs/application_1498544648312_0001
Found 3 items
-rw-r-----   1 op hadoop       8270 2017-06-27 14:25 /yarn/apps/op/logs/application_1498544648312_0001/NM-304-SA5212M4-BIGDATA-645_8041
-rw-r-----   1 op hadoop      54469 2017-06-27 14:25 /yarn/apps/op/logs/application_1498544648312_0001/NM-304-SA5212M4-BIGDATA-646_8041
-rw-r-----   1 op hadoop       6537 2017-06-27 14:25 /yarn/apps/op/logs/application_1498544648312_0001/NM-304-SA5212M4-BIGDATA-672_8041

The aggregation log have only 1 replication in default. But the value of dfs.replication in hdfs-site.xml configuration file is 3

After apply this patch:

[op@NM-304-SA5212M4-BIGDATA-640 ~]$ hdfs dfs -ls /yarn/apps/op/logs/application_1498635214020_0001
Found 3 items
-rw-r-----   3 op hadoop       5729 2017-06-28 15:34 /yarn/apps/op/logs/application_1498635214020_0001/NM-304-SA5212M4-BIGDATA-645_8041
-rw-r-----   3 op hadoop      38439 2017-06-28 15:34 /yarn/apps/op/logs/application_1498635214020_0001/NM-304-SA5212M4-BIGDATA-646_8041
-rw-r-----   3 op hadoop       8270 2017-06-28 15:34 /yarn/apps/op/logs/application_1498635214020_0001/NM-304-SA5212M4-BIGDATA-671_8041

Issue Links

• duplicates: HADOOP-9631 ViewFs should use underlying FileSystem's server side defaults