Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-14604

Fix the bug of aggregation log have one replication in HDFS Federation environment

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Duplicate
    • Affects Version/s: 2.6.0
    • Fix Version/s: 2.9.0, 2.7.4, 3.0.0-alpha4, 2.8.2
    • Component/s: fs, viewfs
    • Labels:
      None
    • Environment:

      Hadoop 2.6.0 with HDFS Federation

    • Tags:
      HDFS Federation, replication, aggregation log

      Description

      With HDFS Federation configured, the aggregation log have a default replication count of 1, rather than the value of dfs.replication in hdfs-site.xml configuration file.

      Before:

      [op@NM-304-SA5212M4-BIGDATA-640 ~]$ hdfs dfs -ls /yarn/apps/op/logs/application_1498544648312_0001
      Found 3 items
      -rw-r-----   1 op hadoop       8270 2017-06-27 14:25 /yarn/apps/op/logs/application_1498544648312_0001/NM-304-SA5212M4-BIGDATA-645_8041
      -rw-r-----   1 op hadoop      54469 2017-06-27 14:25 /yarn/apps/op/logs/application_1498544648312_0001/NM-304-SA5212M4-BIGDATA-646_8041
      -rw-r-----   1 op hadoop       6537 2017-06-27 14:25 /yarn/apps/op/logs/application_1498544648312_0001/NM-304-SA5212M4-BIGDATA-672_8041
      

      The aggregation log have only 1 replication in default. But the value of dfs.replication in hdfs-site.xml configuration file is 3

      After apply this patch:

      [op@NM-304-SA5212M4-BIGDATA-640 ~]$ hdfs dfs -ls /yarn/apps/op/logs/application_1498635214020_0001
      Found 3 items
      -rw-r-----   3 op hadoop       5729 2017-06-28 15:34 /yarn/apps/op/logs/application_1498635214020_0001/NM-304-SA5212M4-BIGDATA-645_8041
      -rw-r-----   3 op hadoop      38439 2017-06-28 15:34 /yarn/apps/op/logs/application_1498635214020_0001/NM-304-SA5212M4-BIGDATA-646_8041
      -rw-r-----   3 op hadoop       8270 2017-06-28 15:34 /yarn/apps/op/logs/application_1498635214020_0001/NM-304-SA5212M4-BIGDATA-671_8041
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                doris Doris Gu
                Reporter:
                clouding dingguotao
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: