Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-11268

Correctly reconstruct erasure coding file from FSImage

    Details

      Description

      Currently, FSImageFormatProtoBuf has the information about whether the file is striped or not and saved file's erasure coding policy ID in replication field. But later, when FSImage is loaded to create the name space, the default system erasure coding policy is used to reconstruct file's block structure. In case if the erasure coding policy of file is not the default erasure coding policy, the content of the file cannot be accessed correctly.

      1. HDFS-11268-001.patch
        7 kB
        SammiChen
      2. HDFS-11268-002.patch
        7 kB
        SammiChen

        Activity

        Hide
        drankye Kai Zheng added a comment -

        This was exposed by HDFS-11152. SammiChen, do you find a good place to store the policy ID? This should be a must since we have various erasure coding policies other than just the default one.

        FSImage only has the information about whether the file is striped or not.

        Not checking the codes yet, but what's the type used to store the 'boolean' info regarding is striped or not?

        Show
        drankye Kai Zheng added a comment - This was exposed by HDFS-11152 . SammiChen , do you find a good place to store the policy ID? This should be a must since we have various erasure coding policies other than just the default one. FSImage only has the information about whether the file is striped or not. Not checking the codes yet, but what's the type used to store the 'boolean' info regarding is striped or not ?
        Hide
        Sammi SammiChen added a comment - - edited

        You're right! there is bool isStriped field in INodeFile message. I plans to add a new field int32 erasureCodingPolicyID field in INodeFile message, just like the way storage policy ID is save with uint32 storagePolicyID.

        Show
        Sammi SammiChen added a comment - - edited You're right! there is bool isStriped field in INodeFile message. I plans to add a new field int32 erasureCodingPolicyID field in INodeFile message, just like the way storage policy ID is save with uint32 storagePolicyID .
        Hide
        arpitagarwal Arpit Agarwal added a comment -

        In case if the ec policy on file is not the default ec policy, then the content of the file cannot be accessed correctly in this case.

        If this is true it sounds like a bug and not an improvement.

        Show
        arpitagarwal Arpit Agarwal added a comment - In case if the ec policy on file is not the default ec policy, then the content of the file cannot be accessed correctly in this case. If this is true it sounds like a bug and not an improvement.
        Hide
        jojochuang Wei-Chiu Chuang added a comment -

        I thought EC policy ID is stored using replication factor field, no?

        Show
        jojochuang Wei-Chiu Chuang added a comment - I thought EC policy ID is stored using replication factor field, no?
        Hide
        andrew.wang Andrew Wang added a comment -

        Hi SammiChen have you made any progress on this issue? I'm changing this to Bug and Critical, since it seems important for EC users.

        Show
        andrew.wang Andrew Wang added a comment - Hi SammiChen have you made any progress on this issue? I'm changing this to Bug and Critical, since it seems important for EC users.
        Hide
        Sammi SammiChen added a comment -

        Hi Andrew Wang, I'am working on it. Will give an update soon.

        Show
        Sammi SammiChen added a comment - Hi Andrew Wang , I'am working on it. Will give an update soon.
        Hide
        Sammi SammiChen added a comment -

        Initial patch

        Show
        Sammi SammiChen added a comment - Initial patch
        Hide
        Sammi SammiChen added a comment -

        Hi Wei-Chiu Chuang], you're right, EC policy ID is stored using replication factor field. After I did more investigation, It turns out that the problem is the EC policy ID loading process from the FsImage file. I updated the JIRA description accordingly.

        Show
        Sammi SammiChen added a comment - Hi Wei-Chiu Chuang ], you're right, EC policy ID is stored using replication factor field. After I did more investigation, It turns out that the problem is the EC policy ID loading process from the FsImage file. I updated the JIRA description accordingly.
        Hide
        hadoopqa Hadoop QA added a comment -
        +1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 16s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 12m 32s trunk passed
        +1 compile 0m 45s trunk passed
        +1 checkstyle 0m 26s trunk passed
        +1 mvnsite 0m 49s trunk passed
        +1 mvneclipse 0m 13s trunk passed
        +1 findbugs 1m 41s trunk passed
        +1 javadoc 0m 39s trunk passed
        +1 mvninstall 0m 45s the patch passed
        +1 compile 0m 43s the patch passed
        +1 javac 0m 43s the patch passed
        -0 checkstyle 0m 23s hadoop-hdfs-project/hadoop-hdfs: The patch generated 3 new + 25 unchanged - 2 fixed = 28 total (was 27)
        +1 mvnsite 0m 48s the patch passed
        +1 mvneclipse 0m 11s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 1m 46s the patch passed
        +1 javadoc 0m 38s the patch passed
        +1 unit 92m 40s hadoop-hdfs in the patch passed.
        +1 asflicense 0m 22s The patch does not generate ASF License warnings.
        116m 49s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:a9ad5d6
        JIRA Issue HDFS-11268
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12847610/HDFS-11268-001.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux eeb52ac147c9 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29 20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 2f8e9b7
        Default Java 1.8.0_111
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/18181/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18181/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18181/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 16s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 12m 32s trunk passed +1 compile 0m 45s trunk passed +1 checkstyle 0m 26s trunk passed +1 mvnsite 0m 49s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 1m 41s trunk passed +1 javadoc 0m 39s trunk passed +1 mvninstall 0m 45s the patch passed +1 compile 0m 43s the patch passed +1 javac 0m 43s the patch passed -0 checkstyle 0m 23s hadoop-hdfs-project/hadoop-hdfs: The patch generated 3 new + 25 unchanged - 2 fixed = 28 total (was 27) +1 mvnsite 0m 48s the patch passed +1 mvneclipse 0m 11s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 46s the patch passed +1 javadoc 0m 38s the patch passed +1 unit 92m 40s hadoop-hdfs in the patch passed. +1 asflicense 0m 22s The patch does not generate ASF License warnings. 116m 49s Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HDFS-11268 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12847610/HDFS-11268-001.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux eeb52ac147c9 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29 20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 2f8e9b7 Default Java 1.8.0_111 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/18181/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18181/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18181/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        andrew.wang Andrew Wang added a comment -

        Fix and unit test LGTM thanks Sammi, but looks like we need a small update since I just committed HDFS-10759 for Ewan. +1 pending that and a fresh Jenkins run.

        Show
        andrew.wang Andrew Wang added a comment - Fix and unit test LGTM thanks Sammi, but looks like we need a small update since I just committed HDFS-10759 for Ewan. +1 pending that and a fresh Jenkins run.
        Hide
        Sammi SammiChen added a comment - - edited

        Hi Andrew Wang, thanks for reviewing the patch! I rebased it.

        Show
        Sammi SammiChen added a comment - - edited Hi Andrew Wang , thanks for reviewing the patch! I rebased it.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 17s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 15m 10s trunk passed
        +1 compile 0m 58s trunk passed
        +1 checkstyle 0m 28s trunk passed
        +1 mvnsite 1m 3s trunk passed
        +1 mvneclipse 0m 13s trunk passed
        +1 findbugs 2m 2s trunk passed
        +1 javadoc 0m 44s trunk passed
        +1 mvninstall 0m 58s the patch passed
        +1 compile 0m 53s the patch passed
        +1 javac 0m 53s the patch passed
        -0 checkstyle 0m 27s hadoop-hdfs-project/hadoop-hdfs: The patch generated 3 new + 25 unchanged - 2 fixed = 28 total (was 27)
        +1 mvnsite 0m 49s the patch passed
        +1 mvneclipse 0m 11s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 1m 52s the patch passed
        +1 javadoc 0m 39s the patch passed
        -1 unit 73m 29s hadoop-hdfs in the patch failed.
        +1 asflicense 0m 33s The patch does not generate ASF License warnings.
        102m 5s



        Reason Tests
        Timed out junit tests org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:a9ad5d6
        JIRA Issue HDFS-11268
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12848239/HDFS-11268-002.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux ce572bc0eb97 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 724bb86
        Default Java 1.8.0_111
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/18206/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/18206/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18206/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18206/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 17s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 15m 10s trunk passed +1 compile 0m 58s trunk passed +1 checkstyle 0m 28s trunk passed +1 mvnsite 1m 3s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 2m 2s trunk passed +1 javadoc 0m 44s trunk passed +1 mvninstall 0m 58s the patch passed +1 compile 0m 53s the patch passed +1 javac 0m 53s the patch passed -0 checkstyle 0m 27s hadoop-hdfs-project/hadoop-hdfs: The patch generated 3 new + 25 unchanged - 2 fixed = 28 total (was 27) +1 mvnsite 0m 49s the patch passed +1 mvneclipse 0m 11s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 52s the patch passed +1 javadoc 0m 39s the patch passed -1 unit 73m 29s hadoop-hdfs in the patch failed. +1 asflicense 0m 33s The patch does not generate ASF License warnings. 102m 5s Reason Tests Timed out junit tests org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HDFS-11268 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12848239/HDFS-11268-002.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux ce572bc0eb97 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 724bb86 Default Java 1.8.0_111 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/18206/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/18206/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18206/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18206/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        andrew.wang Andrew Wang added a comment -

        Thanks for the contribution Sammi, +1 LGTM! I've committed this to trunk.

        Show
        andrew.wang Andrew Wang added a comment - Thanks for the contribution Sammi, +1 LGTM! I've committed this to trunk.
        Hide
        Sammi SammiChen added a comment -

        Thanks Andrew!

        Show
        Sammi SammiChen added a comment - Thanks Andrew!

          People

          • Assignee:
            Sammi SammiChen
            Reporter:
            Sammi SammiChen
          • Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development