Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-11187

Optimize disk access for last partial chunk checksum of Finalized replica

    Details

    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      The patch at HDFS-11160 ensures BlockSender reads the correct version of metafile when there are concurrent writers.

      However, the implementation is not optimal, because it must always read the last partial chunk checksum from disk while holding FsDatasetImpl lock for every reader. It is possible to optimize this by keeping an up-to-date version of last partial checksum in-memory and reduce disk access.

      I am separating the optimization into a new jira, because maintaining the state of in-memory checksum requires a lot more work.

        Attachments

        1. HDFS-11187.001.patch
          24 kB
          Wei-Chiu Chuang
        2. HDFS-11187.002.patch
          17 kB
          Wei-Chiu Chuang
        3. HDFS-11187.003.patch
          17 kB
          Wei-Chiu Chuang
        4. HDFS-11187.004.patch
          17 kB
          Wei-Chiu Chuang
        5. HDFS-11187.005.patch
          17 kB
          Wei-Chiu Chuang
        6. HDFS-11187-branch-2.001.patch
          11 kB
          Gabor Bota
        7. HDFS-11187-branch-2.002.patch
          13 kB
          Gabor Bota
        8. HDFS-11187-branch-2.003.patch
          13 kB
          Gabor Bota
        9. HDFS-11187-branch-2.004.patch
          14 kB
          Gabor Bota
        10. HDFS-11187-branch-2.7.001.patch
          14 kB
          Gabor Bota

          Issue Links

            Activity

              People

              • Assignee:
                gabor.bota Gabor Bota
                Reporter:
                jojochuang Wei-Chiu Chuang
              • Votes:
                0 Vote for this issue
                Watchers:
                12 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: