Details

    • Type: New Feature
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.6.0
    • Component/s: datanode, hdfs-client, namenode
    • Labels:
      None

      Description

      Per discussion with the community on HDFS-5851, we will implement writing to a single replica in DN memory via DataTransferProtocol.

      This avoids some of the issues with short-circuit writes, which we can revisit at a later time.

        Attachments

        1. HDFSWriteableReplicasInMemory.pdf
          127 kB
          Arpit Agarwal
        2. HDFS-6581.merge.01.patch
          231 kB
          Arpit Agarwal
        3. Test-Plan-for-HDFS-6581-Memory-Storage.pdf
          179 kB
          Xiaoyu Yao
        4. HDFS-6581.merge.02.patch
          232 kB
          Arpit Agarwal
        5. HDFS-6581.merge.03.patch
          233 kB
          Arpit Agarwal
        6. HDFS-6581.merge.04.patch
          237 kB
          Arpit Agarwal
        7. HDFS-6581.merge.05.patch
          235 kB
          Arpit Agarwal
        8. HDFS-6581.merge.06.patch
          235 kB
          Arpit Agarwal
        9. HDFS-6581.merge.07.patch
          236 kB
          Arpit Agarwal
        10. HDFS-6581.merge.08.patch
          248 kB
          Arpit Agarwal
        11. HDFS-6581.merge.09.patch
          250 kB
          Arpit Agarwal
        12. HDFS-6581.merge.10.patch
          290 kB
          Arpit Agarwal
        13. Test-Plan-for-HDFS-6581-Memory-Storage.pdf
          202 kB
          Arpit Agarwal
        14. HDFS-6581.merge.11.patch
          310 kB
          Arpit Agarwal
        15. HDFS-6581.merge.12.patch
          307 kB
          Arpit Agarwal
        16. HDFS-6581.merge.14.patch
          279 kB
          Arpit Agarwal
        17. HDFS-6581.merge.15.patch
          260 kB
          Arpit Agarwal

          Issue Links

          1.
          Initial prototype implementation for replicas in memory using tmpfs Sub-task Resolved Arpit Agarwal
          2.
          Add LazyPersist flag to FileStatus Sub-task Closed Arpit Agarwal
          3.
          The HttpFSFileSystem should also return the isLazyPersist flag in the FileStatus Sub-task Resolved Arpit Agarwal
          4.
          Add LazyPersist flag to INodeFile, save it in FsImage and edit logs Sub-task Resolved Arpit Agarwal
          5.
          Propagate LazyPersist flag to DNs via DataTransferProtocol Sub-task Closed Arpit Agarwal
          6.
          Add new RAM_DISK storage type Sub-task Closed Arpit Agarwal
          7.
          DataNode should attempt to place replicas on transient storage first if lazyPersist flag is received Sub-task Closed Arpit Agarwal
          8.
          DN support for saving replicas to persistent storage and evicting in-memory replicas Sub-task Closed Arpit Agarwal
          9.
          Add unit tests Sub-task Closed Arpit Agarwal
          10.
          NN periodically unlinks lazy persist files with missing replicas from namespace Sub-task Closed Arpit Agarwal
          11.
          'hdfs put' command should accept lazyPersist flag for testing Sub-task Closed Arpit Agarwal
          12.
          Bugfix in LazyWriter, fix test case and some refactoring Sub-task Closed Arpit Agarwal
          13.
          Move lazily persisted replicas to finalized directory on DN startup Sub-task Closed Arpit Agarwal
          14.
          Add Additional unit tests for HDFS-6581 Sub-task Closed Xiaoyu Yao
          15.
          Improve replica eviction from RAM disk Sub-task Closed Arpit Agarwal
          16.
          Delete all copies when a block is deleted from the block space Sub-task Closed Arpit Agarwal
          17.
          Notify NN of evicted block before deleting it from RAM disk Sub-task Closed Arpit Agarwal
          18.
          Directory scanner should correctly reconcile blocks on RAM disk Sub-task Closed Arpit Agarwal
          19.
          LazyWriter#evictBlocks misses a null check for replicaState Sub-task Closed Xiaoyu Yao
          20.
          Fix unit test failures in HDFS-6581 branch Sub-task Closed Xiaoyu Yao
          21.
          Few more unit test fixes for HDFS-6581 Sub-task Closed Arpit Agarwal
          22.
          Fix finalize and upgrade unit test failures Sub-task Closed Arpit Agarwal
          23.
          FsDatasetImpl#copyBlockFiles debug log can be improved Sub-task Closed Xiaoyu Yao
          24.
          Add forwarding constructor for INodeFile for existing callers Sub-task Closed Arpit Agarwal
          25.
          Updated editsStored and editsStored.xml to bump layout version and add LazyPersist flag Sub-task Resolved Xiaoyu Yao
          26.
          Fix unit test failures in SimulatedFsDataset Sub-task Closed Arpit Agarwal
          27.
          Make eviction scheme pluggable Sub-task Closed Arpit Agarwal
          28.
          Add unit test for evict/delete RAM_DISK block with open handle Sub-task Closed Xiaoyu Yao
          29.
          Fix findbugs warnings in the HDFS-6581 branch Sub-task Resolved Arpit Agarwal
          30.
          Balancer and Mover tools should ignore replicas on RAM_DISK Sub-task Closed Xiaoyu Yao
          31.
          Fix findbugs warnings in HDFS-6581 branch Sub-task Closed Tsz Wo Nicholas Sze
          32.
          Fix findbugs warnings in RamDiskReplicaTracker Sub-task Closed Tsz Wo Nicholas Sze
          33.
          Integrate HDFS-6581 memory usage with HDFS-4949 Sub-task Resolved Colin P. McCabe
          34.
          Bugfix in createLocatedFileStatus caused by bad merge Sub-task Closed Arpit Agarwal
          35.
          Fix incorrect layout version caused by bad merge Sub-task Resolved Arpit Agarwal
          36.
          Use block storage policy to set lazy persist preference Sub-task Closed Arpit Agarwal
          37.
          Metrics to track usage of memory for writes Sub-task Closed Xiaoyu Yao
          38.
          Fix Jenkins failures in HDFS-6581 branch Sub-task Closed Arpit Agarwal
          39.
          Improve HDFS-6581 eviction configuration Sub-task Closed Xiaoyu Yao
          40.
          Use unbuffered writes when persisting in-memory replicas Sub-task Closed Xiaoyu Yao
          41.
          LazyWriter should use either async IO or one thread per physical disk Sub-task Closed Xiaoyu Yao
          42.
          Move checksum computation off the hot path when writing to RAM disk Sub-task Closed Chris Nauroth
          43.
          Persist in-memory replicas with appropriate unbuffered copy API on POSIX and Windows Sub-task Closed Xiaoyu Yao
          44.
          Feature documentation for HDFS-6581 Sub-task Closed Arpit Agarwal
          45.
          DN should ignore lazyPersist hint if the writer is not local Sub-task Resolved Arpit Agarwal
          46.
          Implement a 2Q eviction strategy for HDFS-6581 Sub-task Patch Available Colin P. McCabe
          47.
          Skip checksum verification when reading from memory Sub-task Open Unassigned
          48.
          Investigate eviction from RAM disk via file truncation Sub-task Open Unassigned
          49.
          DFSOutputStream should query policies from NameNode Sub-task Open Arpit Agarwal
          50.
          Support using ramfs partitions on Linux Sub-task Open Arpit Agarwal

            Activity

              People

              • Assignee:
                arpitagarwal Arpit Agarwal
                Reporter:
                arpitagarwal Arpit Agarwal
              • Votes:
                0 Vote for this issue
                Watchers:
                43 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: