Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-9806 Allow HDFS block replicas to be provided by an external storage system
  3. HDFS-12093

[READ] Share remoteFS between ProvidedReplica instances.

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 3.1.0
    • None
    • None
    • Reviewed

    Description

      When a Datanode comes online using Provided storage, it fills the ReplicaMap with the known replicas. With Provided Storage, this includes ProvidedReplica instances. Each of these objects, in their constructor, will construct an FileSystem using the Service Provider. This can result in contacting the remote file system and checking that the credentials are correct and that the data is there. For large systems this is a prohibitively expensive operation to perform per replica.

      Instead, the ProvidedVolumeImpl should own the reference to the remoteFS and should share it with the ProvidedReplica objects on their creation.

      Attachments

        1. HDFS-12093-HDFS-9806.002.patch
          9 kB
          Virajith Jalaparti
        2. HDFS-12093-HDFS-9806.001.patch
          9 kB
          Virajith Jalaparti

        Activity

          People

            virajith Virajith Jalaparti
            ehiggs Ewan Higgs
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: