Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-4937

ReplicationMonitor can infinite-loop in BlockPlacementPolicyDefault#chooseRandom()

    Details

    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      When a large number of nodes are removed by refreshing node lists, the network topology is updated. If the refresh happens at the right moment, the replication monitor thread may stuck in the while loop of chooseRandom(). This is because the cached cluster size is used in the terminal condition check of the loop. This usually happens when a block with a high replication factor is being processed. Since replicas/rack is also calculated beforehand, no node choice may satisfy the goodness criteria if refreshing removed racks.

      All nodes will end up in the excluded list, but the size will still be less than the cached cluster size, so it will loop infinitely. This was observed in a production environment.

        Attachments

        1. HDFS-4937.patch
          1 kB
          Kihwal Lee
        2. HDFS-4937.v1.patch
          2 kB
          Kihwal Lee
        3. HDFS-4937.v1.patch
          2 kB
          Kihwal Lee
        4. HDFS-4937.v3.patch
          2 kB
          Kihwal Lee

          Issue Links

            Activity

              People

              • Assignee:
                kihwal Kihwal Lee
                Reporter:
                kihwal Kihwal Lee
              • Votes:
                0 Vote for this issue
                Watchers:
                18 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: