Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-15610

Reduce datanode upgrade/hardlink thread

    XMLWordPrintableJSON

Details

    Description

      There is a kernel overhead on datanode upgrade. If datanode with millions of blocks and 10+ disks then block-layout migration will be super expensive during its hardlink operation.  Slowness is observed when running with large hardlink threads(dfs.datanode.block.id.layout.upgrade.threads, default is 12 thread for each disk) and its runs for 2+ hours. 

      I.e 10*12=120 threads (for 10 disks)

      Small test:

      RHEL7, 32 cores, 20 GB RAM, 8 GB DN heap

      dfs.datanode.block.id.layout.upgrade.threads Blocks Disks Time taken
      12 3.3 Million 1 2 minutes and 59 seconds
      6 3.3 Million 1 2 minutes and 35 seconds
      3 3.3 Million 1 2 minutes and 51 seconds

      Tried same test twice and 95% is accurate (only a few sec difference on each iteration). Using 6 thread is faster than 12 thread because of its overhead. 

      Attachments

        Issue Links

          Activity

            People

              kpalanisamy Karthik Palanisamy
              kpalanisamy Karthik Palanisamy
              Votes:
              1 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 1h 10m
                  1h 10m