Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-8578

On upgrade, Datanode should process all storage/data dirs in parallel

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 2.7.3, 3.0.0-alpha1
    • Component/s: datanode
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Right now, during upgrades datanode is processing all the storage dirs sequentially. Assume it takes ~20 mins to process a single storage dir then datanode which has ~10 disks will take around 3hours to come up.

      BlockPoolSliceStorage.java

         for (int idx = 0; idx < getNumStorageDirs(); idx++) {
            doTransition(datanode, getStorageDir(idx), nsInfo, startOpt);
            assert getCTime() == nsInfo.getCTime() 
                : "Data-node and name-node CTimes must be the same.";
          }
      

      It would save lots of time during major upgrades if datanode process all storagedirs/disks parallelly.

      Can we make datanode to process all storage dirs parallelly?

        Attachments

        1. h8578_20160218-branch-2.7-addendum.patch
          2 kB
          Vinayakumar B
        2. h8578_20160218.patch
          20 kB
          Tsz Wo Nicholas Sze
        3. h8578_20160216.patch
          18 kB
          Tsz Wo Nicholas Sze
        4. h8578_20160128b.patch
          19 kB
          Tsz Wo Nicholas Sze
        5. h8578_20160128.patch
          18 kB
          Tsz Wo Nicholas Sze
        6. h8578_20160117.patch
          51 kB
          Tsz Wo Nicholas Sze
        7. h8578_20151213.patch
          50 kB
          Tsz Wo Nicholas Sze
        8. h8578_20151212.patch
          50 kB
          Tsz Wo Nicholas Sze
        9. h8578_20151211b.patch
          46 kB
          Tsz Wo Nicholas Sze
        10. HDFS-8578-17.patch
          25 kB
          Vinayakumar B
        11. h8578_20151211.patch
          43 kB
          Tsz Wo Nicholas Sze
        12. HDFS-8578-16.patch
          23 kB
          Vinayakumar B
        13. h8578_20151210.patch
          41 kB
          Tsz Wo Nicholas Sze
        14. HDFS-8578-15.patch
          20 kB
          Vinayakumar B
        15. HDFS-8578-branch-2.7-003.patch
          20 kB
          Vinayakumar B
        16. HDFS-8578-14.patch
          38 kB
          Vinayakumar B
        17. HDFS-8578-branch-2.7-002.patch
          38 kB
          Vinayakumar B
        18. HDFS-8578-13.patch
          38 kB
          Vinayakumar B
        19. HDFS-8578-branch-2.7-001.patch
          38 kB
          Vinayakumar B
        20. HDFS-8578-12.patch
          19 kB
          Vinayakumar B
        21. HDFS-8578-11.patch
          19 kB
          Vinayakumar B
        22. HDFS-8578-10.patch
          18 kB
          Vinayakumar B
        23. HDFS-8578-09.patch
          18 kB
          Vinayakumar B
        24. HDFS-8578-08.patch
          18 kB
          Vinayakumar B
        25. HDFS-8578-07.patch
          17 kB
          Vinayakumar B
        26. HDFS-8578-06.patch
          16 kB
          Vinayakumar B
        27. HDFS-8578-05.patch
          11 kB
          Vinayakumar B
        28. HDFS-8578-04.patch
          11 kB
          Vinayakumar B
        29. HDFS-8578-03.patch
          11 kB
          Vinayakumar B
        30. HDFS-8578-branch-2.6.0.patch
          4 kB
          Raju Bairishetti
        31. HDFS-8578-02.patch
          10 kB
          Vinayakumar B
        32. HDFS-8578-01.patch
          4 kB
          Vinayakumar B

          Issue Links

            Activity

              People

              • Assignee:
                vinayrpet Vinayakumar B
                Reporter:
                raju.bairishetti Raju Bairishetti
              • Votes:
                0 Vote for this issue
                Watchers:
                34 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: