Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-2049

JT and TT should prune invalid local dirs on startup

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Minor Minor
    • Resolution: Won't Fix
    • Affects Version/s: 0.21.0, 0.22.0
    • Fix Version/s: None
    • Component/s: jobtracker, tasktracker
    • Labels:
      None

      Description

      MRAsyncDiskService fails when it's given local dirs it can't create or write. Per MAPREDUCE-1213 this conflicts with the previous behavior. The TTs will no longer start on hosts where the config has localdirs that can not be created (ie if some hosts don't have the same local mounts).

      The TT and JT should prune out invalid local dirs (ones that fail checkLocalDirs ie that it can not create) before giving them to MRAsyncDiskService and using them otherwise.

      Also mapred-default.xml states that mapred.local.dir that do not exist are ignored, this should be updated as directories that don't exist are created (if possible), they're only ignored if they can not be created.

        Issue Links

          Activity

          Transition Time In Source Status Execution Times Last Executer Last Execution Date
          Open Open Resolved Resolved
          1709d 20h 12m 1 Vinod Kumar Vavilapalli 09/May/15 02:07
          Vinod Kumar Vavilapalli made changes -
          Status Open [ 1 ] Resolved [ 5 ]
          Resolution Won't Fix [ 2 ]
          Hide
          Vinod Kumar Vavilapalli added a comment -

          We haven't done a 1.x/0.21/0.22 based release in a long long time. Unlikely this issue is going to be fixed/released.

          This should happen in 2.x at all, but if if you see something, please file a new ticket.

          Closing this as Won't Fix. Please revert back if you disagree. Thanks.

          Show
          Vinod Kumar Vavilapalli added a comment - We haven't done a 1.x/0.21/0.22 based release in a long long time. Unlikely this issue is going to be fixed/released. This should happen in 2.x at all, but if if you see something, please file a new ticket. Closing this as Won't Fix. Please revert back if you disagree. Thanks.
          Nigel Daley made changes -
          Fix Version/s 0.22.0 [ 12314184 ]
          Fix Version/s 0.21.1 [ 12315272 ]
          Zheng Shao made changes -
          Link This issue duplicates MAPREDUCE-1382 [ MAPREDUCE-1382 ]
          Scott Chen made changes -
          Assignee Zheng Shao [ zshao ]
          Hide
          Scott Chen added a comment -

          Zheng has made some change internally to fix this problem. I am assigning this to him.

          Show
          Scott Chen added a comment - Zheng has made some change internally to fix this problem. I am assigning this to him.
          Eli Collins made changes -
          Field Original Value New Value
          Link This issue is related to MAPREDUCE-1213 [ MAPREDUCE-1213 ]
          Eli Collins created issue -

            People

            • Assignee:
              Zheng Shao
              Reporter:
              Eli Collins
            • Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development