Whirr
  1. Whirr
  2. WHIRR-351

configure_hadoop should create+chown all data dirs listed in hadoop-hdfs.dfs.data.dir

    Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 0.6.0
    • Fix Version/s: 0.8.0
    • Component/s: service/hadoop
    • Labels:
      None

      Description

      If the directories listed in dfs.data.dir do not exist,
      the datanodes will try to create them. This fails if the hadoop user
      has sufficient permissions.
      To avoid the problem, the directories should be created before starting the datanodes.

      1. WHIRR-351.patch
        9 kB
        Karel Vervaeke
      2. WHIRR-351.patch
        9 kB
        Karel Vervaeke
      3. WHIRR-351.patch
        9 kB
        Karel Vervaeke

        Issue Links

          Activity

          Hide
          Karel Vervaeke added a comment -

          I think WHIRR-189 and WHIRR-639 fix this.

          Show
          Karel Vervaeke added a comment - I think WHIRR-189 and WHIRR-639 fix this.
          Hide
          Karel Vervaeke added a comment -

          Indeed. The non-cdh installation uses the name 'hadoop' (not sure about the group)

          Show
          Karel Vervaeke added a comment - Indeed. The non-cdh installation uses the name 'hadoop' (not sure about the group)
          Hide
          Andrei Savu added a comment -

          I'm seeing chown: invalid user: `mapred:mapred' when running the services/hadoop integration tests. I guess this user only exists for CDH.

          Show
          Andrei Savu added a comment - I'm seeing chown: invalid user: `mapred:mapred' when running the services/hadoop integration tests. I guess this user only exists for CDH.
          Hide
          Karel Vervaeke added a comment -

          Fix ownership and permissins for mapred local dir (if the tasktracker detects that it's not mapred:mapred and 0755 it tries to fix it)

          Show
          Karel Vervaeke added a comment - Fix ownership and permissins for mapred local dir (if the tasktracker detects that it's not mapred:mapred and 0755 it tries to fix it)
          Hide
          Tom White added a comment -

          +1

          Show
          Tom White added a comment - +1
          Hide
          Karel Vervaeke added a comment -

          Updated patch chowns the mapred local dir to mapred:mapred instead of root:root
          (TaskTracker doesn't like it if ownership isn't right, even if permissions are ok)

          Show
          Karel Vervaeke added a comment - Updated patch chowns the mapred local dir to mapred:mapred instead of root:root (TaskTracker doesn't like it if ownership isn't right, even if permissions are ok)
          Hide
          Karel Vervaeke added a comment -

          Attached patch takes care of
          dfs.data.dir, dfs.name.dir, fs.checkpoint.dir and mapred.local.dir

          Please review.

          Show
          Karel Vervaeke added a comment - Attached patch takes care of dfs.data.dir, dfs.name.dir, fs.checkpoint.dir and mapred.local.dir Please review.

            People

            • Assignee:
              Unassigned
              Reporter:
              Karel Vervaeke
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development