Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-4397

Introduce HADOOP_SECURITY_CONF_DIR for task-controller

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.1.2
    • Component/s: task-controller
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Target Version/s:

      Description

      The linux task controller currently hard codes the directory in which to look for its config file at compile time (via the HADOOP_CONF_DIR macro). Adding a new environment variable to look for task-controller's conf dir (with strict permission checks) would make installation much more flexible.

      1. test-patch.result
        491 kB
        Yu Gao
      2. mapreduce-4397-branch-1.patch
        4 kB
        Yu Gao

        Activity

        Hide
        Robert Joseph Evans added a comment -

        This is very similar work to what was done in MAPREDUCE-4219. MAPREDUCE-4219 was done for yarn and not branch-1 so you probably want to look at that first to reconcile the differences between the two approaches. Any changes being made to branch-1 in this respect would be nice to have ported to trunk and branch-2 for the container-executor because they are very similar still.

        Show
        Robert Joseph Evans added a comment - This is very similar work to what was done in MAPREDUCE-4219 . MAPREDUCE-4219 was done for yarn and not branch-1 so you probably want to look at that first to reconcile the differences between the two approaches. Any changes being made to branch-1 in this respect would be nice to have ported to trunk and branch-2 for the container-executor because they are very similar still.
        Hide
        Matt Foley added a comment -

        Moved to 1.2.0 upon release of 1.1.0.

        Show
        Matt Foley added a comment - Moved to 1.2.0 upon release of 1.1.0.
        Hide
        Luke Lu added a comment -

        MAPREDUCE-4219 is actually orthogonal to this jira. The patch for branch-1 lgtm. Thanks Yu!

        Show
        Luke Lu added a comment - MAPREDUCE-4219 is actually orthogonal to this jira. The patch for branch-1 lgtm. Thanks Yu!
        Hide
        Yu Gao added a comment -

        Posted test-patch result. Here is the test result of test-task-controller:
        [exec] Starting tests
        [exec]
        [exec] Testing get_user_directory()
        [exec]
        [exec] Testing get_job_directory()
        [exec]
        [exec] Testing get_attempt_directory()
        [exec]
        [exec] Testing get_task_launcher_file()
        [exec]
        [exec] Testing get_job_log_dir()
        [exec]
        [exec] Testing get_config_path
        [exec]
        [exec] Testing check_configuration_permissions
        [exec] File /tmp/test-task-controller must be owned by root, but is owned by 500
        [exec]
        [exec] Testing get_task_log_dir()
        [exec]
        [exec] Testing delete_task()
        [exec] Unreadable directory /tmp/test-task-controller/local-2/taskTracker/biadmin/jobcache/job_1/task_1/work/who/let, chmoding.
        [exec]
        [exec] Testing delete_job()
        [exec] Unreadable directory /tmp/test-task-controller/local-2/taskTracker/biadmin/jobcache/job_2/task_1/work/who/let, chmoding.
        [exec]
        [exec] Testing delete_user
        [exec]
        [exec] Testing test_check_user
        [exec] Requested user lp has id 4, which is below the minimum allowed 100
        [exec] Running as root is not allowed
        [exec] User mapred not found
        [exec]
        [exec] Testing test_create_log_directory
        [exec]
        [exec] Testing delete_log_directory
        [exec]
        [exec] Running test test_signal_task in child process
        [exec]
        [exec] Testing signal_task
        [exec] Child task launched as 26548
        [exec] Killing process 26548 with 3
        [exec]
        [exec] Running test test_signal_task_group in child process
        [exec]
        [exec] Testing group signal_task
        [exec] Child task launched as 26550
        [exec] Killing process group 26550 with 9
        [exec]
        [exec] Finished tests
        [exec] PASS: test-task-controller
        [exec] ==================
        [exec] All 1 tests passed
        [exec] ==================

        Show
        Yu Gao added a comment - Posted test-patch result. Here is the test result of test-task-controller: [exec] Starting tests [exec] [exec] Testing get_user_directory() [exec] [exec] Testing get_job_directory() [exec] [exec] Testing get_attempt_directory() [exec] [exec] Testing get_task_launcher_file() [exec] [exec] Testing get_job_log_dir() [exec] [exec] Testing get_config_path [exec] [exec] Testing check_configuration_permissions [exec] File /tmp/test-task-controller must be owned by root, but is owned by 500 [exec] [exec] Testing get_task_log_dir() [exec] [exec] Testing delete_task() [exec] Unreadable directory /tmp/test-task-controller/local-2/taskTracker/biadmin/jobcache/job_1/task_1/work/who/let, chmoding. [exec] [exec] Testing delete_job() [exec] Unreadable directory /tmp/test-task-controller/local-2/taskTracker/biadmin/jobcache/job_2/task_1/work/who/let, chmoding. [exec] [exec] Testing delete_user [exec] [exec] Testing test_check_user [exec] Requested user lp has id 4, which is below the minimum allowed 100 [exec] Running as root is not allowed [exec] User mapred not found [exec] [exec] Testing test_create_log_directory [exec] [exec] Testing delete_log_directory [exec] [exec] Running test test_signal_task in child process [exec] [exec] Testing signal_task [exec] Child task launched as 26548 [exec] Killing process 26548 with 3 [exec] [exec] Running test test_signal_task_group in child process [exec] [exec] Testing group signal_task [exec] Child task launched as 26550 [exec] Killing process group 26550 with 9 [exec] [exec] Finished tests [exec] PASS: test-task-controller [exec] ================== [exec] All 1 tests passed [exec] ==================
        Hide
        Luke Lu added a comment -

        +1. Committed to branch-1 and 1.1. Thanks Yu!

        Show
        Luke Lu added a comment - +1. Committed to branch-1 and 1.1. Thanks Yu!
        Hide
        Matt Foley added a comment -

        Patch accepted to 1.1.2.

        Show
        Matt Foley added a comment - Patch accepted to 1.1.2.
        Hide
        Owen O'Malley added a comment -

        This patch weakens security, so it has been reverted by MAPREDUCE-5202.

        Show
        Owen O'Malley added a comment - This patch weakens security, so it has been reverted by MAPREDUCE-5202 .
        Hide
        Vinod Kumar Vavilapalli added a comment -

        One problem is that this is shipped in 1.1.2, so mark MAPREDUCE-5202 as incompatible?

        I didn't look at MAPREDUCE-4219 unfortunately, but that sets the path to be relative to the binary for YARN, is that an issue too?

        I think the fundamental point of this and MAPREDUCE-4219 is easy relocation of the conf by the installers, do we have solution for that?

        Show
        Vinod Kumar Vavilapalli added a comment - One problem is that this is shipped in 1.1.2, so mark MAPREDUCE-5202 as incompatible? I didn't look at MAPREDUCE-4219 unfortunately, but that sets the path to be relative to the binary for YARN, is that an issue too? I think the fundamental point of this and MAPREDUCE-4219 is easy relocation of the conf by the installers, do we have solution for that?
        Hide
        Luke Lu added a comment -

        As I mentioned in MAPREDUCE-5202 and the security mailing list, this feature and its equivalent for YARN do NOT "really" weakens security on a properly configured cluster.

        Show
        Luke Lu added a comment - As I mentioned in MAPREDUCE-5202 and the security mailing list, this feature and its equivalent for YARN do NOT "really" weakens security on a properly configured cluster.

          People

          • Assignee:
            Yu Gao
            Reporter:
            Luke Lu
          • Votes:
            0 Vote for this issue
            Watchers:
            12 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development