Hadoop Common
  1. Hadoop Common
  2. HADOOP-2366

Space in the value for dfs.data.dir can cause great problems

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.21.0
    • Component/s: conf
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      The following configuration causes problems:

      <property>
      <name>dfs.data.dir</name>
      <value>/mnt/hstore2/hdfs, /home/foo/dfs</value>
      <description>
      Determines where on the local filesystem an DFS data node should store its bl
      ocks. If this is a comma-delimited list of directories, then data will be stor
      ed in all named directories, typically on different devices. Directories that
      do not exist are ignored.
      </description>
      </property>

      The problem is that the space after the comma causes the second directory for storage to be " /home/foo/dfs" which is in a directory named <SPACE> which contains a sub-dir named "home" in the hadoop datanodes default directory. This will typically cause the user's home partition to fill, but will be very hard for the user to understand since a directory with a whitespace name is hard to understand.

      My proposed solution would be to trimLeft all path names from this and similar property after splitting on comma. This still allows spaces in file and directory names but avoids this problem.

      1. HADOOP-2366.patch
        6 kB
        Michele Catasta

        Issue Links

          Activity

          Ted Dunning created issue -
          Steve Loughran made changes -
          Field Original Value New Value
          Link This issue is part of HADOOP-3582 [ HADOOP-3582 ]
          Todd Lipcon made changes -
          Assignee Todd Lipcon [ tlipcon ]
          Michele Catasta made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Affects Version/s 0.13.1 [ 12312579 ]
          Michele Catasta made changes -
          Attachment HADOOP-2366.patch [ 12410350 ]
          Michele Catasta made changes -
          Attachment HADOOP-2366.patch [ 12410351 ]
          Michele Catasta made changes -
          Attachment HADOOP-2366.patch [ 12410350 ]
          Michele Catasta made changes -
          Attachment HADOOP-2366.patch [ 12410353 ]
          Michele Catasta made changes -
          Attachment HADOOP-2366.patch [ 12410351 ]
          Michele Catasta made changes -
          Attachment HADOOP-2366.patch [ 12410578 ]
          Michele Catasta made changes -
          Attachment HADOOP-2366.patch [ 12410353 ]
          Michele Catasta made changes -
          Attachment HADOOP-2366.patch [ 12410749 ]
          Michele Catasta made changes -
          Attachment HADOOP-2366.patch [ 12410578 ]
          Tsz Wo Nicholas Sze made changes -
          Link This issue relates to HADOOP-4212 [ HADOOP-4212 ]
          Michele Catasta made changes -
          Attachment HADOOP-2366-trimmed.patch [ 12410876 ]
          Michele Catasta made changes -
          Attachment HADOOP-2366-trimmed.patch [ 12411002 ]
          Michele Catasta made changes -
          Attachment HADOOP-2366-trimmed.patch [ 12410876 ]
          Tom White made changes -
          Assignee Todd Lipcon [ tlipcon ] Michele (aka pirroh) Catasta [ pirroh ]
          Michele Catasta made changes -
          Attachment HADOOP-2366.patch [ 12412109 ]
          Michele Catasta made changes -
          Attachment HADOOP-2366-trimmed.patch [ 12411002 ]
          Michele Catasta made changes -
          Attachment HADOOP-2366.patch [ 12410749 ]
          Michele Catasta made changes -
          Link This issue blocks HDFS-455 [ HDFS-455 ]
          Tsz Wo Nicholas Sze made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Hadoop Flags [Reviewed]
          Fix Version/s 0.21.0 [ 12313563 ]
          Resolution Fixed [ 1 ]
          Todd Lipcon made changes -
          Link This issue relates to MAPREDUCE-1441 [ MAPREDUCE-1441 ]
          Tom White made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Gavin made changes -
          Link This issue blocks HDFS-455 [ HDFS-455 ]
          Gavin made changes -
          Link This issue is depended upon by HDFS-455 [ HDFS-455 ]

            People

            • Assignee:
              Michele Catasta
              Reporter:
              Ted Dunning
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development