Uploaded image for project: 'Samza'
  1. Samza
  2. SAMZA-276

Set -Djava.io.tmpdir to be usercache dir in YARN

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.6.0
    • Fix Version/s: 0.8.0
    • Component/s: None
    • Labels:
      None

      Description

      I don't believe we set -Djava.io.tmpdir in any of our shell scripts right now. When running Samza jobs in YARN, we should set -Djava.io.tmpdir to be the usercache directory or ./tmp or something, not /tmp on the root mount.

      1. SAMZA-276.patch
        1.0 kB
        Yan Fang
      2. SAMZA-276.2.patch
        1.0 kB
        Yan Fang
      3. SAMZA-276.1.patch
        0.9 kB
        Yan Fang

        Activity

        Hide
        closeuris Yan Fang added a comment -

        added the java tmp dir configuration. set to ./tmp as the default. Thanks.

        Show
        closeuris Yan Fang added a comment - added the java tmp dir configuration. set to ./tmp as the default. Thanks.
        Hide
        martinkl Martin Kleppmann added a comment -

        Does the tmp directory already exist by default? Might be safest to create it explicitly in the shell script.

        Also, I think it would be a bit safer to use an absolute path (e.g. "$base_dir/tmp") rather than a relative path, in case something changes the container's working directory after it starts up.

        Show
        martinkl Martin Kleppmann added a comment - Does the tmp directory already exist by default? Might be safest to create it explicitly in the shell script. Also, I think it would be a bit safer to use an absolute path (e.g. "$base_dir/tmp") rather than a relative path, in case something changes the container's working directory after it starts up.
        Hide
        closeuris Yan Fang added a comment -

        Thank you for the comment, Martin. You are right. Now the script checks the directory and creates one if not exits, also use absolute path instead of relative one. Thank you.

        Show
        closeuris Yan Fang added a comment - Thank you for the comment, Martin. You are right. Now the script checks the directory and creates one if not exits, also use absolute path instead of relative one. Thank you.
        Hide
        criccomini Chris Riccomini added a comment -

        Do we need the if/fi block, or can we just always call `mkdir -p`? Seems to be fine to call `mkdir -p` on a directory that already exists.

             -p      Create intermediate directories as required.  If this option is
                     not specified, the full path prefix of each operand must already
                     exist.  On the other hand, with this option specified, no error
                     will be reported if a directory given as an operand already
                     exists.  Intermediate directories are created with permission
                     bits of rwxrwxrwx (0777) as modified by the current umask, plus
                     write and search permission for the owner.
        

        This seems to work:

        criccomi-mn:tmp criccomi$ ls foobarbaz
        ls: foobarbaz: No such file or directory
        criccomi-mn:tmp criccomi$ mkdir -p foobarbaz
        criccomi-mn:tmp criccomi$ ls foobarbaz
        criccomi-mn:tmp criccomi$ mkdir -p foobarbaz
        
        Show
        criccomini Chris Riccomini added a comment - Do we need the if/fi block, or can we just always call `mkdir -p`? Seems to be fine to call `mkdir -p` on a directory that already exists. -p Create intermediate directories as required. If this option is not specified, the full path prefix of each operand must already exist. On the other hand, with this option specified, no error will be reported if a directory given as an operand already exists. Intermediate directories are created with permission bits of rwxrwxrwx (0777) as modified by the current umask, plus write and search permission for the owner. This seems to work: criccomi-mn:tmp criccomi$ ls foobarbaz ls: foobarbaz: No such file or directory criccomi-mn:tmp criccomi$ mkdir -p foobarbaz criccomi-mn:tmp criccomi$ ls foobarbaz criccomi-mn:tmp criccomi$ mkdir -p foobarbaz
        Hide
        closeuris Yan Fang added a comment -

        No, we do not need it. Without if/fi, it still works. Rebased and attached the new patch, which removed the if/fi tag. Thank you.

        Show
        closeuris Yan Fang added a comment - No, we do not need it. Without if/fi, it still works. Rebased and attached the new patch, which removed the if/fi tag. Thank you.
        Hide
        criccomini Chris Riccomini added a comment -

        +1 Merged and committed to master.

        Show
        criccomini Chris Riccomini added a comment - +1 Merged and committed to master.

          People

          • Assignee:
            closeuris Yan Fang
            Reporter:
            criccomini Chris Riccomini
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development