Hadoop Common
  1. Hadoop Common
  2. HADOOP-8274

In pseudo or cluster model under Cygwin, tasktracker can not create a new job because of symlink problem.

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Won't Fix
    • Affects Version/s: 0.20.205.0, 1.0.0, 1.0.1, 0.22.0
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None
    • Environment:

      windows7+cygwin 1.7.11-1+jdk1.6.0_31+hadoop 1.0.0

      Description

      The standalone model is ok. But, in pseudo or cluster model, it always throw errors, even I just run wordcount example.

      The HDFS works fine, but tasktracker can not create threads(jvm) for new job. It is empty under /logs/userlogs/job-xxxx/attempt-xxxx/.

      The reason looks like that in windows, Java can not recognize a symlink of folder as a folder.

      The detail description is as following,

      ======================================================================================================

      First, the error log of tasktracker is like:

      ======================
      12/03/28 14:35:13 INFO mapred.JvmManager: In JvmRunner constructed JVM ID: jvm_201203280212_0005_m_-1386636958
      12/03/28 14:35:13 INFO mapred.JvmManager: JVM Runner jvm_201203280212_0005_m_-1386636958 spawned.
      12/03/28 14:35:17 INFO mapred.JvmManager: JVM Not killed jvm_201203280212_0005_m_-1386636958 but just removed
      12/03/28 14:35:17 INFO mapred.JvmManager: JVM : jvm_201203280212_0005_m_-1386636958 exited with exit code -1. Number of tasks it ran: 0
      12/03/28 14:35:17 WARN mapred.TaskRunner: attempt_201203280212_0005_m_000002_0 : Child Error
      java.io.IOException: Task process exit with nonzero status of -1.
      at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:258)
      12/03/28 14:35:21 INFO mapred.TaskTracker: addFreeSlot : current free slots : 2
      12/03/28 14:35:24 INFO mapred.TaskTracker: LaunchTaskAction (registerTask): attempt_201203280212_0005_m_000002_1 task's state:UNASSIGNED
      12/03/28 14:35:24 INFO mapred.TaskTracker: Trying to launch : attempt_201203280212_0005_m_000002_1 which needs 1 slots
      12/03/28 14:35:24 INFO mapred.TaskTracker: In TaskLauncher, current free slots : 2 and trying to launch attempt_201203280212_0005_m_000002_1 which needs 1 slots
      12/03/28 14:35:24 WARN mapred.TaskLog: Failed to retrieve stdout log for task: attempt_201203280212_0005_m_000002_0
      java.io.FileNotFoundException: D:\cygwin\home\timwu\hadoop-1.0.0\logs\userlogs\job_201203280212_0005\attempt_201203280212_0005_m_000002_0\log.index (The system cannot find the path specified)
      at java.io.FileInputStream.open(Native Method)
      at java.io.FileInputStream.<init>(FileInputStream.java:120)
      at org.apache.hadoop.io.SecureIOUtils.openForRead(SecureIOUtils.java:102)
      at org.apache.hadoop.mapred.TaskLog.getAllLogsFileDetails(TaskLog.java:188)
      at org.apache.hadoop.mapred.TaskLog$Reader.<init>(TaskLog.java:423)
      at org.apache.hadoop.mapred.TaskLogServlet.printTaskLog(TaskLogServlet.java:81)
      at org.apache.hadoop.mapred.TaskLogServlet.doGet(TaskLogServlet.java:296)
      at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
      at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
      at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
      at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
      at org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
      at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
      at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
      at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
      at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
      at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
      at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
      at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
      at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
      at org.mortbay.jetty.Server.handle(Server.java:326)
      at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
      at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
      at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
      at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
      at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
      at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
      at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
      12/03/28 14:35:24 WARN mapred.TaskLog: Failed to retrieve stderr log for task: attempt_201203280212_0005_m_000002_0
      java.io.FileNotFoundException: D:\cygwin\home\timwu\hadoop-1.0.0\logs\userlogs\job_201203280212_0005\attempt_201203280212_0005_m_000002_0\log.index (The system cannot find the path specified)
      at java.io.FileInputStream.open(Native Method)
      at java.io.FileInputStream.<init>(FileInputStream.java:120)
      at org.apache.hadoop.io.SecureIOUtils.openForRead(SecureIOUtils.java:102)
      at org.apache.hadoop.mapred.TaskLog.getAllLogsFileDetails(TaskLog.java:188)
      at org.apache.hadoop.mapred.TaskLog$Reader.<init>(TaskLog.java:423)
      at org.apache.hadoop.mapred.TaskLogServlet.printTaskLog(TaskLogServlet.java:81)
      at org.apache.hadoop.mapred.TaskLogServlet.doGet(TaskLogServlet.java:296)
      at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
      at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
      at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
      at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
      at org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
      at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
      at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
      at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
      at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
      at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
      at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
      at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
      at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
      at org.mortbay.jetty.Server.handle(Server.java:326)
      at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
      at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
      at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
      at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
      at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
      at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
      at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)

      =======================================

      I've tried to remote debug tasktracker. In

      org.apache.hadoop.mapredTaskLog.createTaskAttemptLogDir(TaskAttemptID, boolean, String[]) line: 97:
      public static void createTaskAttemptLogDir(TaskAttemptID taskID,
      boolean isCleanup, String[] localDirs) throws IOException{
      String cleanupSuffix = isCleanup ? ".cleanup" : "";
      String strAttemptLogDir = getTaskAttemptLogDir(taskID,
      cleanupSuffix, localDirs);
      File attemptLogDir = new File(strAttemptLogDir);
      if (!attemptLogDir.mkdirs())

      { throw new IOException("Creation of " + attemptLogDir + " failed."); }

      String strLinkAttemptLogDir =
      getJobDir(taskID.getJobID()).getAbsolutePath() + File.separatorChar +
      taskID.toString() + cleanupSuffix;
      if (FileUtil.symLink(strAttemptLogDir, strLinkAttemptLogDir) != 0)

      { throw new IOException("Creation of symlink from " + strLinkAttemptLogDir + " to " + yestrAttemptLogDir + " failed."); }

      //Set permissions for target attempt log dir
      FsPermission userOnly = new FsPermission((short) 0777); //FsPermission userOnly = new FsPermission((short) 0700);
      FileUtil.setPermission(attemptLogDir, userOnly);
      }
      and symlink() function
      public static int symLink(String target, String linkname) throws IOException{
      String cmd = "ln -s " + target + " " + linkname;
      Process p = Runtime.getRuntime().exec(cmd, null);
      int returnVal = -1;
      try

      { returnVal = p.waitFor(); }

      catch(InterruptedException e)

      { //do nothing as of yet }

      if (returnVal != 0)

      { LOG.warn("Command '" + cmd + "' failed " + returnVal + " with: " + copyStderr(p)); }

      return returnVal;
      }

      we know hadoop will create a log folder in $

      {hadoop.tmp.dir}

      , and then invoke "ln -s " to create its symlink under /logs/userlog/job-xxx/attermp-xxxx.

      In my case,
      strLinkAttemptLogDir = D:\cygwin\home\timwu\hadoop-1.0.0\logs\userlogs\job_201203280212_0005\attempt_201203280212_0005_m_000002_1
      strAttemptLogDir=/tmp/hadoop-timwu/mapred/local\userlogs\job_201203280212_0005\attempt_201203280212_0005_m_000002_1

      After a subtrack is created by tasktracker, it runs error in the following function:

      in org.apache.hadoop.mapred.java , DefaultTaskController.launchTask(String, String, String, List<String>, List<String>, File, String, String) line: 107
      ...............
      //mkdir the loglocation
      String logLocation = TaskLog.getAttemptDir(jobId, attemptId).toString();
      if (!localFs.mkdirs(new Path(logLocation)))

      { throw new IOException("Mkdirs failed to create " + logLocation); }

      ..............

      mkdir() return false, because logLocation is a symlink file. In my case, it is ogLocation=D:\cygwin\home\timwu\hadoop-1.0.0\logs\userlogs\job_201203280212_0005\attempt_201203280212_0005_m_000002_1. If I open it from explorer in windows, it is just a file, but not a folder or shortcut. And its content is like,
      <symlink>/tmp/hadoop-timwu/mapred/local\userlogs\job_201203280212_0005\attempt_201203280212_0005_m_000002_1

      Because the mkdir() is
      public boolean mkdirs(Path f) throws IOException

      { Path parent = f.getParent(); File p2f = pathToFile(f); return (parent == null || mkdirs(parent)) && (p2f.mkdir() || p2f.isDirectory()); }

      So, p2f.isDirectory returns false. And p2f.isFile will return true. So, for java, it is a file. Hence, IOException("Mkdirs failed to create D:\cygwin\home\timwu\hadoop-1.0.0\logs\userlogs\job_201203280212_0005\attempt_201203280212_0005_m_000001_1")
      will be throws in child threads, and return -1. Then, we will get the above exception in main thread.

      Is it any way to close this symlink? Or any other way I can follow?

      BTW, in core-site.xml, I set hadoop.tmp.dir = /tmp/hadoop-$

      {user.name}

      , and my $User.name is timwu. So, it should create a tmp folder /tmp/hadoop-timwu under cygwin's. However, in deed , it create a folder of d:/tmp/hadoop-timwu. If in cygwin, it is /cygdriver/d/tmp/hadoop-timwu. Is it correct?

        Activity

        Hide
        FKorning added a comment -

        see https://issues.apache.org/jira/browse/HADOOP-7682

        as mentioned, java io doesn't understand symlinks or junctions.

        the workaround is to ensure that your windows and cygwin paths are equal.
        Create a /cygwin symlink pointing back to the cygwin root ( /cygwin -> / )
        Use driveless paths and use the cygwin tmp dir for your hadoop_tmp_dir.

        hadoop_tmp_dir=/cygwin/tmp/hadoop-cyg_server <==> c:\cygwin\tmp\hadoop-cyg_server

        Show
        FKorning added a comment - see https://issues.apache.org/jira/browse/HADOOP-7682 as mentioned, java io doesn't understand symlinks or junctions. the workaround is to ensure that your windows and cygwin paths are equal. Create a /cygwin symlink pointing back to the cygwin root ( /cygwin -> / ) Use driveless paths and use the cygwin tmp dir for your hadoop_tmp_dir. hadoop_tmp_dir=/cygwin/tmp/hadoop-cyg_server <==> c:\cygwin\tmp\hadoop-cyg_server
        Hide
        FKorning added a comment -

        If the symlinks really are necessary, here's a java.io.File wrapper
        that understands cygwin symlinks (and windows shortcuts).

        package net.muxbus.io;
        
        import java.io.BufferedReader;
        import java.io.File;
        import java.io.FileReader;
        import java.io.IOException;
        import java.net.URI;
        
        
        /**
         * 
         * @author      Francis Korning   fkorning@yahoo.ca
         * @date        2012-04-30    
         * 
         * @project     muxbus            muxbus.net
         * @source      sourceforge       https://sourceforge.net/p/muxbus/ 
         *     
         * @copyright   2012              F.Korning
         * @license     LGPL              http://www.gnu.org/copyleft/lesser.html
         * 
         * @credit      Jesse Hager       jessehager@iname.com    (.lnk shortcut file format).
         * 
         * 
         * LinkedFile is a java.io.File wrapper that understands both Cygwin Symbolic Links
         * and Windows Explorer Shorcut Links (.lnk).  If a given File is a Link, it tries
         * to resolve the File by following link paths recursively until a final canonical
         * target is found, and if the file exists will act as a wrapper for the link
         * target. A LinkedFile always tracks its source and provides link reflection.
         *
         * A common frustration for POSIX platform developers is that Java on windows does
         * not understand symbolic Links which aren't handled by the windows io native libs.
         * This means that Cygwin or Interix (aka Winterix or Windows Service for Unix)
         * integrated java applications that depend on POSIX tools may break when it comes
         * to resolution of paths with linked files and directories.
         * 
         * The first workaround strategy, which works where links are predictably created,
         * is to ensure that cygwin paths and windows paths are equivalent and identical.
         * For example, you create a circular "/cygwin" link that maps back to the root "/":
         * 
         * <code>
         * <pre>                
         *     /cygwin      ->  /      ->  \cygwin\      ->  C:\cygwin\
         *     /cygwin/tmp  ->  /tmp/  ->  \cygwin\tmp\  ->  C:\cygwin\tmp\
         * </pre>
         * </code>
         * 
         * But you often have no control over much of the integrated codebase and tools.
         * If some of the components create symlinks dynamically where java expects real
         * files and directories this workaround will fail.
         * 
         * Now the only OS-native IO alternative would be to use Windows NTFS Junctions.
         * These are completely transparent and work in shells, in Explorer, and for java.
         * The best implementation would be the junction provided in the SysInternals suite.
         * You could wrap the cygwin 'ln' command and force it to use a junction instead,
         * but this breaks some POSIX compatibility and requires Administrator superuser
         * rights which you may not want to assign to all your running service daemons.
         * 
         * Next Interix symbolic links and the mklink command are built-in since Windows Vista.
         * These fall in between as they act like native symlinks in the shell and in java,
         * look like cygwin links inside the shell, but again they can only be created by the
         * Administrator, and also break POSIX. Unlike cygwin they do appear as shortcuts in
         * a Windows Explorer, with the caveat that these are broken because of permissions.
         * 
         * Note that both strategies only work if the tools create links via the 'ln' command.
         * But what if the offending integrated tools are in perl or python, or even better yet in
         * compiled c/c++ code making use of libc calls for which we have no source nor license?
         * This means that we will need to adopt a strategy to adapt java.io.File to Cygwin libc.
         * Also, most multi-platform java developers favour Cygwin instead of Microsoft's Interix.
         * All of the POSIX daemons we're looking at expect cygwin shells, and this compatibility
         * is our main focus here; we ideally an a model that works without having to wrap 'ln'.
         * 
         * Now Both Cygwin Symlinks and Explorer Shortcuts are just plain binary files underneath
         * with some information pointing to their link targets.  They have the advantage that
         * we can parse these and thus this allows us to add some link reflection to java.io.
         * (This also creates a strange case where windows would have more functionality than
         * unix, unless one would write a unix java native libc wrapper for link functions).
         * They also require no special Administrator rights for creation and maintenance.
         * 
         * A side-effect is that shortcuts created under Windows Explorer have the feature that 
         * a (.lnk) extension is appended, which is obscured from the explorer.  As the names
         * differ, both a Cygwin Symlink and a Windows Shortcut can co-exist in the same place
         * and make for a consistent interface both in the cygwin shell and Windows Explorer.
         *
         * This means that one could well mirror every cygwin symlink with a windows shortcut,
         * and thus have symbolic links that work even under non-Administrator user shells and
         * are mirrored by corresponding Windows Shortcuts that work in the Windows Explorer.
         * Cygwin also comes with its own programmatic 'mkshortcut' command to create Shorcuts
         * from a shell (which you can't under windows by default - doesn't Windows suck?).
         * 
         * <code>
         * <pre>
         *     dir       ->         /dir/            symlink
         *     dir.lnk   ->         /dir/            shortcut
         *     file      ->         /dir/file        symlink
         *     file.lnk  ->         /dir/file        shortcut
         * </pre>
         * </code>
         * 
         * Note that synchronizing of Cygwin symlinks and Windows Shortcuts is left out-of-scope.
         * Our primary concern is to get our POSIX daemons running, not to add sugar to Explorer.
         * However LinkedFile will support and resolve links for both link formats, with first
         * preferrence given to Cygwin Symlinks if found, and then to Windows Shortcuts of the
         * same name (without the .lnk extension), which you may extend for such a facility.
         * 
         * For those wishing to extend the functionality to support transparent mirroring between
         * Explorer and the shell, note that you will have to map links to their (.lnk) shortcuts.
         * Also note that Windows Shortcuts are never interpreted as links by the Cygwin shell.
         * Most critically note that Windows shortcuts are unpredictable in the shell if moved,
         * as they seem to only recalculate relative paths after a link is accessed in Explorer. 
         *
         * 
         * @see cygwin:             http://www.cygwin.com
         * @see cygwin  symlinks:   http://cygwin.com/ml/cygwin/2010-11/msg00394.html 
         * @see interix links:      http://technet.microsoft.com/en-us/library/bb463204.aspx
         * @see windows shortcuts:  http://ithreats.files.wordpress.com/2009/05/lnk_the_windows_shortcut_file_format.pdf
         * @see windows junctions:  http://technet.microsoft.com/en-us/sysinternals
         * 
         */
        public class LinkedFile
            extends java.io.File
        {
        
            
            public static final String TOK_SYMLINK = "!<symlink>\u00FF\u00FE";
            
            public static final String TOK_SHORTCUT = "\u004C\u0000\u0000\u0000\u0001\u0014\u0002";
            public static final String TOK_SHORTCUT_SEP = "\u0000";
            
            public static final int    SHORTCUT_MAX = 1024;    // shortcuts should probably never be that big
            public static final int    SHORTCUT_OFFSET = 76;   // first shortcut section length offset at 0x4C    
        
        
            
            // utilities
            
            
            /**
             * read a link header.
             * 
             * both cygwin symlinks and windows shortcuts are plain binary files.
             * cygwin is faily simple, consisting of magic header and the target.
             * shortcuts have variable length structures but will not exceed 1kb.
             */
            public static String header (File file)
                throws IOException
            {
                if (file == null)
                    return null;
                
                String header = null;
                
                if (file.exists() && file.canRead())
                {
                    if (file.isFile() || (!file.isDirectory()))
                    {                
                        char[] buf = new char [SHORTCUT_MAX];
                        int length = new BufferedReader(new FileReader(file)).read(buf, 0, SHORTCUT_MAX);
                        header = new String(buf);
        
                        //System.err.println ("length: " + length);
                        //System.err.println ("header: " + header);
                        
                        if (isSymlinkHeader(header))
                        {
                            //System.err.println("isSymlink: true");                    
                        }
                        else if (isShortcutHeader(header))
                        {
                            //System.err.println("isShortcut: true");
                        }
                        else
                        {
                            header = null;
                        }
                        
                    }
                }
                
                return header;
            }
        
            
            public static boolean isLinkHeader (String header)
                throws IOException
            {
                return ( isSymlinkHeader(header) || isShortcutHeader(header) );
            }
            
            
            public static boolean isSymlinkHeader (String header)
                throws IOException
            {
                if (header == null)
                    return false;
                
                if (header.startsWith(TOK_SYMLINK))
                {
                    return true;
                }
                
                return false;
            }
        
            
            public static boolean isShortcutHeader (String header)
                throws IOException
            {
                if (header == null)
                    return false;
                
                if (header.startsWith(TOK_SHORTCUT))
                {
                    return true;
                }
                
                return false;
            }
        
        
            /**
             * parse a 2-char short length offset from a position in a windows shortcut file.
             */
            public static int shortcutHeaderOffset (String header, int pos)
            {
                if (header == null)
                    return 0;
        
                char small = header.charAt(pos);
                char large = header.charAt(pos +1);
                int offset = small + (large * 16 * 16);
                
                //System.err.println ("offset [" + pos + "] '" + (short)small + " " + (short)large + "' : " + offset);        
                return offset;
            }
            
            
            /**
             * parse the final link target from a cygwin symlink or windows shortcut.
             * 
             * cygwin symlink targets are immediately after the header "<!symlink> 0xFF xFE".
             * 
             * windows shortcuts start with the header "L 0x00 0x00 0x00 0x01 0x14 0x02".
             * the windows shortcut files are organized into sections of variable lengths,
             * starting after the header at 0x4c (76). the lengths are 2-char short offsets.
             * windows shortcut final name is at the end of the second "file location" section.
             */
            public static String headerTarget (String header)
            {
                if (header == null)
                    return null;
        
                String target = null;
        
                try
                {
                    if (isSymlinkHeader(header))
                    {
                        StringBuffer buf = new StringBuffer("");
                        for (int i = TOK_SYMLINK.length();  i < header.length(); i+=2)
                        {
                            buf.append (header.charAt(i));
                        }
                        
                        target = buf.toString();                
                    }
                    else if (isShortcutHeader(header))
                    {
                        // first get the length of the next "item id" section 
                        int a = shortcutHeaderOffset(header, SHORTCUT_OFFSET);
                        // now get the length of the next "file location" section
                        int b = shortcutHeaderOffset(header, SHORTCUT_OFFSET + a  + 2);
                        
                        // the final name is the last field of the "file location" section 
                        int j = SHORTCUT_OFFSET + a + b;                
                        int i = header.lastIndexOf(TOK_SHORTCUT_SEP, j - 2) + 1;
                        
                        target = header.substring (i,j);                
                    }
                }
                catch (IOException e) {}
        
                //System.err.println ("target: " + target);
                return target;
            }
            
            
            
            public static boolean isLink (File file)
            {
                if (file == null)
                    return false;
                
                try
                {
                    if (file.exists() && file.canRead())
                    {
                        if (file.isFile() || (!file.isDirectory()))
                        {
                            String header = header(file);
                            return isLinkHeader(header);
                        }
                    }
                }
                catch (IOException e) {}
                
                return false;
            }
        
            
            public static boolean isSymlink (File file)
            {
                if (file == null)
                    return false;
                
                try
                {
                    if (file.exists() && file.canRead())
                    {
                        if (file.isFile() || (!file.isDirectory()))
                        {
                            String header = header(file);
                            return isSymlinkHeader(header);
                        }
                    }
                }
                catch (IOException e) {}
                
                return false;
            }
        
            
            public static boolean isShortcut (File file)
            {
                if (file == null)
                    return false;
                
                try {
                    if (file.exists() && file.canRead())
                    {
                        if (file.isFile() || (!file.isDirectory()))
                        {
                            String header = header(file);
                            return isShortcutHeader(header);
                        }
                    }
                }
                catch (IOException e) {}
                
                return false;
            }
            
            
            /**
             * Resolve a target by recursively traversing links until the canonical target.
             * If the file is not a link, return the file's canonical AbsolutePath.
             */
            public static String resolveTarget (File file)
            {
                String target = file.getAbsolutePath();
                //System.err.println("\path: " + target);
                
                try
                {
                    if (file.exists() && file.canRead())
                    {
                        if (file.isFile() || (!file.isDirectory()))
                        {
                            String header = header(file);
                            if (isLinkHeader(header))
                            {
                                target = headerTarget(header);
                                boolean absolute = (target.substring(1)).startsWith(":");
                                boolean logical = (target.startsWith("/") || target.startsWith("\\"));
                                boolean parent = (target.startsWith("../") || target.startsWith("..\\"));                        
                                boolean current = (target.startsWith("./") || target.startsWith(".\\"));
                                
                                File link = null;
                                if (absolute)
                                    link = new File (target.substring(0,2), target.substring(2));
                                else if (logical)
                                    link = new File(target.substring(0,1), target.substring(1));
                                else if (parent)
                                    link = new File(target.substring(0,2), target.substring(2));                        
                                else if (current)
                                    link = new File(target.substring(0,1), target.substring(1));
                                else
                                    link = new File (target);
                                                        
                                if (! file.equals(link))
                                {
                                    target = resolveTarget(link);
                                }
                            }
                        }
                    }
                }
                catch (Exception e)
                {
                    e.printStackTrace();
                }
                
                //System.err.println("resolve: " + target);
                return target;
            }
        
            
            /**
             * 
             * Resolve a file by recursively traversing links until the canonical target.
             * If the file is not a link, return the file itself.
        
             * @param file
             * @return
             */
            public static File resolveFile (File file)
            {
                //System.err.println("file: " + file.getAbsolutePath());
                
                try
                {
                    if (file.exists() && file.canRead())
                    {
                        if (file.isFile() || (!file.isDirectory()))
                        {
                            String header = header(file);
                            if (isLinkHeader(header))
                            {
                                String target = headerTarget(header);
                                boolean absolute = (target.substring(1)).startsWith(":");
                                boolean logical = (target.startsWith("/") || target.startsWith("\\"));
                                boolean parent = (target.startsWith("../") || target.startsWith("..\\"));                        
                                boolean current = (target.startsWith("./") || target.startsWith(".\\"));
                                
                                File link = null;
                                if (absolute)
                                    link = new File (target.substring(0,2), target.substring(2));
                                else if (logical)
                                    link = new File(target.substring(0,1), target.substring(1));
                                else if (parent)
                                    link = new File(target.substring(0,2), target.substring(2));                        
                                else if (current)
                                    link = new File(target.substring(0,1), target.substring(1));
                                else
                                    link = new File (target);
                                
                                if (! file.equals(link))
                                {    
                                    file = resolveFile(link.getAbsoluteFile());
                                }
                            }
                        }
                    }
                }
                catch (Exception e)
                {
                    e.printStackTrace();
                }
        
                //System.err.println("resolve: " + file.getAbsolutePath());
                return file;
            }
            
            
            // commmands
        
            
            /**
             * LinkedFile command resolves a link and prints its final link target name.
             * 
             * @param args
             * @throws IOException
             */
            public static void main (String[] args)
                throws IOException
            {
                try
                {
                    if (args.length == 0)
                    {
                        System.err.println ("usage: LinkedFile <filepath>");
                        System.exit (1);
                    }
                    
                    LinkedFile file = new LinkedFile(args[0]);
                    System.out.println(file);
                }
                catch (Exception e)
                {
                    e.printStackTrace();
                    System.exit (1);
                }
                
            }
        
        
            
            
            // fields
            
            
            protected String source;        // the file source
            protected String target;        // the resolved target path
            
            protected String header;        // the link header, if source is a link
            protected boolean symlink;      // file is a cygwin symlink
            protected boolean shortcut;     // file is a windows shortcut
        
            protected File link;            // the original source file link
        
            
            
            
        
            // instantiators 
        
            
            // constructors
            
            /**
             * Creates a new File instance by converting the given pathname string into
             * an abstract pathname.
             */
            public LinkedFile(String pathname)
            {
                super((resolveFile(new File(pathname))).getAbsolutePath());
                
                this.target = getAbsolutePath();
                this.source = "" + pathname;
                
                this.link = new File(pathname);
                this.symlink = isSymlink(link);
                this.shortcut = isShortcut(link);
            }
        
            
            /**
             * Creates a new File instance from a parent abstract pathname and a child
             * pathname string.
             */
            public LinkedFile(File parent, String child)
            {
                super((resolveFile(new File(parent, child))).getAbsolutePath());
                
                this.target = getAbsolutePath();
                this.source = "" + parent.getName() + "/" + child;
                
                this.link = new File(parent, child);
                this.symlink = isSymlink(link);
                this.shortcut = isShortcut(link);
            }
        
            
            /**
             * Creates a new File instance from a parent pathname string and a child
             * pathname string.
             */
            public LinkedFile(String parent, String child)
            {
                super((resolveFile(new File(parent, child))).getAbsolutePath());
                
                this.target = getAbsolutePath();
                this.source = "" + parent + "/" + child;
                
                this.link = new File(parent, child);
                this.symlink = isSymlink(link);
                this.shortcut = isShortcut(link);
            }
        
            
            /**
             * Creates a new File instance by converting the given file: URI into an
             * abstract pathname.
             */
            public LinkedFile(URI uri)
            {
                super((resolveFile(new File(uri))).getAbsolutePath());
                
                this.target = getAbsolutePath();
                this.source = "" + uri;
                
                this.link = new File(uri);
                this.symlink = isSymlink(link);
                this.shortcut = isShortcut(link);
            }
        
            
            // accessors
        
        
            public boolean isLink()
            {
                return (symlink || shortcut);
            }
        
            
            public boolean isSymlink()
            {
                return (symlink);
            }
        
        
            public boolean isShortcut()
            {
                return (shortcut);
            }
        
            
            public String getSource()
            {
                return source;
            }
        
            public String getTarget()
            {
                return target;
            }
            
            
            
            // overrides
            
            
            public String getName()
            {
                return super.getName().replace('\\', '/');
            }
        
            public String getPath()
            {
                return super.getPath().replace('\\', '/');
            }
            
            public String getAbsolutePath()
            {
                return super.getAbsolutePath().replace('\\', '/');
            }
            
            public String getCanonicalPath()
                throws IOException
            {
                return super.getCanonicalPath().replace('\\', '/');
            }
        
            
            public String toString()
            {
        
        //        System.err.println("");
        //        System.err.println("file: " + getAbsolutePath());
        //        System.err.println("path: " + getPath());        
        //        System.err.println("name: " + getName());
        //                
        //        System.err.println("source: " + getSource());
        //        System.err.println("target: " + getTarget());
        //
        //        System.err.println("exists: " + exists());        
        //        System.err.println("canRead: " + canRead());
        //        System.err.println("canWrite: " + canWrite());
        //        System.err.println("canExecute: " + canExecute());
        //
        //        System.err.println("isDirectory: " + isDirectory());
        //        System.err.println("isSymlink: " + isSymlink());
        //        System.err.println("isShortcut: " + isShortcut());
                
        
                StringBuffer s = new StringBuffer("");
        
                if (isLink())
                    s.append(getSource() + " -> ");
        
                s.append(getTarget());
        
                if (isDirectory())
                    s.append("/");
                
                return s.toString();
            }
        
        }
        
        Show
        FKorning added a comment - If the symlinks really are necessary, here's a java.io.File wrapper that understands cygwin symlinks (and windows shortcuts). package net.muxbus.io; import java.io.BufferedReader; import java.io.File; import java.io.FileReader; import java.io.IOException; import java.net.URI; /** * * @author Francis Korning fkorning@yahoo.ca * @date 2012-04-30 * * @project muxbus muxbus.net * @source sourceforge https: //sourceforge.net/p/muxbus/ * * @copyright 2012 F.Korning * @license LGPL http: //www.gnu.org/copyleft/lesser.html * * @credit Jesse Hager jessehager@iname.com (.lnk shortcut file format). * * * LinkedFile is a java.io.File wrapper that understands both Cygwin Symbolic Links * and Windows Explorer Shorcut Links (.lnk). If a given File is a Link, it tries * to resolve the File by following link paths recursively until a final canonical * target is found, and if the file exists will act as a wrapper for the link * target. A LinkedFile always tracks its source and provides link reflection. * * A common frustration for POSIX platform developers is that Java on windows does * not understand symbolic Links which aren't handled by the windows io native libs. * This means that Cygwin or Interix (aka Winterix or Windows Service for Unix) * integrated java applications that depend on POSIX tools may break when it comes * to resolution of paths with linked files and directories. * * The first workaround strategy, which works where links are predictably created, * is to ensure that cygwin paths and windows paths are equivalent and identical. * For example, you create a circular "/cygwin" link that maps back to the root "/" : * * <code> * <pre> * /cygwin -> / -> \cygwin\ -> C:\cygwin\ * /cygwin/tmp -> /tmp/ -> \cygwin\tmp\ -> C:\cygwin\tmp\ * </pre> * </code> * * But you often have no control over much of the integrated codebase and tools. * If some of the components create symlinks dynamically where java expects real * files and directories this workaround will fail. * * Now the only OS- native IO alternative would be to use Windows NTFS Junctions. * These are completely transparent and work in shells, in Explorer, and for java. * The best implementation would be the junction provided in the SysInternals suite. * You could wrap the cygwin 'ln' command and force it to use a junction instead, * but this breaks some POSIX compatibility and requires Administrator superuser * rights which you may not want to assign to all your running service daemons. * * Next Interix symbolic links and the mklink command are built-in since Windows Vista. * These fall in between as they act like native symlinks in the shell and in java, * look like cygwin links inside the shell, but again they can only be created by the * Administrator, and also break POSIX. Unlike cygwin they do appear as shortcuts in * a Windows Explorer, with the caveat that these are broken because of permissions. * * Note that both strategies only work if the tools create links via the 'ln' command. * But what if the offending integrated tools are in perl or python, or even better yet in * compiled c/c++ code making use of libc calls for which we have no source nor license? * This means that we will need to adopt a strategy to adapt java.io.File to Cygwin libc. * Also, most multi-platform java developers favour Cygwin instead of Microsoft's Interix. * All of the POSIX daemons we're looking at expect cygwin shells, and this compatibility * is our main focus here; we ideally an a model that works without having to wrap 'ln'. * * Now Both Cygwin Symlinks and Explorer Shortcuts are just plain binary files underneath * with some information pointing to their link targets. They have the advantage that * we can parse these and thus this allows us to add some link reflection to java.io. * (This also creates a strange case where windows would have more functionality than * unix, unless one would write a unix java native libc wrapper for link functions). * They also require no special Administrator rights for creation and maintenance. * * A side-effect is that shortcuts created under Windows Explorer have the feature that * a (.lnk) extension is appended, which is obscured from the explorer. As the names * differ, both a Cygwin Symlink and a Windows Shortcut can co-exist in the same place * and make for a consistent interface both in the cygwin shell and Windows Explorer. * * This means that one could well mirror every cygwin symlink with a windows shortcut, * and thus have symbolic links that work even under non-Administrator user shells and * are mirrored by corresponding Windows Shortcuts that work in the Windows Explorer. * Cygwin also comes with its own programmatic 'mkshortcut' command to create Shorcuts * from a shell (which you can't under windows by default - doesn't Windows suck?). * * <code> * <pre> * dir -> /dir/ symlink * dir.lnk -> /dir/ shortcut * file -> /dir/file symlink * file.lnk -> /dir/file shortcut * </pre> * </code> * * Note that synchronizing of Cygwin symlinks and Windows Shortcuts is left out-of-scope. * Our primary concern is to get our POSIX daemons running, not to add sugar to Explorer. * However LinkedFile will support and resolve links for both link formats, with first * preferrence given to Cygwin Symlinks if found, and then to Windows Shortcuts of the * same name (without the .lnk extension), which you may extend for such a facility. * * For those wishing to extend the functionality to support transparent mirroring between * Explorer and the shell, note that you will have to map links to their (.lnk) shortcuts. * Also note that Windows Shortcuts are never interpreted as links by the Cygwin shell. * Most critically note that Windows shortcuts are unpredictable in the shell if moved, * as they seem to only recalculate relative paths after a link is accessed in Explorer. * * * @see cygwin: http: //www.cygwin.com * @see cygwin symlinks: http: //cygwin.com/ml/cygwin/2010-11/msg00394.html * @see interix links: http: //technet.microsoft.com/en-us/library/bb463204.aspx * @see windows shortcuts: http: //ithreats.files.wordpress.com/2009/05/lnk_the_windows_shortcut_file_format.pdf * @see windows junctions: http: //technet.microsoft.com/en-us/sysinternals * */ public class LinkedFile extends java.io.File { public static final String TOK_SYMLINK = "!<symlink>\u00FF\u00FE" ; public static final String TOK_SHORTCUT = "\u004C\u0000\u0000\u0000\u0001\u0014\u0002" ; public static final String TOK_SHORTCUT_SEP = "\u0000" ; public static final int SHORTCUT_MAX = 1024; // shortcuts should probably never be that big public static final int SHORTCUT_OFFSET = 76; // first shortcut section length offset at 0x4C // utilities /** * read a link header. * * both cygwin symlinks and windows shortcuts are plain binary files. * cygwin is faily simple, consisting of magic header and the target. * shortcuts have variable length structures but will not exceed 1kb. */ public static String header (File file) throws IOException { if (file == null ) return null ; String header = null ; if (file.exists() && file.canRead()) { if (file.isFile() || (!file.isDirectory())) { char [] buf = new char [SHORTCUT_MAX]; int length = new BufferedReader( new FileReader(file)).read(buf, 0, SHORTCUT_MAX); header = new String (buf); // System .err.println ( "length: " + length); // System .err.println ( "header: " + header); if (isSymlinkHeader(header)) { // System .err.println( "isSymlink: true " ); } else if (isShortcutHeader(header)) { // System .err.println( "isShortcut: true " ); } else { header = null ; } } } return header; } public static boolean isLinkHeader ( String header) throws IOException { return ( isSymlinkHeader(header) || isShortcutHeader(header) ); } public static boolean isSymlinkHeader ( String header) throws IOException { if (header == null ) return false ; if (header.startsWith(TOK_SYMLINK)) { return true ; } return false ; } public static boolean isShortcutHeader ( String header) throws IOException { if (header == null ) return false ; if (header.startsWith(TOK_SHORTCUT)) { return true ; } return false ; } /** * parse a 2- char short length offset from a position in a windows shortcut file. */ public static int shortcutHeaderOffset ( String header, int pos) { if (header == null ) return 0; char small = header.charAt(pos); char large = header.charAt(pos +1); int offset = small + (large * 16 * 16); // System .err.println ( "offset [" + pos + "] '" + ( short )small + " " + ( short )large + "' : " + offset); return offset; } /** * parse the final link target from a cygwin symlink or windows shortcut. * * cygwin symlink targets are immediately after the header "<!symlink> 0xFF xFE" . * * windows shortcuts start with the header "L 0x00 0x00 0x00 0x01 0x14 0x02" . * the windows shortcut files are organized into sections of variable lengths, * starting after the header at 0x4c (76). the lengths are 2- char short offsets. * windows shortcut final name is at the end of the second "file location" section. */ public static String headerTarget ( String header) { if (header == null ) return null ; String target = null ; try { if (isSymlinkHeader(header)) { StringBuffer buf = new StringBuffer (""); for ( int i = TOK_SYMLINK.length(); i < header.length(); i+=2) { buf.append (header.charAt(i)); } target = buf.toString(); } else if (isShortcutHeader(header)) { // first get the length of the next "item id" section int a = shortcutHeaderOffset(header, SHORTCUT_OFFSET); // now get the length of the next "file location" section int b = shortcutHeaderOffset(header, SHORTCUT_OFFSET + a + 2); // the final name is the last field of the "file location" section int j = SHORTCUT_OFFSET + a + b; int i = header.lastIndexOf(TOK_SHORTCUT_SEP, j - 2) + 1; target = header.substring (i,j); } } catch (IOException e) {} // System .err.println ( "target: " + target); return target; } public static boolean isLink (File file) { if (file == null ) return false ; try { if (file.exists() && file.canRead()) { if (file.isFile() || (!file.isDirectory())) { String header = header(file); return isLinkHeader(header); } } } catch (IOException e) {} return false ; } public static boolean isSymlink (File file) { if (file == null ) return false ; try { if (file.exists() && file.canRead()) { if (file.isFile() || (!file.isDirectory())) { String header = header(file); return isSymlinkHeader(header); } } } catch (IOException e) {} return false ; } public static boolean isShortcut (File file) { if (file == null ) return false ; try { if (file.exists() && file.canRead()) { if (file.isFile() || (!file.isDirectory())) { String header = header(file); return isShortcutHeader(header); } } } catch (IOException e) {} return false ; } /** * Resolve a target by recursively traversing links until the canonical target. * If the file is not a link, return the file's canonical AbsolutePath. */ public static String resolveTarget (File file) { String target = file.getAbsolutePath(); // System .err.println( "\path: " + target); try { if (file.exists() && file.canRead()) { if (file.isFile() || (!file.isDirectory())) { String header = header(file); if (isLinkHeader(header)) { target = headerTarget(header); boolean absolute = (target.substring(1)).startsWith( ":" ); boolean logical = (target.startsWith( "/" ) || target.startsWith( "\\" )); boolean parent = (target.startsWith( "../" ) || target.startsWith( "..\\" )); boolean current = (target.startsWith( "./" ) || target.startsWith( ".\\" )); File link = null ; if (absolute) link = new File (target.substring(0,2), target.substring(2)); else if (logical) link = new File(target.substring(0,1), target.substring(1)); else if (parent) link = new File(target.substring(0,2), target.substring(2)); else if (current) link = new File(target.substring(0,1), target.substring(1)); else link = new File (target); if (! file.equals(link)) { target = resolveTarget(link); } } } } } catch (Exception e) { e.printStackTrace(); } // System .err.println( "resolve: " + target); return target; } /** * * Resolve a file by recursively traversing links until the canonical target. * If the file is not a link, return the file itself. * @param file * @ return */ public static File resolveFile (File file) { // System .err.println( "file: " + file.getAbsolutePath()); try { if (file.exists() && file.canRead()) { if (file.isFile() || (!file.isDirectory())) { String header = header(file); if (isLinkHeader(header)) { String target = headerTarget(header); boolean absolute = (target.substring(1)).startsWith( ":" ); boolean logical = (target.startsWith( "/" ) || target.startsWith( "\\" )); boolean parent = (target.startsWith( "../" ) || target.startsWith( "..\\" )); boolean current = (target.startsWith( "./" ) || target.startsWith( ".\\" )); File link = null ; if (absolute) link = new File (target.substring(0,2), target.substring(2)); else if (logical) link = new File(target.substring(0,1), target.substring(1)); else if (parent) link = new File(target.substring(0,2), target.substring(2)); else if (current) link = new File(target.substring(0,1), target.substring(1)); else link = new File (target); if (! file.equals(link)) { file = resolveFile(link.getAbsoluteFile()); } } } } } catch (Exception e) { e.printStackTrace(); } // System .err.println( "resolve: " + file.getAbsolutePath()); return file; } // commmands /** * LinkedFile command resolves a link and prints its final link target name. * * @param args * @ throws IOException */ public static void main ( String [] args) throws IOException { try { if (args.length == 0) { System .err.println ( "usage: LinkedFile <filepath>" ); System .exit (1); } LinkedFile file = new LinkedFile(args[0]); System .out.println(file); } catch (Exception e) { e.printStackTrace(); System .exit (1); } } // fields protected String source; // the file source protected String target; // the resolved target path protected String header; // the link header, if source is a link protected boolean symlink; // file is a cygwin symlink protected boolean shortcut; // file is a windows shortcut protected File link; // the original source file link // instantiators // constructors /** * Creates a new File instance by converting the given pathname string into * an abstract pathname. */ public LinkedFile( String pathname) { super ((resolveFile( new File(pathname))).getAbsolutePath()); this .target = getAbsolutePath(); this .source = "" + pathname; this .link = new File(pathname); this .symlink = isSymlink(link); this .shortcut = isShortcut(link); } /** * Creates a new File instance from a parent abstract pathname and a child * pathname string. */ public LinkedFile(File parent, String child) { super ((resolveFile( new File(parent, child))).getAbsolutePath()); this .target = getAbsolutePath(); this .source = "" + parent.getName() + " /" + child; this .link = new File(parent, child); this .symlink = isSymlink(link); this .shortcut = isShortcut(link); } /** * Creates a new File instance from a parent pathname string and a child * pathname string. */ public LinkedFile( String parent, String child) { super ((resolveFile( new File(parent, child))).getAbsolutePath()); this .target = getAbsolutePath(); this .source = "" + parent + " /" + child; this .link = new File(parent, child); this .symlink = isSymlink(link); this .shortcut = isShortcut(link); } /** * Creates a new File instance by converting the given file: URI into an * abstract pathname. */ public LinkedFile(URI uri) { super ((resolveFile( new File(uri))).getAbsolutePath()); this .target = getAbsolutePath(); this .source = "" + uri; this .link = new File(uri); this .symlink = isSymlink(link); this .shortcut = isShortcut(link); } // accessors public boolean isLink() { return (symlink || shortcut); } public boolean isSymlink() { return (symlink); } public boolean isShortcut() { return (shortcut); } public String getSource() { return source; } public String getTarget() { return target; } // overrides public String getName() { return super .getName().replace('\\', '/'); } public String getPath() { return super .getPath().replace('\\', '/'); } public String getAbsolutePath() { return super .getAbsolutePath().replace('\\', '/'); } public String getCanonicalPath() throws IOException { return super .getCanonicalPath().replace('\\', '/'); } public String toString() { // System .err.println(""); // System .err.println( "file: " + getAbsolutePath()); // System .err.println( "path: " + getPath()); // System .err.println( "name: " + getName()); // // System .err.println( "source: " + getSource()); // System .err.println( "target: " + getTarget()); // // System .err.println( "exists: " + exists()); // System .err.println( "canRead: " + canRead()); // System .err.println( "canWrite: " + canWrite()); // System .err.println( "canExecute: " + canExecute()); // // System .err.println( "isDirectory: " + isDirectory()); // System .err.println( "isSymlink: " + isSymlink()); // System .err.println( "isShortcut: " + isShortcut()); StringBuffer s = new StringBuffer (""); if (isLink()) s.append(getSource() + " -> " ); s.append(getTarget()); if (isDirectory()) s.append( "/" ); return s.toString(); } }
        Hide
        FKorning added a comment -

        I've managed to get this working to the point where jobs are dispatched, tasks executed, and results compiled.

        However we still need to get the servlets to understand cygwin symlinks. I have no idea how to do this in Jetty.

        These two links show how to allow Tomcat and jetty to follow symlinks, but I don't know if this works in cygwin.

        Otherwise we'll have to open up the jetty code and replace java.io.File with org.apache.hadoop.fs.LinkedFile.

        Show
        FKorning added a comment - I've managed to get this working to the point where jobs are dispatched, tasks executed, and results compiled. http://en.wikisource.org/wiki/User:Fkorning/Code/Haddoop-on-Cygwin However we still need to get the servlets to understand cygwin symlinks. I have no idea how to do this in Jetty. These two links show how to allow Tomcat and jetty to follow symlinks, but I don't know if this works in cygwin. http://www.lamoree.com/machblog/index.cfm?event=showEntry&entryId=A2F0ED76-A500-41A6-A1DFDE0D1996F925 http://stackoverflow.com/questions/315093/configure-symlinks-for-single-directory-in-tomcat Otherwise we'll have to open up the jetty code and replace java.io.File with org.apache.hadoop.fs.LinkedFile.
        Show
        FKorning added a comment - make that: http://en.wikisource.org/wiki/User:Fkorning/Code/Hadoop-on-Cygwin
        Hide
        JY chen added a comment -

        I found there are some problems the LinkedFile.java above in my windows 7. When the stdout/stderr/syslog are created under tmp folder and hadoop create a linux symlink to corresponding files in local logs/userlogs/job_2XXXXXXXX/attempt_XXXXX/stdout. However, this folder
        logs/userlogs/job_2XXXXXXXX/attempt_XXXXX is a symlink and stdout is a file in a symlink directory.

        I modify LinkeFile.java as

        ........................

        public LinkedFile(File parent, String child)
        {
        //work around when parent is linked symbol
        super(resolveFile(new File(resolveFile(new File(parent.toString())).getAbsolutePath()
        , child)).getAbsolutePath());
        this.target = getAbsolutePath();

        ........................

        public LinkedFile(String parent, String child)
        {
        //work around when parent is linked symbol
        super(resolveFile(new File(resolveFile(new File(parent)).getAbsolutePath()

        ........................

        that's only work around for this case

        Show
        JY chen added a comment - I found there are some problems the LinkedFile.java above in my windows 7. When the stdout/stderr/syslog are created under tmp folder and hadoop create a linux symlink to corresponding files in local logs/userlogs/job_2XXXXXXXX/attempt_XXXXX/stdout. However, this folder logs/userlogs/job_2XXXXXXXX/attempt_XXXXX is a symlink and stdout is a file in a symlink directory. I modify LinkeFile.java as ........................ public LinkedFile(File parent, String child) { //work around when parent is linked symbol super(resolveFile(new File(resolveFile(new File(parent.toString())).getAbsolutePath() , child)).getAbsolutePath()); this.target = getAbsolutePath(); ........................ public LinkedFile(String parent, String child) { //work around when parent is linked symbol super(resolveFile(new File(resolveFile(new File(parent)).getAbsolutePath() ........................ that's only work around for this case
        Hide
        Harsh J added a comment -

        For Windows, since the mainstream branch does not support it actively, am closing this as a Won't Fix.

        I'm certain the same issue does not happen on the branch-1-win 1.x branch (or the branch-trunk-win branch), and I urge you to use that instead if you wish to continue using Windows for development or other usage. Find the Windows-optimized sources at http://svn.apache.org/repos/asf/hadoop/common/branches/branch-1-win/ or http://svn.apache.org/repos/asf/hadoop/common/branches/branch-trunk-win/.

        Show
        Harsh J added a comment - For Windows, since the mainstream branch does not support it actively, am closing this as a Won't Fix. I'm certain the same issue does not happen on the branch-1-win 1.x branch (or the branch-trunk-win branch), and I urge you to use that instead if you wish to continue using Windows for development or other usage. Find the Windows-optimized sources at http://svn.apache.org/repos/asf/hadoop/common/branches/branch-1-win/ or http://svn.apache.org/repos/asf/hadoop/common/branches/branch-trunk-win/ .
        Hide
        FKorning added a comment -

        Yes, You'll need to make LinkedFile recursively traverse through symlinks.
        I just did a quick hack to get it to resolve the last basename as a link.

        Show
        FKorning added a comment - Yes, You'll need to make LinkedFile recursively traverse through symlinks. I just did a quick hack to get it to resolve the last basename as a link.

          People

          • Assignee:
            Unassigned
            Reporter:
            tim.wu
          • Votes:
            2 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development