Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-17858

Yarn mode, windows and linux environment should be interlinked

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Minor
    • Resolution: Later
    • Affects Version/s: 1.10.0
    • Fix Version/s: None
    • Component/s: Deployment / YARN
    • Labels:
      None

      Description

      my flink version: 1.10.0

      my yarn version: 2.7.6

      When I use windows idea to try to submit a job to Linux's yarn cluster using java, am container throws an exception "Error: Could not find or load main class org.apache.flink.yarn.entrypoint.YarnJobClusterEntrypoint".

      Then find out that the CLASSPATH and _FLINK_CLASSPATH delimiter of the container startup script "launch_container.sh" are different, windows is ";", linux is ":", so the container cannot get the dependent packages.

      I think the operating environment should be left to Yarn to judge.

      Locate in the flink-yarn module
      1. The startAppMaster method of org.apache.flink.yarn.YarnClusterDescriptor
      Change "File.pathSeparator" to "ApplicationConstants.CLASS_PATH_SEPARATOR"
      Yarn will replace special characters according to the environment, but this only solves _FLINK_CLASSPATH

       

      private ApplicationReport startAppMaster(.....) {
          .....
          StringBuilder classPathBuilder = new StringBuilder();
          if (userJarInclusion == YarnConfigOptions.UserJarInclusion.FIRST) {
             for (String userClassPath : userClassPaths) {
                // here
                classPathBuilder.append(userClassPath).append(File.pathSeparator);
             }
          }
          for (String classPath : systemClassPaths) {
             // here
             classPathBuilder.append(classPath).append(File.pathSeparator);
          }    
          // Setup jar for ApplicationMaster
          .....    
          // here 
          classPathBuilder.append(flinkJarPath.getName()).append(File.pathSeparator);
          .....
      }

       

      2. The addToEnvironment method of org.apache.flink.yarn.Utils
      Change "File.pathSeparator" to "ApplicationConstants.CLASS_PATH_SEPARATOR"
      Can solve the CLASSPATH

      public static void addToEnvironment(Map<String, String> environment,
            String variable, String value) {
         String val = environment.get(variable);
         if (val == null) {
            val = value;
         } else {
            // here
            val = val + File.pathSeparator + value;
         }
         environment.put(StringInterner.weakIntern(variable),
               StringInterner.weakIntern(val));
      }
      

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              barneyye ChengWei Ye
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: