Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-34114

Parse error while $internal.application.program-args contains '#' in Yarn/K8s Application Mode

    XMLWordPrintableJSON

Details

    Description

      When run job by K8s or Yarn Application Mode use org.apache.flink.configuration.GlobalConfiguration#loadYAMLResource method parse config。

       

      private static Configuration loadYAMLResource(File file) {
          final Configuration config = new Configuration();
      
          try (BufferedReader reader =
                  new BufferedReader(new InputStreamReader(new FileInputStream(file)))) {
      
              String line;
              int lineNo = 0;
              while ((line = reader.readLine()) != null) {
                  lineNo++;
                  // 1. check for comments
                  String[] comments = line.split("#", 2);
                  String conf = comments[0].trim();
      
                  // 2. get key and value
                  if (conf.length() > 0) {
                      String[] kv = conf.split(": ", 2);
      
                      // skip line with no valid key-value pair
                      if (kv.length == 1) {
                          LOG.warn(
                                  "Error while trying to split key and value in configuration file "
                                          + file
                                          + ":"
                                          + lineNo
                                          + ": Line is not a key-value pair (missing space after ':'?)");
                          continue;
                      }
      
                      String key = kv[0].trim();
                      String value = kv[1].trim();
      
                      // sanity check
                      if (key.length() == 0 || value.length() == 0) {
                          LOG.warn(
                                  "Error after splitting key and value in configuration file "
                                          + file
                                          + ":"
                                          + lineNo
                                          + ": Key or value was empty");
                          continue;
                      }
      
                      config.setString(key, value);
                  }
              }
          } catch (IOException e) {
              throw new RuntimeException("Error parsing YAML configuration.", e);
          }
      
          return config;
      } 

      if config value contains '#' like

       

      $internal.application.program-args: '{test:test#jsonstring}'

       

       

      the following code is not correct

      line.split("#", 2) 

      To fix this,i think we should import snakeyaml to fix complex situation

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              ChunJi jonasjc
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: