Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-34114

Parse error while $internal.application.program-args contains '#' in Yarn/K8s Application Mode

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    Description

      When run job by K8s or Yarn Application Mode use org.apache.flink.configuration.GlobalConfiguration#loadYAMLResource method parse config。

       

      private static Configuration loadYAMLResource(File file) {
          final Configuration config = new Configuration();
      
          try (BufferedReader reader =
                  new BufferedReader(new InputStreamReader(new FileInputStream(file)))) {
      
              String line;
              int lineNo = 0;
              while ((line = reader.readLine()) != null) {
                  lineNo++;
                  // 1. check for comments
                  String[] comments = line.split("#", 2);
                  String conf = comments[0].trim();
      
                  // 2. get key and value
                  if (conf.length() > 0) {
                      String[] kv = conf.split(": ", 2);
      
                      // skip line with no valid key-value pair
                      if (kv.length == 1) {
                          LOG.warn(
                                  "Error while trying to split key and value in configuration file "
                                          + file
                                          + ":"
                                          + lineNo
                                          + ": Line is not a key-value pair (missing space after ':'?)");
                          continue;
                      }
      
                      String key = kv[0].trim();
                      String value = kv[1].trim();
      
                      // sanity check
                      if (key.length() == 0 || value.length() == 0) {
                          LOG.warn(
                                  "Error after splitting key and value in configuration file "
                                          + file
                                          + ":"
                                          + lineNo
                                          + ": Key or value was empty");
                          continue;
                      }
      
                      config.setString(key, value);
                  }
              }
          } catch (IOException e) {
              throw new RuntimeException("Error parsing YAML configuration.", e);
          }
      
          return config;
      } 

      if config value contains '#' like

       

      $internal.application.program-args: '{test:test#jsonstring}'

       

       

      the following code is not correct

      line.split("#", 2) 

      To fix this,i think we should import snakeyaml to fix complex situation

      Attachments

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            Unassigned Unassigned
            ChunJi jonasjc
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment