Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-21024

Dynamic properties get exposed to job's main method if user parameters are passed

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    Description

      A bug was identified in the user ML by Alexey exposing dynamic properties into the job user code.

      I was able to reproduce this issue by slightly adapting the WordCount example (org.apache.flink.streaming.examples.wordcount.WordCount2 in attached WordCount.jar ).

      Initiating a standalone job without using the --input parameter will result in printing an empty array:

      ./bin/standalone-job.sh start --job-classname org.apache.flink.streaming.examples.wordcount.WordCount2
      

      The corresponding *.out file looks like this:

      []
      Executing WordCount2 example with default input data set.
      Use --input to specify file input.
      Printing result to stdout. Use --output to specify output path.
      

      In contrast, initiating the standalone job using the --input parameter will expose the dynamic properties:

      ./bin/standalone-job.sh start --job-classname org.apache.flink.streaming.examples.wordcount.WordCount2 --input /opt/flink/config/flink-conf.yaml
      

      Resulting in the following output:

      [--input, /opt/flink/config/flink-conf.yaml, -D, jobmanager.memory.off-heap.size=134217728b, -D, jobmanager.memory.jvm-overhead.min=201326592b, -D, jobmanager.memory.jvm-metaspace.size=268435456b, -D, jobmanager.memory.heap.size=1073741824b, -D, jobmanager.memory.jvm-overhead.max=201326592b]
      Printing result to stdout. Use --output to specify output path.
      

      Interestingly, this cannot be reproduced on a local standalone session cluster.

      Attachments

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            xtsong Xintong Song
            mapohl Matthias Pohl
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment