Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-6176

Add JARs to CLASSPATH deterministically

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 1.2.0
    • 1.2.2, 1.3.0
    • None
    • None

    Description

      The config.sh script uses the following shell-script function to build the FLINK_CLASSPATH variable from a listing of JAR files in the $FLINK_LIB_DIR directory:

      constructFlinkClassPath() {
      
          while read -d '' -r jarfile ; do
              if [[ $FLINK_CLASSPATH = "" ]]; then
                  FLINK_CLASSPATH="$jarfile";
              else
                  FLINK_CLASSPATH="$FLINK_CLASSPATH":"$jarfile"
              fi
          done < <(find "$FLINK_LIB_DIR" ! -type d -name '*.jar' -print0)
      
          echo $FLINK_CLASSPATH
      }
      

      The find command as specified will return files in directory-order, which varies by OS and filesystem.

      The inconsistent ordering of directory contents caused problems for me when installing a Flink Docker image onto new machine with a newer version of Docker and different filesystem (UFS). The differences in the Docker filesystem implementation led to different ordering of the directory contents; this affected the FLINK_CLASSPATH ordering and generated very puzzling NoClassNotFoundException errors when running my Flink application.

      This should be addressed by deterministically ordering JAR files added to the FLINK_CLASSPATH.

      Attachments

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            greghogan Greg Hogan
            skidder Scott Kidder
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment