Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-6176

Add JARs to CLASSPATH deterministically

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.2.0
    • Fix Version/s: 1.3.0, 1.2.2
    • Component/s: Core
    • Labels:
      None

      Description

      The config.sh script uses the following shell-script function to build the FLINK_CLASSPATH variable from a listing of JAR files in the $FLINK_LIB_DIR directory:

      constructFlinkClassPath() {
      
          while read -d '' -r jarfile ; do
              if [[ $FLINK_CLASSPATH = "" ]]; then
                  FLINK_CLASSPATH="$jarfile";
              else
                  FLINK_CLASSPATH="$FLINK_CLASSPATH":"$jarfile"
              fi
          done < <(find "$FLINK_LIB_DIR" ! -type d -name '*.jar' -print0)
      
          echo $FLINK_CLASSPATH
      }
      

      The find command as specified will return files in directory-order, which varies by OS and filesystem.

      The inconsistent ordering of directory contents caused problems for me when installing a Flink Docker image onto new machine with a newer version of Docker and different filesystem (UFS). The differences in the Docker filesystem implementation led to different ordering of the directory contents; this affected the FLINK_CLASSPATH ordering and generated very puzzling NoClassNotFoundException errors when running my Flink application.

      This should be addressed by deterministically ordering JAR files added to the FLINK_CLASSPATH.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                greghogan Greg Hogan
                Reporter:
                skidder Scott Kidder
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: