Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-12136

rddToFileName does not properly handle prefix and suffix parameters

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Minor
    • Resolution: Fixed
    • None
    • 1.6.0, 2.0.0
    • DStreams

    Description

      See code here: https://github.com/apache/spark/blob/master/streaming/src/main/scala/org/apache/spark/streaming/StreamingContext.scala#L894

      private[streaming] def rddToFileName[T](prefix: String, suffix: String, time: Time): String = {
      if (prefix == null)

      { time.milliseconds.toString }

      else if (suffix == null || suffix.length ==0)

      { prefix + "-" + time.milliseconds }

      else

      { prefix + "-" + time.milliseconds + "." + suffix }

      }

      This code does not seem to properly handle the cases where the prefix is null, but suffix is not null - the suffix should be used but is not.

      Also, the check for length == 0 is only applied to the suffix, bot the prefix. It seems the check should be consistent between the two.

      Is there a reason not to address these two issues and change the code?

      Attachments

        Activity

          People

            mengbo Bo Meng
            bwebb Brian Webb
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: