Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-28366

Logging in driver when loading single large unsplittable file

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Minor
    • Resolution: Fixed
    • 3.0.0
    • 3.0.0
    • Spark Core
    • None

    Description

      For a large gzipped file, since they are not splittable, spark have to use only one partition task to read and decompress it. This could be very slow.

      We should log for this case in driver side.

      Attachments

        Issue Links

          Activity

            People

              weichenxu123 Weichen Xu
              weichenxu123 Weichen Xu
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: