Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-21718

Heavy log of type: "Skipping partition based on stats ..."

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Trivial
    • Resolution: Fixed
    • 2.1.0
    • 2.3.0
    • SQL
    • None
    • Linux

    Description

      My spark job have three partions. This cause many logs entry of type:
      "Skipping partition based on stats ..." in SPARK_WORKDIR/<my_app_id>/stderr

      We should add a check to disable this kind of working or move the loglevel from Info to Debug

      Attachments

        Activity

          People

            srowen Sean R. Owen
            meox Gian Lorenzo Meocci
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: