Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-7204

Call sites in UI are not accurate for DataFrame operations

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Critical
    • Resolution: Fixed
    • 1.3.1
    • 1.3.2, 1.4.0
    • SQL
    • None

    Description

      Spark core computes callsites by climbing up the stack until we reach the stack frame at the boundary of user code and spark code. The way we compute whether a given frame is internal (Spark) or user code does not work correctly with the new dataframe API.

      Once the scope work goes in, we'll have a nicer way to express units of operator scope, but until then there is a simple fix where we just make sure the SQL internal classes are also skipped as we climb up the stack.

      Attachments

        Activity

          People

            pwendell Patrick Wendell
            pwendell Patrick Wendell
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: