Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-25696

The storage memory displayed on spark Application UI is incorrect.

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.3.2
    • Fix Version/s: 3.0.0
    • Component/s: Spark Core, Web UI
    • Labels:
    • Target Version/s:
    • Docs Text:
      In Spark 3.0, the web UI and log statements now consistently report units in KiB, MiB, etc, (i.e. multiples of 1024) rather than KB and MB (i.e. multiples of 1000). For example, 1024000 bytes is now displayed as 1000 KiB rather than 1024 KB.

      Description

      In the reported heartbeat information, the unit of the memory data is bytes, which is converted by the formatBytes() function in the utils.js file before being displayed in the interface. The cardinality of the unit conversion in the formatBytes function is 1000, which should be 1024.

      function formatBytes(bytes, type)

      {    if (type !== 'display') return bytes;    if (bytes == 0) return '0.0 B';    var k = 1000;    var dm = 1;    var sizes = ['B', 'KB', 'MB', 'GB', 'TB', 'PB', 'EB', 'ZB', 'YB'];    var i = Math.floor(Math.log(bytes) / Math.log(k)); return parseFloat((bytes / Math.pow(k, i)).toFixed(dm)) + ' ' + sizes[i]; }

       

        Attachments

          Activity

            People

            • Assignee:
              hantiantian hantiantian
              Reporter:
              hantiantian hantiantian
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: