Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-21776

How to use the memory-mapped file on Spark??

    XMLWordPrintableJSON

    Details

      Description

      In generation, we have to use the Spark full quantity loaded HBase table based on one dimension table to generate business, because the base table is total quantity loaded, the memory will pressure is very big, I want to see if the Spark can use this way to deal with memory mapped file?Is there such a mechanism?How do you use it?
      And I found in the Spark a parameter: spark.storage.memoryMapThreshold=2m, is not very clear what this parameter is used for?
      There is a putBytes and getBytes method in DiskStore.scala with Spark source code, is this the memory-mapped file mentioned above?How to understand?
      Let me know if you have any trouble..

      Wish to You!!

        Attachments

        1. screenshot-1.png
          93 kB
          Aaron-Mhs
        2. screenshot-2.png
          14 kB
          Aaron-Mhs

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              扎啤 Aaron-Mhs
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Time Tracking

                Estimated:
                Original Estimate - 12h
                12h
                Remaining:
                Remaining Estimate - 12h
                12h
                Logged:
                Time Spent - Not Specified
                Not Specified