Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-15080

Fix the issue in reading persistent memory cached data with an offset

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.3.0, 3.1.4, 3.2.2
    • Component/s: caching, datanode
    • Labels:
      None

      Description

      Some applications can read a segment of pmem cache with an offset specified. The previous implementation for pmem cache read with DirectByteBuffer didn't cover this situation.

      Let me explain further. In our test, we used spark SQL to run some TPC-DS workload to read the cache data and hits read exception. This was due to the missed seek offset arg, which is used in spark SQL to read data packet by packet.

        Attachments

        1. HDFS-15080-000.patch
          1.0 kB
          Feilong He
        2. HDFS-15080-branch-3.1-000.patch
          1.0 kB
          Feilong He
        3. HDFS-15080-branch-3.2-000.patch
          1.0 kB
          Feilong He

          Issue Links

            Activity

              People

              • Assignee:
                PhiloHe Feilong He
                Reporter:
                PhiloHe Feilong He
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: