XMLWordPrintableJSON

    Details

    • Target Version/s:

      Description

      Optimize read performance for the following scenarios

      1. Read small files completely
        Files that are of size smaller than the read buffer size can be considered as small files. In case of such files it would be better to read the full file into the AbfsInputStream buffer.
      2. Read last block if the read is for footer
        If the read is for the last 8 bytes, read the full file.
        This will optimize reads for parquet files. Parquet file format

      Both these optimizations will be present under configs as follows

      1. fs.azure.read.smallfilescompletely
      2. fs.azure.read.optimizefooterread

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                bilahari.th Bilahari T H
                Reporter:
                bilahari.th Bilahari T H
              • Votes:
                0 Vote for this issue
                Watchers:
                7 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 12.5h
                  12.5h