Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-9697

Support priority based scratch directory selection

    XMLWordPrintableJSON

Details

    • Task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • Impala 4.0.0
    • None
    • None
    • ghx-label-5

    Description

      The `‑‑scratch_dirs` startup flag uses the given scratch directories in a round robin manner. This may not always be ideal since these directories could come from different class of storage system volumes having different performance characteristics (SSD vs HDD, local storage vs network attached storage, etc.). Giving user an option to configure the priority of their scratch directories could help them optimize their workload based on their storage system configuration.

      One possible way could be that the user pass the priority as part of the `–scratch_dirs` startup flag using <directory>:<spill_priority>. The directories will be selected for spilling based on their priorities and if multiple directories have the same priority then they will be selected in a round robin fashion. In the below example, dir1 will be used as a spill victim until its full and then dir2, dir3, and dir4 will be used in a round robin fashion.

      ‑‑scratch_dirs="dir1:200GB:0, dir2:1024GB:1, dir3:1024GB:1, dir4:1024GB:1"

       

      Attachments

        Activity

          People

            arawat Abhishek Rawat
            arawat Abhishek Rawat
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: