Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-6078

IMPALA query fails if a scratch directory goes bad with a disk failure

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Duplicate
    • Affects Version/s: Impala 2.9.0
    • Fix Version/s: Impala 2.9.0
    • Component/s: Backend
    • Labels:
      None
    • Epic Color:
      ghx-label-3

      Description

      Impala was configured with two scratch directories on 2 different disks.

      We simulated disk failure on one of the Scratch directories and it became inaccessible

      In such scenarios, We get the following error when a query tries to spill to the scratch directories and the query fails:

      >>>>>>>>>
      > select avg(t1.a1), max(t2.a2) from impala_disk_test.impala_tst1 as t1 full join impala_disk_test.impala_tst1 as t2 ON t1.a1=t2.a1;
      Query: select avg(t1.a1), max(t2.a2) from impala_disk_test.impala_tst1 as t1 full join impala_disk_test.impala_tst1 as t2 ON t1.a1=t2.a1
      Cancelling Query
      Application Exception : Default TException.
      t2.a1;t.impala_tst1 as t1 full join impala_disk_test.impala_tst1 as t2 ON t1.a1=t
      Query: select avg(t1.a1), max(t2.a2) from impala_disk_test.impala_tst1 as t1 full join impala_disk_test.impala_tst1 as t2 ON t1.a1=t2.a1
      WARNINGS:
      No usable temporary files: space could not be allocated on any temporary device.
      Create file /srv/impala2/impala/impala-scratch/5b431fe3571b9bf1:ef5b367fb74fd1b9_597f261f-792e-4ade-8b32-f4198c945142 failed with errno=5 description=Error(5): Input/output error

      No usable temporary files: space could not be allocated on any temporary device.
      >>>>>>>>>

      It does not look good from HA point of view to have an Impala daemon in the cluster that will sometimes fail queries just because it has a disk failed a few hour ago.

      If a disk fails during a query there is really nothing you can do, This request is to address query failures that started after the disk has failed.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                kamal.chandrasekaran kamal Chandrasekaran
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: