Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-14134

Idempotent operations throwing RemoteException should not be retried by the client

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Patch Available
    • Priority: Critical
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: hdfs, hdfs-client, ipc
    • Labels:
      None

      Description

      Currently, some operations that throw IOException on the NameNode are evaluated by RetryPolicy as FAILOVER_AND_RETRY, but they should just fail fast.

      For example, when calling getXAttr("user.some_attr", file") where the file does not have the attribute, NN throws an IOException with message "could not find attr". The current client retry policy determines the action for that to be FAILOVER_AND_RETRY. The client then fails over and retries until it reaches the maximum number of retries. Supposedly, the client should be able to tell that this exception is normal and fail fast.

      Moreover, even if the action was FAIL, the RetryInvocationHandler looks at all the retry actions from all requests, and FAILOVER_AND_RETRY takes precedence over FAIL action.

        Attachments

        1. HDFS-14134_retrypolicy_change_proposal_1.pdf
          79 kB
          Lukas Majercak
        2. HDFS-14134_retrypolicy_change_proposal.pdf
          80 kB
          Lukas Majercak
        3. HDFS-14134.001.patch
          6 kB
          Lukas Majercak
        4. HDFS-14134.002.patch
          8 kB
          Lukas Majercak
        5. HDFS-14134.003.patch
          27 kB
          Lukas Majercak
        6. HDFS-14134.004.patch
          30 kB
          Lukas Majercak
        7. HDFS-14134.005.patch
          30 kB
          Lukas Majercak
        8. HDFS-14134.006.patch
          32 kB
          Lukas Majercak
        9. HDFS-14134.007.patch
          32 kB
          Lukas Majercak

          Issue Links

            Activity

              People

              • Assignee:
                lukmajercak Lukas Majercak
                Reporter:
                lukmajercak Lukas Majercak
              • Votes:
                0 Vote for this issue
                Watchers:
                10 Start watching this issue

                Dates

                • Created:
                  Updated: