Uploaded image for project: 'Kudu'
  1. Kudu
  2. KUDU-2812

Problem with error reporting in kudu-backup

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.9.0
    • Fix Version/s: 1.10.0
    • Component/s: backup
    • Labels:

      Description

      In KuduRestore.scala we have code like

                // Fail the task if there are any errors.
                val errorCount = session.getPendingErrors.getRowErrors.length
                if (errorCount > 0) {
                  val errors =
                    session.getPendingErrors.getRowErrors.take(5).map(_.getErrorStatus).mkString
                  throw new RuntimeException(
                    s"failed to write $errorCount rows from DataFrame to Kudu; sample errors: $errors")
                }
      

      There's similar code in KuduContext.scala:

            val errorCount = pendingErrors.getRowErrors.length
            if (errorCount > 0) {
              val errors =
                pendingErrors.getRowErrors.take(5).map(_.getErrorStatus).mkString
              throw new RuntimeException(
                s"failed to write $errorCount rows from DataFrame to Kudu; sample errors: $errors")
            }
      

      I've seen the former fail to print any sample errors. Taking a reference to session.getPendingErrors.getRowErrors and using that through fixes this, so it seems like there's some TOCTOU problem that can occur, probably because multiple batches can be in flight at once.

      The latter is most likely vulnerable to this as well.

      This issue made diagnosing KUDU-2809 harder.

        Attachments

          Activity

            People

            • Assignee:
              granthenke Grant Henke
              Reporter:
              wdberkeley William Berkeley
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: