Uploaded image for project: 'Apache Hudi'
  1. Apache Hudi
  2. HUDI-498 End to end to check HiveIncrementalPuller
  3. HUDI-485

Check for where clause is wrong in HiveIncrementalPuller

    XMLWordPrintableJSON

    Details

      Description

      HiveIncrementalPuller checks the clause in incrementalSqlFile like this -> 

      if (!incrementalSQL.contains("`_hoodie_commit_time` > '%targetBasePath'"))

      { LOG.info("Incremental SQL : " + incrementalSQL + " does not contain `_hoodie_commit_time` > %targetBasePath. Please add " + "this clause for incremental to work properly."); throw new HoodieIncrementalPullSQLException( "Incremental SQL does not have clause `_hoodie_commit_time` > '%targetBasePath', which " + "means its not pulling incrementally"); }

      Basically we are trying to add a placeholder here which is later replaced with config.fromCommitTime here - 

      incrementalPullSQLtemplate.add("incrementalSQL", String.format(incrementalSQL, config.fromCommitTime));

      Hence, the above check needs to replaced with `_hoodie_commit_time` > %s

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Pratyaksh Pratyaksh Sharma
                Reporter:
                Pratyaksh Pratyaksh Sharma
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated: