Uploaded image for project: 'Spot'
  1. Spot
  2. SPOT-19

[Setup] Zero prefix on Hive partition names

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Minor
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Labels:
    • Environment:
      Using spot branch; MapR 5.2; Hive 1.2, HBase 1.1.1

      Description

      Spot-ml looks for data in days and months numbers prefixed by zero when the number is a single digit:

      Exception in thread "main" java.lang.AssertionError: assertion failed: No predefined schema found, and no Parquet data files or summary files found under maprfs:///user/spot/flow/hive/y=2016/m=12/d=01. <<<<---- zero pad

      While spot-ingest has stored them in Hive without the zero prefix:

      hadoop fs -ls /user/spot/flow/hive/y=2016/m=12/
      drwxr-xrwx - spot spot 24 2016-12-01 18:12 /user/spot/flow/hive/y=2016/m=12/d=1 <<<---- No zero pad

      If the directory/partition is manually renamed in HDFS, and 'MSCK REPAIR TABLE flow;' is run manually from Hive, ML then will run for that day.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                natedogs911 nathanael Smith
                Reporter:
                thealy Terry Healy
              • Votes:
                1 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:

                  Time Tracking

                  Estimated:
                  Original Estimate - 48h
                  48h
                  Remaining:
                  Remaining Estimate - 48h
                  48h
                  Logged:
                  Time Spent - Not Specified
                  Not Specified