Uploaded image for project: 'Crunch'
  1. Crunch
  2. CRUNCH-657

Can't activate Snappy compression with AvroPathPerKeyTarget

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 0.11.0
    • Fix Version/s: None
    • Component/s: IO
    • Labels:
      None
    • Environment:
      Cloudera

      Description

      Compress.snappy(AvroPathPerKeyTarget) doesn't work.

      I needed to create my own class that extends AvroPathPerKeyTarget.

      public class CompressedAvroPathPerKeyTarget extends AvroPathPerKeyTarget {
      
          private Map<String, String> extraConf = Maps.newHashMap();
      
          public CompressedAvroPathPerKeyTarget(Path path) {
              super(path);
          }
      
          @Override
          public Target outputConf(String key, String value) {
              extraConf.put(key, value);
              return this;
          }
      
          @Override public void configureForMapReduce(Job job, PType<?> ptype, Path outputPath, String name) {
              AvroType<?> atype = (AvroType) ((PTableType) ptype).getValueType();
              FormatBundle bundle = FormatBundle.forOutput(AvroPathPerKeyOutputFormat.class);
              String schemaParam;
              if (name == null) {
                  schemaParam = "avro.output.schema";
              } else {
                  schemaParam = "avro.output.schema." + name;
              }
      
              for (Map.Entry<String, String> e : extraConf.entrySet()) {
                  bundle.set(e.getKey(), e.getValue());
              }
      
              bundle.set(schemaParam, atype.getSchema().toString());
              AvroMode.fromType(atype).configure(bundle);
              configureForMapReduce(job, AvroWrapper.class, NullWritable.class, bundle, outputPath, name);
          }
      }
      

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              kazuya Jihed JOOBEUR
            • Votes:
              1 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated: