Currently, partition filter push down is quite costly. For example, if you have many nested or/and expressions, Pig hangs:
Note that IN operator is converted to nested OR's by Pig parser.
Looking at the thread dump, I found it creates almost 60 stack frames and makes JVM suffer. (I will attach full stack trace.)
Although the filter expression can be simplified, it seems possible to make PColFilterExtractor more efficient.