RVDPartitioner interval in IR

1

I’m encountering a problem while lowering tableIRs that change partitioning – I need partition intervals in the IR, but these aren’t actually real TInterval objects because they contain optionally truncated structs.

I’m proposing two changes:

  1. We back the partition interval ordering out of the type system and make it a standalone ordering used in the interval partitioning. Struct/Tuple ordering in the IR should always assume that every field is included in the comparison.

  2. We use an IR function to build the comparison function to be used in IR.

2

I think this is great! A few concrete questions.

The truncated row will be represented struct { key: T, length: int32 }, where T is a struct type of the key, right? I assume the key fields after the length will be missing?

As you mention, the natural comparison on the truncated row isn’t the appropriate one, so I assume you’ll represent the partition interval explicitly with something like struct { start: TruncatedRow, end: TruncatedRow, includesStart: bool, includesEnd: bool }, where TruncatedRow is the type above? Is that right?

Finally, will the analogues in Scala used in the partitioner use the annotation type corresponding to the above types? That is, Any or Row. I’ve been using in this direction for contexts in the importers and it involves some casting but seems to work well.

3

Everything in your comment is a more precise definition of what I’m proposing, yes!