LowerTableIR and nested CDAs

[tpoterba]

@cseed

There user should explicitly use rbind to write a subquery free pipeline:

This doesn’t work right now – How do I embed this in a TableFilter, for instance, without writing the entire table query in a single Python expression? Python would also need to have different binding/scope rules.

@cseed

I’m confused why these will be relational lets

We’ll have patterns like:

RelationalLet 
  foo
  TableCollect ...
  TableAggregate
    TableFilter
       some table
       some expression of RelationalRef("foo")
    some query

A normal Let won’t put “foo” into the environment where it’s needed, which is inside a relational IR within the body.

@cseed

I’m also confused by this. If you have a relational node in a loop, you need to run it on each iteration. Or do you mean in a loop inside e.g. TableFilter? That has to get pulled out.

I’m referring to IR that might be generated by a user like:

StreamMap
  Stream Range 0 N 1
  idx
  ArrayRef
    TableCollect ...
     idx

We obviously want to run the TableCollect only once. Proposal #1 has the nice property that this will be lifted out. I think it’s hard to use the _localize=False stuff to do horrible things in this model, which I like.