I had an idea last week about a potential way to give users better error messages in some cases, and I wanted to see if you all thought it sounded feasible. This is inspired by Tim’s trick with
ArrayRef here: https://github.com/hail-is/hail/pull/7792/ , though I don’t want to individually update IR nodes like this, and I don’t want to send the python trace to the JVM.
Create a new IR node, working name is
UserError. This represents an error that happens because a user makes a mistake that won’t be identified until runtime, like an index out of bounds, bad dictionary key or something. It would look like:
UserError(msg: String, id: Int). This is a lot like the current
Die, except that it 1. Has an id number that will correspond to a saved python stack trace, and 2. Won’t throw a
FatalError, but instead some new type of error.
CaseBuilder.or_erroror create some new function that both returns one of these IRs and also saves the current python stack trace using
tracebackto an array or something on
HailContext. In the
ArrayRefcase, we’d now generate an IR in Python that does a bounds check before indexing, throwing a
UserErrorif out of bounds
Add a new type of error on the Java side that is thrown when
UserErrornodes are evaluated. When it’s thrown, it’ll include the
idpresent in the
UserErrorIR. In Python, we’ll receive this error, use the id to look up the corresponding stack trace, and then throw the python stack trace that we saved. That should give an effect similar to
ArrayRef, without bloating the IR.
For the most part, this seems like a reasonable proposal to me. My 2 concerns are
Writing checks in Python instead of Scala. While I like the idea / developer experience of writing checks in Python, I worry there’s the potential to emit slower code this way.
stack_tracesarray that I’m saving on the python
HailContextgoing to grow forever? When can I clear it? I think maybe this can be solved by not saving to the
HailContext, but instead saving it on the
UserErrorIR, and then only when we are about to send to JVM will we do a pass that pulls the stack traces out of the IR, replaces them with ID numbers, and saves them in a
stack_tracesarray which then only lives until the JVM responds with a result.
Even if this proposal doesn’t work out, I think improving the error messages story is something worth discussing.