Type.scala cleanup

[dking]

@cseed

Every Type corresponds to a Spark type. Question: should the injection into Spark types for ComplexTypes be defined in terms of the corresponding fundamental type?

Spark type is too strong here. We use types in annotation not supported by Spark (e.g. Variant). The answer to this question is currently now (e.g. Variant). I think the fundamental type is a “region value” world thing and the JVM realization of typed values as an “annotation” world thing, with the latter going away.

I think we misunderstand one another. For every type, we have a way to convert annotations of that type to values of a “Spark type” (SparkAnnotationImpex). We currently define this mapping for every type. The question/suggestion I made: the correspondence between values (whether realized as region values or annotations) should only be defined on FundamentalType. A ComplexType can be imported-to/exported-from Spark by using the ComplexType’s realization. I suppose this is irrelevant if we are imminently removing the ability to convert a KeyTable to/from Spark DataFrame.

@cseed

Every Type has an alignment and byteSize. Question: should this move to FundamentalType?

There are many comments about types vs representations in this thread. I have been using the name virtual type to refer to a conceptual datatype with an accessor interface and physical type to its concrete realization (not just layout, but implementation of the accessor interface). I will write this up in another post. byteSize and alignment should both move to physical types.

I agree that this is about physical types. If physical types will be added co-temporally with the above proposed changes, then these operations belong there. If not, I think it simplifies the code and the conception to treat FundamentalTypes as the only types that have alignment and byteSize (i.e. until further notice FundamentalTypes are one-to-one with the physical types).

@tpoterba

There are cases this won’t work, though. Variant, for instance – as soon as the reference stuff is a bit more built up, we can represent variant as a struct where contig is an integer (reference contig) but if we export to spark / json / anywhere else, we want that to be a string.

I envision a Variant’s JSON-representation including the genome reference. A la:

{ gr: "GRCh38"
, contig: 1
, pos: 1
, ref: "A"
, alt: "B"
}

I suppose there are situations wherein we want to define the genome reference once for an array of Variants, but that behavior seems somewhat custom and requiring a fair bit more complicated logic than the current JSONAnnotationImpex contains.

@cseed

We keep using views to pretend one struct is another, even though the fields are mismatched and disordered. What does it mean for a hail expression to have type Variant?

We only do this for genotype. A variant is Variant type in the code and will not match a struct.

Ah, I misunderstood your comment on my PR to rv-ify export_plink. I thought you wanted to created a VariantView so that we could treat anything with the right fields as a Variant.

@cseed

I suppose this behavior mimics most of the expression language because our users never write functions wherein they would be forced to decide what they will accept as input.

Well, that’s just true of Python (duck typing).

Agreed. I’m referring to User-defined functions, wherein the user must explicitly annotate types on functions. It seems reasonable that I might write a function on structs with a GT and and a PL field. The user can rewrite their function to take two arguments, gt and pl. This is fine, but certainly not the nicest possible programming interface. This is not an urgent matter (UDFs don’t currently exist), but it comes to mind when I think about how our users can best interact with Hail.