Command line tool

We’re going to fold cloudtools into the hail repo and into the hail PyPI package. At the same time, with the rise of services, I thought it might make sense to redesign the Hail command line in the style of gcloud or kubectl. Let’s call it hailctl (or just hail?) List of things we want to support now:

Existing cloudtools functions on a dataproc subcommand:

hailctl dataproc start ...

We’ll want to be able to perform an authentication flow to obtain a ~/.hail/token to use with the services:

hailctl auth login


hailctl auth show

which shows the user bucket and other account details.

We’ll also want to be able to list batch jobs:

hailctl batch list

and so on: list the jobs of a batch, get the logs for a job, submit a single job batch with an image and command line arguments, and submit a Python script to run in an image with Hail client libraries installed which can submit jobs to batch and/or apiserver.

Finally, we’ll probably want hailctl dev ... for interacting with the cluster and ci2, which has commands for retesting PRs, asking for specific SHAs to be tested and/or deployed, and tearing down deployed versions.


Implementing this on top of a Python API that people can use as well would be very compelling

This all sounds great!

Tim, I agree! For the service stuff, hail/pipeline/batch is already that API. Really, we just need to reorganize cloudtools into a Python library.