Test organization

Good discussion on testing this morning. I have a concrete example I’d like to workshop. Here are some tests I wrote yesterday for set functions:

  @Test
  def testSetFunctions() {
    val a = MakeArray(Seq(I32(1), I32(3), I32(7)), TArray(TInt32()))
    val s = ToSet(a)
    val t = ToSet(MakeArray(Seq(I32(3), I32(8)), TArray(TInt32())))

    assertEvalsTo(s, Set(1, 3, 7))

    assertEvalsTo(invoke("toSet", a), Set(1, 3, 7))

    assertEvalsTo(invoke("contains", s, I32(3)), true)
    assertEvalsTo(invoke("contains", s, I32(4)), false)

    ...

    assertEvalsTo(invoke("union", s, t), Set(1, 3, 7, 8))
    assertEvalsTo(invoke("intersection", s, t), Set(3))
  }

Here is what it looks like with “one condition per test”. To share the common setup, I broke out the test into its own test class.

class TestSetFunctions {
  val a = MakeArray(Seq(I32(1), I32(3), I32(7)), TArray(TInt32()))
  val s = ToSet(a)
  val t = ToSet(MakeArray(Seq(I32(3), I32(8)), TArray(TInt32())))

  @Test def testToSet() {
    assertEvalsTo(s, Set(1, 3, 7))
    assertEvalsTo(invoke("toSet", a), Set(1, 3, 7))
  }

  @Test def testContains() {
    assertEvalsTo(invoke("contains", s, I32(3)), true)
    assertEvalsTo(invoke("contains", s, I32(4)), false)
  }

  ...

  @Test def testUnion() {
    assertEvalsTo(invoke("union", s, t), Set(1, 3, 7, 8))
  }

  @Test def testIntersection() {
    assertEvalsTo(invoke("intersection", s, t), Set(3))
  }
}

Which do we prefer? This still feels like a lot of boilerplate to me.

The latter gives me more information about what exactly succeeded and failed. I prefer it and use it in my personal projects.

I agree there’s more typing involved, but, personally, it doesn’t make it harder for me to read or for me to modify.

Here are the tradeoffs I see:

  • The second is definitely noisier, but it’s a level of noise I could live with. I think the second could be slightly denoised with simpler test method names, e.g. TestSetFunctions.toSet, TestSetFunctions.contains.
  • I think the main advantage of the second is in failure reporting. Suppose you make a change which breaks exactly toSet and intersection (for some reason). The first will only show that the toSet assertion failed, while the second tells you exactly which passed and which failed. That extra information might help you see what you did wrong faster.
  • In the second, the objects a, s, and t are created per test, while in the first they are created only once. Though since these are immutable, they could be factored out into a persistent fixture, such as fields on a singleton object. (In this case that’s probably not necessary, I’m speaking more about this pattern where the fixture may be more expensive to construct.)

I’m not sure which I prefer in this case. As the number of set functions being tested gets larger, I think the scales tip more towards the second.

There are probably features of TestNG and/or ScalaTest we could take advantage of to reduce the boilerplate while retaining the test granularity. After a quick look at TestNG, I think this would work (see http://testng.org/doc/documentation-main.html#parameters-dataproviders):

class TestSetFunctions {
  val a = MakeArray(Seq(I32(1), I32(3), I32(7)), TArray(TInt32()))
  val s = ToSet(a)
  val t = ToSet(MakeArray(Seq(I32(3), I32(8)), TArray(TInt32())))

  @DataProvider(name = "set data")
  def createData(): Array(Array(Any)) = Array(
    Array(s, Set(1, 3, 7)),
    Array(invoke("toSet", a), Set(1, 3, 7)),
    Array(invoke("contains", s, I32(3)), true),
    Array(invoke("contains", s, I32(4)), false),
    ...
    Array(invoke("union", s, t), Set(1, 3, 7, 8)),
    Array(invoke("intersection", s, t), Set(3)))

  @Test(dataProvider = "set data")
  def test(ir: IR, shouldBe: Any) {
    assertEvalsTo(ir, shouldBe)
  }
}