Cautiously sniffling your UI

Yay, tests!

Tests are awesome, they’re helpful sentinels that keep us out of trouble and away from harm. But we’re not often in a position when, if asked, we’d say that written enough of them, or we’re satisfied with the ones we did write.

This is in part due to the various _(hubris) realities _(capitalism) of the software development industry. But also because many of the tests that we’d like to write are an absolute pain in the neck. This is most evident when it comes to testing user interfaces.

As I quite enjoy testing and thought I could do something to help (something something hubris). I began work on a Haskell Webdriver RFC Protocol bindings package called: Cautious Sniffle. Yes, a new package, because I thought the existing webdriver package needed a rebuild rather than a refactor.

Consequently, I had intended this to be a ‘release’ post for Cautious Sniffle. But I think I succeeded with this package because quite frankly, it isn’t terribly interesting. It’s a bag of types and functions that enable you to speak a recent version of the WebDriver Protocol from Haskell. Yay. So I will point at the things I think might be of interest in cautious-sniffle and then move on with a cool thing you can do with it.

The moderately interesting parts are:

• Servant Client Generics to make GHC write all the borning functions for us.
  • See Protocol.Webdriver.ClientAPI.SessionAPI for more.
• dependent-map for type-safe structures to hold all the wild and wacky capabilities the various webdrivers require.

The really interesting part.

NB : This post assumes familiarity with Property Based Testing, and some knowledge of Property Based State Machine Testing. I’m going to skim over parts that other people have explained in greater detail.

The super interesting part is what happens when you combine cautious-sniffle with a property based testing package, hedgehog in this case.

Specifically when you combine the ability to manipulate and inspect a webdriver capable interface, with the capacity to randomly generate both the inputs AND actions for said interface!!

We leverage the random generators from property based testing to create inputs for test cases, but we also leverage these generators to create our test cases. Thus side-stepping most of the problems associated with static unit tests, both in our inputs, and in the construction of the test cases themselves.

An example

Our example widget with multiple inputs:

All sorts of colourful validation could be buried in such a widget. You might want to ensure that Scales is automatically selected when the base creature is a ‘Goldfish’, but un-selected automatically when another ‘Base Creature’ is chosen. Selecting ‘Hamster’ could trigger a new set options for weaponry. Writing tests for the varying combinations would take an awful long time and be a torturous maintenance burden.

Let’s try to tackle this by empowering our hedgehog with a cautious-sniffle! We’re going to be leveraging the techniques discussed in Jacob Stanley’s post about state machine testing. We will define individual actions, with their own pre/post conditions and input generation. And create a model to track our view of the world, and to help check our assumptions.

First up, we want to describe the different things we want to be able to do. Create a sum type to enumerate the various actions we want to be able to perform. We’ll start with specifying some actions to select a creature and select ‘Horns’ or ‘Scales’.

data Creature
  = Dog
  | Cat
  | Hamster
  | Parrot
  | Spider
  | Goldfish
  deriving Show

data Feature
  = Horns
  | Scales
  | Both

data FormAction
  = SelectCreature -- Select the given creature in the dropdown menu
  | Check -- Check the given checkbox element
  | Uncheck -- Uncheck the given checkbox element

To generate a “test case” for our form, we will ask hedgehog to generate a random list of FormActions.

To track our assumptions about the state of the world, we need that model.

data Model = Model
  { _modelBaseCreature :: Maybe Creature
    -- ^ if we've selected a base creature, and which one.
  , _modelHasHorns :: Bool
    -- ^ Have we checked 'Horns'
  , _modelHasScales :: Bool
    -- ^ Have we checked 'Scales'
  }

My favourite part of writing types like those above is the simple act of writing out such structures will often provide new insights! The mental exercise alone is often worth it.

To interact with the webdriver API, ‘cautious-sniffle’ provides records of functions, we then use those to do things like navigate to pages, find elements, input text, screenshot elements, etc. We will need to access them as we execute our commands, so we can create a separate structure for them:

data Api = Api
  { _apiWDCore :: WDCore IO
  , _apiSession :: SessionAPI (ClientM IO)
  , _apiSelectBaseCreature :: ElementAPI (ClientM IO)
  , _apiHorns :: ElementAPI (ClientM IO)
  , _apiScales :: ElementAPI (ClientM IO)
  }

Now to start to write the functions that will run the different actions:

creatureKey :: Creature -> Text
creatureKey = T.toLower . T.pack . show

cSelectBaseCreature
  :: ( MonadTest m
       -- ^ Each of these functions is a "test" that can fail or succeed
     , MonadIO m
       -- ^ We're going to calling out to webdriver so IO will be needed
     , MonadState Model m
       -- ^ Put the 'Model' in a StateT so we can modify as required
     , MonadwReader Api m
       -- ^ Put the 'Api' in a ReaderT for easy access
     )
  => m () -- We don't need to return anything for our purposes
cSelectBaseCreature = do
  api <- ask

  -- Ask 'cautious-sniffle' for the select element at the given selector
  selElem <- evalIO $ successValue <$> findElement api (ByCSS sel)

  -- Ask 'hedgehog' to randomly select a base creature for us.
  option <- forAll $ Gen.element
    [ Dog
    , Cat
    , Hamster
    , Parrot
    , Spider
    , Goldfish
    ]

  -- We pass in the 'value' or key of our desired choice
  _ <- evalIO $ selectOption
    (_apiWDCore api)
    (_apiSession api)
    selElem
    (creatureKey option)

  -- We've selected a value on the form, now we update our model
  modelBaseCreature ?= option

Now we have an action that will pick a thing at random and update our form, whilst tracking the choice in our model. Yay. We could do something similar for selecting a feature: ‘Horns’, or ‘Scales’.

cCheckFeature ...

This would ask ‘hedgehog’ to choose, ‘Horns’, ‘Scales’, or maybe both! Then based on that choice it would try to activate the checkbox. We can start adding in some pre/post conditions there by not trying to click a checkbox that is already active, or make decisions based on the state of our model.

Earlier we suggested that if you select ‘Goldfish’ then ‘Scales’ would be automatically selected. In the cCheckFeature command you could add a pre-condition that checks if the selected base creature is a ‘Goldfish’. Then instead of clicking the checkbox it checks that ‘Scales’ is already selected.

Depending on how you prefer to structure things this may seem like you’re adding the pre-conditions to the wrong spot. If you have other similar pre-conditions you might end up with a laundry list of checks inside this one function, and we’ve lost any real benefits from using property based testing.

A solution to this is the notion that “actions” or “commands” don’t have to be concrete actions. They can be ‘meta-commands’ that inspect the model, and do sanity checks, ensuring things are as they should be.

For example… We add the command: GoldfishHasScales to our Command type, it’s purpose is to see if the currently selected base creature is ‘Goldfish’, and if so checks that ‘Scales’ is selected. Both techniques have their advantages.

Generating test cases

With our Command type equipped with all sorts of interesting actions and validation checks, ready to unleash havoc on our unsuspecting form. We can now, intead of lining these things up ourselves, ask hedgehog to generate random lists of Commands for us to run against the page.

genCommand :: MonadGen m => m FormAction
genCommand = Gen.choice
  [ pure SelectCreature
  , pure Check
  , pure Uncheck
  , pure GoldfishHasScales
  ]

We can then use this to create a list, sized to our choosing:

genCommandList :: MonadGen m => m [FormAction]
genCommandList = Gen.list (Range.linear 0 100) genCommand

This is hedgehog automatically generating test-cases for us! With tests that generate their own input!! That are testing a web interface!!! Ahhhh!!!

There’s a lot of potential and flexibility with this approach.

• You could use tagsoup after grabbing chunks of HTML to understand what you’re looking at and use that to refine the list of test cases you choose.

• You can build a smarter test case generator that looks at previous steps and selects a more applicable subset of the possible test cases. Hedgehog can do this for you, but the types are a bit more complicated, see the QFPL state machine course for a breakdown of how to utilise that.

I’m certainly excited about what could be achieved with this combination and will be pulling on this thread to see what chaos I can cause. If you try out cautious-sniffle then please let me know how it goes! (I’m sure it’ll be on Hackage at some point.)

Bonus delicious

Hedgehog includes classification and statistical reporting that you can include in your commands, so you can see a distribution of the used commands in the output of your tests. Here is an example from the cautious-sniffle test-suite:

Webdriver Tests
  State Machine
    Enter some text.:                   OK (0.43s)
        ✓ Enter some text. passed 10 tests.
          Send Keys       40% ████████············
          Check sent keys 10% ██··················
          Find element    70% ██████████████······
          Navigate to     80% ████████████████····

If we find that certain actions aren’t being included often enough. We can use hedgehog functions like frequency that let us add weightings to the different possibilities.