Browser based testing - a QA perspective

This is the first in what will be a series of posts on my experiences of testing a major Website (http://www.betfair.com/) .

I am definitely an automation QA guy rather than a developer who writes tests, so don't expect any ground breaking code. Having said that we're well on the way to developing a very decent testing toolset that covers everything from DB messaging to Web services to browser testing, but more about that later.

The Problem - How to test a Website that is constantly changing

I'm not especially interested in testing Websites per-se as that question is an old one. In our company it was answered around 3 years ago when a team of QTP guys developed several hundred tests covering all major functions of the site. The problem is that only about 5% of these tests still work despite continual (if sporadic) maintenance on an ever more, er, "challenging" codebase.

So, the problem as I see it is essentially one of maintainability. Performance and cross-browser capabilities were, for me, of somewhat secondary importance. With this in mind I set about scouring the net for a way to write maintainable Web tests, with all options open. Firstly I looked at the major vendors, then the minor vendors, then open source tools. I settled on Selenium in the end for the following reasons:

1. Another team within the company was in the process of developing a Java based test tool. This would enable me to "leverage" their hard work and their test utilities.
2. It's open source.
3. I liked the generic nature of the API.
4. I figured I could write a re-usable framework with time and some input from a developer or two.

All well and good, except that as usual real life got in the way and the new project I was working on would need some moderately complex Web tests within the next 2 months, so point 4 suddenly became all important. Going back over my old research, I revisited something called AOST, which by then had been renamed Tellurium (http://code.google.com/p/aost/).

The Answer - The Tellurium Test Framework

Well, maybe it's not the only answer but it's certainly a very good one! Tellurium is essentially a framework that uses Selenium to drive the browser, but does a whole lot of very clever stuff in addition.

So the tests I needed to write were dealing with a farily complex and extremely important part of our site, namely what we call the Market View. This is basically a dynamic table, which can have anything between 2 and 9 columns and up to 100 or so rows. A little more trivial than the usual "Google Search" example that everyone seems to use, yet within a few days I was successfully reading dynamic data and writing tests. Considering that at that stage I had all of 3 weeks experience of automating Web browser tests (my background is Web service testing), I was suitably impressed. Tellurium is especially powerful and easy to use when dealing with Tables and Lists.

I had written a similar tests using just Selenium, but I considered this code to be brittle in the extreme.

The Selenium code:

selenium.click("//td[@id='EBP_0_9191692']/div");

selenium.type("pv1_EBP_9191692_price", "2");

selenium.type("pv1_EBP_9191692_size", "10");

selenium.click("//td[@id='ELP_0_9191694']/div");

selenium.type("pv1_ELP_9191694_price", "5");

selenium.type("pv1_ELP_9191694_size", "8");

The equivalent Tellurium code:

marketView.click("market.prices[1].back1");

betManager.type("betManager.placeBets.backBetTable[1].price", 2");

betManager.type("betManager.placeBets.backBetTable[1].size", "10");

marketView.click("market.prices[2].lay1");

betManager.type("betManager.placeBets.layBetTable[1].price", "2");

betManager.type("betManager.placeBets.layBetTable[1].size", "10");

For maintainability, the advantages for me are:

1. The element locators are kept separate from the test code. This is a key Tellurium feature. Selenium has it's ui Map, but for me this very much inferior to the Tellurium ui Module (more on this later).
2. These ui Modules are very easy to define.
3. The tests are readable, which should not be underestimated as a maintenance aid.

During this project I found a few problems with Tellurium, all of which were fixed within days (often within the day), my questions were answered promptly and the documentation is generally very good. A slightly unexpected but very welcome bonus was that our manual QA people could understand what the tests were doing and could therefore start to write tests themselves. Our test tool uses Excel spreadsheets for input, which no doubt aided this process.

Currently we are about to re-write the whole of our user interface, and will in all likeliness use Tellurium during this process. That is unless anyone can find anything else that can do a better job. Personally I doubt it.

In later posts I hope to be exploring such subjects as CI for Web tests, mocking of Ajax requests, more on Tellurium, and my experiences of training manual QA in automation techniques.