A few weeks ago I investigated Flex automated testing to determine it's feasibility for our new Flex project. On Wednesday my boss has asked me to follow through with that original investigation and get it up-and-running on our build loop ASAP.

The tools I originally looked at:
  1. HP QuickTest Professional or QTP. This is the big daddy of Flex automation testing, and costs an arm and a leg. The funny thing is I couldn't seem to get this working last time. That and it looks like an old Windows 95 app (superficial I know).
  2. RIATest. In many ways this is the nicest of the tools. Quick, simple and it just works (TM). However it's in very early stages, has its own IDE/language and only works on Windows.
  3. Rational Functional Tester or RFT, a set of Rational plugins for Eclipse. The business preferred option as we use this for other products as well.
  4. Custom Automated Agents, the adobe sample for automation testing. It's worth noting that all the above products are essentially just rebadged versions of this. This has the most power of all these tools, but is very bare bones and quite complicated to understand.
Originally I had decided on RFT a few weeks ago and wrote some basic tests. However, today I spent some more time with it and ran into a few snags.

Firstly our toolbar is a Flex MenuBar, which (for some stupid reason) doesn't handle ItemClick events for root items. Without this you can't perform the simplest of actions like searching. To get this working we would have to use something else. Strike one.

Secondly the Grid data verification points were now returning blanks in every cell. After some snooping it turns out setting a custom itemRenderer for columns (ie combobox or checkbox) causes this to happen. Sadly we need to do this and I couldn't seem to fix the problem. Strike two.

While none of these are show stoppers it made me realise I really didn't like using RFT, it felt flaky, so I thought I'd better re-examine my options.

The other, as of yet unexplored option, was Selenium. Just the other day (literally) the new flex API was released. Behind the scenes this is simply an exposed API via the Flex ExternalInterface (basically a Flex/Javascript bridge). As I was looking at this I realised we could just expose our own custom API to compliment Selenium's.

For some reason this latest option appeals to me as it potentially grants the most power and least amount of headaches. Using the above tools feels like we're getting locked into a specific way of using Flex. When something doesn't work, like it did today, there isn't much we can do except change our application to fit the required mold. Instead it would be nice if the automated tools were as flexible as we need them to be.

What we loose is the recording capability, which I don't think is a huge loss (but sad). Some of our other functional tests at work are also written by hand. It will also require some more initial tweaking while we flesh out our own internal API. My other concern is that it's not exactly the same as users interacting with the app, but then again neither are the above tools.

I'm still not sure at this stage which option I'm going to settle on. Time is of the essence, and I really need to have something to show by next week. I'm hoping with some more testing tomorrow, Selenium will be just what the doctor ordered.