July 5, 2014 · Weekend Testing

WTA-52 Experience Report

Here's a quick(ish!) summary of my notes from Weekend Testing Americas #52: "Going Deep with 'Deep Testing'", facilitated by Justin Rohrman. (I have a habit of promising quick reports, and then taking two hours to write them up, for which I make no apologies!)

We were introduced to the application-under-test for this session, TitanPad, a real-time collaborative document editor. We saw an introduction to deep testing, and Michael Bolton provided the definition as it is stated in the Rapid Testing Intensive course:

Testing is “deep” to the degree that it reliably and COMPREHENSIVELY fulfills its mission AND to the degree that substantial skill, effort, preparation, time, or tooling is required to do so.

Each participant/group was asked to pick an area of the application on which they would perform deep testing. I decided to focus on the Chat area of the application, as this seemed to be an important part of document collaboration.

Sending and receiving chat messages

With TitanPad seeming to deal solely in plain-text messaging, I looked into how I could vary the content/format of these posts to cause unexpected results. (A few such problems can be found in the Issues section below.)

In order to "get deep", I harnessed some tools which I had at my fingertips, to allow me to easily perform some tasks which might otherwise have some setup overhead:

m     {"type":"COLLABROOM","data":{"type":"CLIENT_MESSAGE","payload":{"type":"chat","userId":"g.sp0d8sm4k0zav46d","lineText":"dummy","senderName":"NeilCHANGED","authId":"g.sp0d8sm4k0zav46d"}}}

Firebug console

Performance and connectivity

Given that the application is focused around multi-user editing, and with the chat panel therefore expected to handle users entering and leaving at regular intervals, I decided that I wanted to perform a stress-test of this functionality. With only a short amount of time for the session, I didn't have time to create a script for this purpose, but I had a different brainwave.

As TitanPad deals with public documents (no login required), I utilised a tool which normally serves a completely different purpose: Browsershots. It's a handy tool for quickly previewing your document's layout/design in a wide range of browsers, firing-up 148 different connections in a short space of time, and then (after automatically capturing a screengrab of the loaded page) disconnecting again. By giving Browsershots the URL to my TitanPad document, I was able to effectively add and remove 150 users to my document with no effort on my part. (Ideally I would've liked for each of those users to submit one or more chat messages; I'd need to invest some time to create a harness for this, but it would seem to be worthwhile.)

Shortly after I started this Browsershots test, I noticed that my Firefox document session had lost its connection. It appeared that other participants were experiencing the same problem...

[17:36:17] amy.kroh: anybody seeing load issues?
[17:36:26] Neil Studd: Yes, me too
[17:36:31] Neil Studd: "Lost connection with the EtherPad synchronization server." in top-right
[17:36:37] Neil Studd: Um... I think I may have caused it ;)
[17:36:39] richardsbradshaw: we saw this too
[17:36:39] amy.kroh: reconnect
[17:36:40] justin_rohrman: I am having that same problem
[17:36:44] Neil Studd: I will be repeating in a minute when it settles down.
[17:36:49] justin_rohrman: reconnect fixed it :)
[17:36:54] jayshreejrathod: yes I am too seeing same connection error

I repeated the same test near the end of the session, and again my fellow participants were reporting some connectivity issues. This TitanPad support topic suggests that this is a known ongoing issue, so even if I was the guilty party, it might be that this is again a known issue.

In fact, at the very end of the session, while investigating something completely different, I stumbled across the TitanPad "Limitations" Help page, and slapped my head when I saw this:

All Pads have a limit of 64 simultaneous users. Please note that usability degrades quite harshly after more than a dozen users since coordination of many people editing the same text isn't an easy task.

Doh!

I wished that I'd looked for this at the start of the task. It's the only Help page on the site, so it wasn't exactly hard to find. It would've recontextualised much of my testing, as I would've shifted focus away from these known performance problems.

Issues

During my one-hour investigation, these are the issues that I discovered related to the Chat functionality:

Here are some other things which I noted in passing, which I didn't investigate further, as I didn't want to distract from my Chat-testing mission:

Wrap-up

We discussed whether up-front planning was necessary to perform deep testing. I think that an element of planning is certainly useful; had I performed the necessary due diligence, I would've reviewed the help page (discovering the documented 64-user limitation) and browsed the support forum (discovering that document deletion issues are important to users, and that connectivity issues have been reported by others) which could have limited my testing in certain areas, freeing up some time for increased depth elsewhere.

I posited that "You can't go deep until you know how deep deep is". While you certainly can't cater for all of this up-front, I think that there is value in some up-front analysis (e.g. discovering the parameters that you will be able to vary during your testing, and lowering the barriers to testability). You will certainly discover more layers (and more opportunities to "go deep") as you proceed further down. Even if you don't describe these as preparatory activities, it's very possible that these will be the first activities that you perform when you begin deep testing.

In conclusion, it was a very interesting session which provoked some different ways of thinking, and some introspection in an area which I often take for granted. I found some potentially valuable issues, made some classic mistakes, and gained an awareness into the activity of deep testing which will stand me in good stead for the future.

Next, my focus shifts to the forthcoming Weekend Testing Europe session #47 which I will be co-facilitating. More about that in the weeks to come...

  • LinkedIn
  • Tumblr
  • Reddit
  • Google+
  • Pinterest
  • Pocket
Comments powered by Disqus