April 14, 2014

Black Ops: Stealth Mode Debrief

This is a brief(ish!) record of some notes that I made prior to the Black Ops Testing webinar, 14th April 2014. The webinar exercise related to the testing of a "Stealth Mode" button which had been added to the Black Ops Testing website, for the purposes of this seminar.

Before I began

I made one big assumption in my test approach: I assumed that the "Stealth Mode" button wasn't a real product feature. After all, if there was a real requirement for a "Stealth Mode" (which obscures the contents of the email address field) then much of the observed behaviour is actually 'Functions As Designed' - although I would still argue that it provides a poor user experience.

With that in mind, I proceeded with the rest of the task as if the "Stealth Mode" button was actually a magic "make the bug happen" button. If this were a real product example, I would be seeking some clarification from the Product Owner to establish the desired behaviour of the "Stealth Mode" button.

My test approach

I adopted an exploratory approach to the task, utilising Firefox (my preferred browser) and Firebug (my preferred browser debug tool) to gain an understanding of what was happening.

Firebug allowed me to quickly work out what was happening, thanks to two features:

Stealth Mode button in Firebug
Firebug made it very easy to identify the button's action, as it highlights modifications.

Once I was certain that I understood the scope of the impact, I moved into thinking of possible workarounds. I used knowledge/experience of similar issues that I've encountered in products I've tested (and products I've used), to establish that there were many workarounds which matched my known patterns for similar issues. I also inverted a couple of my normal web test scenarios (using High Contrast and Screen Reader modes) to show how these could be utilised to bypass the problem.

I spent 15-20 minutes on establishing (in note format) everything that's described below. I then spent another 60 minutes embellishing the report into the format that you see below, primarily for the purpose of illustrating during the webinar.

Isn't this a bit over-the-top?

For a real-world test scenario, yes. In reality I would've taken a much more lightweight approach. I suspect a simple mention-in-passing "Did you know there's a button that makes the email address field disappear?" might've been all that was needed, though if I faced disagreement/resistance/confusion then a lightweight formal bug report could be created.

But, for the purposes of showing my thought patterns and analysis, this seems like approximately the correct amount of writing :)

What is it?

The mc_embed_signup div (which controls the Email Address input field and Subscribe button) becomes renamed to mc_embed_signup_stealthmode, breaking some of the CSS rules and making it harder to enter the email address and Subscribe button.

How would you describe it?

Pressing the "Stealth Mode" button causes the font colour in the Email Address field to change from black to white, effectively rendering it invisible and preventing the user from checking whether they have entered the correct email address. The text on the 'Subscribe' button is also modified to a harder-to-read font.

Users running in Stealth Mode are likely to have a much lower newsletter signup rate, due to the increased difficulty of processing.

Stealth Mode bug
When 'Stealth Mode' is pressed, the Email Address field ceases to be readable.

Secondary bug: Additionally, clicking the Stealth Mode button a second time will generate a script error (which, depending on user's browser configuration, may be surfaced to them via warnings/popups) as it attempts to rename a control which no longer exists.

Script Error
Script Error appears when pressing Stealth Mode a second time.

How many workarounds can you identify?

Here are some workarounds which will allow you to continue with newsletter signup. They're listed in ascending order of difficulty/feasibility.

(For a normal bug report, either of the first two would suffice; however, I took this webinar as a challenge to find as many workarounds as possible!)

Invalid email address
If you can't see your email address, just invalidate the field contents and press Subscribe - you'll get a chance to correct the error on the next page.

Making yourself immune to the bug

During investigation of possible workarounds for the bug, I came across a couple of scenarios which would prevent you from ever encountering the bug in the first place!

Workaround with High Contrast mode
When running with a High Contrast theme, field text color cannot be overridden.

Conclusions / What did I learn?

I found this an enjoyable exercise, which (although seemingly simple on the surface) allowed me to better understand some of the thought patterns that I follow during a typical debug scenario. It also allowed me to better prepare for the Black Ops webinar discussion itself, so that I could better relate to the topics that were being discussed.

One of my most useful discoveries during this exercise was finding out about the animated screen recorder tool LICEcap. I'd taken some video clips during testing using Jing, and was struggling to work out how to embed them within my post; Jing's raw output is in .swf format, so I was searching for some way of converting .swf for upload to YouTube or .gif format, when I discovered that LICEcap would do exactly what I wanted. The animated images above were created using LICEcap, and it's a tool which I can see myself (and my colleagues) harnessing a lot in the future, for creating simple in-line embeds for our bug reports.

  • LinkedIn
  • Tumblr
  • Reddit
  • Google+
  • Pinterest
  • Pocket
Comments powered by Disqus