Automating Screen Reader Testing On macOS Using Auto VO
About The Author
Cameron Cundiff is a Software Engineer and leader in Accessibility and Inclusive Design. He builds AccessLint and is the co-founder and organizer of the New …
Automated testing is an important part of any software project, including testing for accessibility. There are already tools for linting and integration testing accessibility, but what about end-to-end testing with real assistive technology? Since I hadn’t seen this before, I set out to build Auto VO, a driver for the VoiceOver screen reader.
If you’re an accessibility nerd like me, or just curious about assistive technology, you’ll dig Auto-VO. Auto-VO is a node module and CLI for automated testing of web content with the VoiceOver screen reader on macOS.
I created Auto VO to make it easier for developers, PMs, and QA to more quickly perform repeatable, automated tests with real assistive technology, without the intimidation factor of learning how to use a screen reader.
First, let’s see it in action, and then I’ll go into more detail about how it works. Here’s running
auto-vo CLI on smashingmagazine.com to get all the VoiceOver output as text.
Seems like a reasonable page structure: we’ve got skip navigation links, well-structured lists, and semantic navigation. Great work! Let’s dig a little deeper though. How’s the heading structure?
Hmm! Something’s a little funky with our heading hierarchy. We ought to see an outline, with one heading level one and an ordered hierarchy after that. Instead, we see a bit of a mishmash of level 1, level 2, and an errant level 4. This needs attention since it impacts screen reader users’ experience navigating the page.
Having the screen reader output as text is great because this sort of analysis becomes much easier.
VoiceOver is the screen reader on macOS. Screen readers let people read application content aloud, and also interact with content. That means that people with low vision or who are blind can in theory access content, including web content. In practice though, 98% of landing pages on the web have obvious errors that can be captured with automated testing and review.
There are many automated testing and review tools out there, including AccessLint.com for automated code review (disclosure: I built AccessLint), and Axe, a common go-to for automation. These tools are important and useful, but they are only part of the picture. Manual testing is equally or perhaps more important, but it’s also more time-consuming and can be intimidating.
You may have heard guidance to “just turn on your screen reader and listen” to give you a sense of the blind experience. I think this is misguided. Screen readers are sophisticated applications that can take months or years to master, and are overwhelming at first. Using it haphazardly to simulate the blind experience might lead you to feel sorry for blind people, or worse, try to “fix” the experience the wrong ways.
I’ve seen people panic when they enable VoiceOver, not knowing how to turn it off. Auto-VO manages the lifecycle of the screen reader for you. It automates the launch, control, and closing of VoiceOver, so you don’t have to. Instead of trying to listen and keep up, the output is returned as text, which you can then read, evaluate, and capture for later as a reference in a bug or for automated snapshotting.
Right now, because of the requirement to enable AppleScript for VoiceOver, this may require custom configuration of CI build machines to run.
Scenario 1: QA & Acceptance
Let’s say I (the developer) have a design with blueline annotations – where the designer has added descriptions of things like accessible name and role. Once I’ve built the feature and reviewed the markup in Chrome or Firefox dev tools, I want to output the results to a text file so that when I mark the feature as complete, my PM can compare the screen reader output with the design specs. I can do that using the auto-vo CLI and outputting the results to a file or the terminal. We saw an example of this earlier in the article:
Scenario 2: Test Driven Development
Here I am again as the developer, building out my feature with a blueline annotated design. I want to test drive the content so that I don’t have to refactor the markup afterward to match the design. I can do that using the auto-vo node module imported into my preferred test runner, e.g. Mocha.
Under the Hood
Auto-VO uses a combination of shell scripting and AppleScript to drive VoiceOver. While digging into the VoiceOver application, I came across a CLI that supports starting VoiceOver from the command line.
I’d love to hear your experience with
auto-vo, and welcome contributions on GitHub. Try it out and let me know how it goes!
How do we make design systems work? In his upcoming Smashing workshop on Successful Design Systems, Brad Frost explains how to plan, execute and maintain a successful design system at your organization. Online, and live. June 29 – July 13, 2021.
This content was originally published here.