An Exploding Myth (Posted August 7, 2005)
Gez Lemon of Juicy Studio has written an article, Testing Invalid Content with Accessibility Validators, about accessibility testing tools and what they can do - well more accurately, what they can't do. Joe Clark characterizes this article as "Exploding the myth of automated accessibility checking" on the Web Accessibility Initiative Interest Group mailing list. The mailing lists I normally watch are buzzing with comments about Joe's assertion and about Gez' article.
Prior to the publication of Constructing Accessible Web Sites, I published an early version of my chapter on testing tools. The first section of that chapter, Testing Tools, outlines what can and what cannot be tested relative to the Section 508 Web Accessibility Standards. Although Gez Lemon doesn't phrase it that way, what he has done is something I wanted to do; he has created a web page that exemplifies violations of the WCAG checkpoints (instead of Section 508) that will probably not be detected as errors by the testing tools. That will be a great resource for training in accessibility. We all know it is possible to fool testing tools, and Gez Lemon's example page provides a wonderful set of examples and a wonderful teaching tool.
Testing tools are important. The simple fact is that there is a high correlation between errors that can be checked by software like missing alt text and form controls without labels and those that can't be checked. Perhaps more important, missing alt text and form controls without labels are very important errors .Testing tools can catch these errors much better than a human reviewer can and these errors are important. The process at IBM of monitoring over 13 million pages with automated accessibility checking over the last several years has proved effective in improving the overall accessibility of the IBM internet and intranet web presence - but the necessity of human review is without question and will be discussed in a forthcoming article in the IBM Systems Journal, Managing usability for people with disabilities in a large Web presence.
My only criticism of the Gez Lemon article is the suggestion that his results represent the testing tools. In fact they represent only the free single page accessibility checkers, which he called the "leading accessibility validators." You do get what you pay for, and the warnings offered by the real products might have pointed to the possibility of errors in cases he presents.
Since posting this, Gez has contacted me to say that he has modified his article to say that "the test page was tested with accessibility validators that are free to use."
The document that breaks the rules is http://juicystudio.com/experiments/invalid.html. It is a great resource for trainers to emphasize the importance of human review. Stay tuned here for a study of what the real accessibility validators do with what I consider to be checkable accessibility violations. Of course I consider that a much more interesting question.
The only myth exploded is Joe Clark.