Introduction
For years, I’ve heard anecdotal reports that JAWS, the world’s most popular screen reader, has more bugs, is less reliable, more unstable and of a generally poorer quality than some of its competitors. In that same period, starting in 1998 and continuing until today, I have never seen a single bit of quantitative evidence demonstrating that this is true. I hear people around the community make these claims based on personal experience, experience that is certainly valid but no one has published a scorecard listing every feature in every application supported by each screen reader, tested each and published the results. I’ve also never seen any detailed reports of reliability, only the same sorts of personal stories.
In this article, I want to explore some of the generally accepted notions about screen reader quality and functionality and ask why, if JAWS is such a bad piece of software, does it maintain a marketshare over 50% and why does it still dominate in most professional settings. Furthermore, I want to explore some of the issues discussed in my article, “Remembering GW Micro” that I published last month.
As a matter of disclosure, I don’t use JAWS. For the most part, my primary system is a Macbook Air running OSX Mavericks with the VoiceOver screen reader. I do use Windows with some frequency but, on that system, I use NVDA because I really like how it works in FireFox. This is a second theme I hope to explore in this piece, does the opportunity provided for career advancement, educational opportunities and other advanced computer usage provided in JAWS more valuable than having fewer bugs if, indeed, JAWS does have more bugs than its competitors.
Ted henter
Before there was a JAWS, Ted henter, its inventor and leader for many years, came to a realization. Specifically, while some talking computer technology had already emerged, none of it was vocationally oriented. In those days, Ted worked for Dean Blazie, a close friend of his until today, where they made the Braille & Speak (BNS), a truly remarkable device in its day. A blind user could do a lot with a BNS but it provided no access to the programs that one might use in a job or university.
To solve this problem, Ted found an investor and started working on the DOS version of a program he called Job Access With Speech. From day one, the defining value behind JAWS was to provide access to professional situations and, to this day, it remains the dominant access technology for blind people in professional settings.
GW Micro Marketing
I joined Henter-Joyce in October of 1998. Among the first things I noticed was that the GW Micro web site claimed that Window-Eyes was “rock solid.” I’ve heard this claim repeated in their marketing materials and in reports from their users. What I’ve never seen is the scorecard I mention in the introduction of this article. I try to base my opinions in evidence, when I did my evaluation of Android, I tested every single feature that came out-of-the-box on my Nexus/7. Before I make a claim of quality or lack thereof, I try to perform as full an evaluation that I can or find a published report that contains such written by a credible source. In the 16 years since I’ve been following screen readers, I’ve never seen a single report card of this sort for Windows screen readers, just lots of personal reports, lots of anecdote without evidence.
Does the lack of quantitative evidence mean that the assertions that JAWS is less stable than its competitors are untrue? Absolutely not, it just means that there is no data that can answer this question so I’ll leave it unanswered. It’s not unreasonable for someone making a purchasing decision to rely on the anecdotal reports written by other users as, in the lack of real data, its all a blind consumer might have.
Regarding Window-Eyes, when Microsoft announced that one could get a copy at no extra cost if they owned Office, I grabbed a copy. I did not perform an extensive evaluation of the product as the reliability problems I found in the first half hour of using the product convinced me that continuing with my evaluation was a waste of time. Specifically, on the Windows login screen, if one mistypes their password, Window-Eyes does not read the error box that comes up saying that something was wrong beyond the “OK” button so a user doesn’t know what he’s saying “OK” to. Then, I discovered that when a user launches Window-Eyes, it may not read applications that were opened before it was started – a problem that does not exist in either JAWS or NVDA. Others whom I trust intimately have reported other major bugs as well. If Window-Eyes is, indeed, “rock solid,” I don’t see it.
Meanwhile, Window-Eyes remains the only screen reader on Windows that still does not support either touch gestures for navigation or Aria on the Internet (yeah, I know, GW Micro says it’s coming but it took them a decade to get Java supported so “is coming” may mean in 2025). Window-Eyes, in my mind, remains highly buggy and as feature poor as anything on the market today.
Let’s Look At Some Numbers
According to the 2014 WebAIM statistics, JAWS holds a marketshare in excess of 50% with NVDA approaching 20% and Window-Eyes falling in with about six points. To make the arithmetic easier, let’s say that JAWS has 8 times the number of users as does Window-Eyes. Hence, it is run on 8 times as wide a variety of hardware, in 8 times as many sets of personal settings, setups, and Windows configurations. Let’s also assume that there are 8 times as many JAWS users discussing their problems online and, therefore, it’s 8 times as likely that a JAWS bug will be seen by the Internet reading public as would a bug in Window-Eyes. Is it possible that JAWS much broader user base and much larger exposure in online media (formal and otherwise) may lead one to believe that it is actually more buggy? In absence of the aforementioned scorecard, we cannot know.
JAWS Broader Feature Set
No one questions that JAWS is more feature rich than any other screen reader. It became so because of Ted’s commitment to providing a tool that blind people could use in professional settings. As far as anyone can tell, JAWS is still dominant in these settings because of its feature set, features which are absolutely necessary for many people to hold a job or further their education.
After I wrote the article describing my memories of GW Micro, a reader posted a comment reasserting, without any evidence, that GW won’t release a feature until “it’s rock solid” parroting Window-Eyes marketing literature. The person who posted the comment continued by stating that GW didn’t add Java support to Window-Eyes until version 8.0 and suggested that the near decade it took them to catch up to JAWS in this area was because of their commitment to quality. This implies that GW Micro had been working on their Java support for all of that time but chose not to release it until it was “rock solid” which, of course, is false. GW Micro didn’t add Java support until they were absolutely forced to do so by market demands.
What if the JAWS team had also decided to wait many years before they added Java support? A year after JAWS first supported the Java Access Bridge, University of Florida (a in the top twenty public engineering colleges in the US) decided to change its computer science and computer engineering curriculum from being based in the Scheme programming language (a Lisp like language developed at MIT in the sixties) to Java. A blind student in that program could have, if he so chose, used Window-Eyes, it was among the approved AT provided by the university, but, if he had made that choice, he would have had to drop out of the program as, using Window-Eyes, he could not possibly have done his class work. I suppose that the person who wrote the post considered this when he posted his statement and I suppose also that he thinks that waiting a decade for your AT to catch up to the reality of the technological world is also a good idea. our hypothetical blind student had no choice, he either chose JAWS or he failed out of college.
Personally, I think that saving that student’s college career is the most important thing a screen reader team can do with its time but, as always, I’d like to hear your comments.
A Data Point I’d Like To See
WebAIM statistics are nice especially because they run year to year and allow us to observe trends. It’s also a self selecting survey which, like all self selecting surveys, is wrought with problems. Is one screen reader under represented in the report while another is over represented? This is data that the WebAIM report cannot answer. It would be impractical to expand the WebAIM survey to include some other more personal information about screen reader users. Unfortunately, there is very little other data published that can tell us much about the make up of the screen reading using public.
The data points I’d like to hear, in a real, well constructed study, would help us learn much more about the efficacy of a particular screen reader. Specifically, I’d like to learn what is the median income of an employed JAWS users versus the median income of users of other screen readers. I’d also like to learn the average level of education accomplished by users of JAWS versus the other screen readers. Based purely in anecdote and in complete absence of real statistical data, I’m willing to bet anyone $100 that JAWS users are A: more likely to be employed, B: make more money and C: more well educated than users of any other screen reader except, perhaps, NVDA. Of course, it would cost much more than a hundred bucks to do the study properly so the bet is probably not worth taking.
As I wrote in “Remembering,” I believe this is why Window-Eyes failed in the market and is why GW Micro is no longer a going concern. JAWS did everything possible to build a base in employment sectors, NVDA came along and grabbed a whole lot of the more technical blinks and SystemAccess grabbed the novice users while Window-Eyes offered nothing special at all.
Fanboyism
Earlier this year, when I published the three Android reviews, I expected and received a spanking from its loyal enthusiasts. Years ago, when I wrote BlindConfidential articles with titles like “Apple Just Sucks,” I got spanked by Apple’s fanboys. When I write critically about Window-Eyes, I hear from its loyal users as well. I understand that people love the things they use, the technology in which they’ve invested a lot of time and energy learning and they respond to criticism of their favorite things. I admit, I cringe when I hear some of my favorite things criticized as well.
What I didn’t expect from the Android series, though, was the celebration tossed by the iOS fans. In my mind, celebrating accessibility failures is never a good idea. I really like my Macbook Air and my iPhone 5S but I want all devices to be equally or more accessible. I take no joy in writing a review of accessibility that, based upon testing I’ve done or published reports from credible sources, is substandard. Because you chose a device that my blog suggests is “better” is a bad reason to celebrate that other devices may not be as good. This isn’t a game, Apple ain’t the Red Sox and Google ain’t the Yankees and there’s no reason to root for one massively profit generating corporation over another.
When I write a critical piece, I do so to inform my readers of results I have learned about some bit of technology. I do not do so to “gloat” that I had made a particular purchasing decision over another. I have no skin in this game, if a new device comes out tomorrow that I think will like, I’ll go get it no matter the vendor. I view technology as tools and nothing more and I don’t root for Craftsman versus Snap-on either.
Conclusions
In general, I think that the access technology business needs much more real data driving the opinion pieces that are so rampant in this community. We all have our favorite things and it’s good that some people write about such, create tutorials and do all of the other things that make using computing devices much simpler for our community but it’s also essential that we try to stick to facts, find the data to support our assertions and view all marketing literature with a very skeptical approach.
While editing this piece, I went through my usual process of adding links to as many of the proper nouns in this article as possible. I usually add a link to the first occurrence of any proper noun I use in an article. I always prefer including a link to a Wikipedia entry instead of a company or personal web site as Wikipedia’s crowdsourced manner of creating content is far more likely to be objective than are web sites written by businesses as marketing tools or by individuals about themselves. In this piece, I found an Wikipedia article I could link to about Ted Henter but not one about Dean Blazie. Some popular screen readers have Wikipedia entries, some do not.
Perhaps it’s a result of poor accessibility in the Wikipedia interface one uses to add or edit an article but, no matter the reason, the history of access technology, the products, the people who created them and the steady improvement of such is hardly reflected on Wikipedia. This is the one forum where we, as consumers, advocates, developers and users can write our own history and it’s something that we should do as soon as possible.
While I agree with at least 95% of your assertions and statistics, I would point out that eight times the marketshare should not be equated to eight times anything else–discussion, bug-reporting, etc.–that you proportionalized in your otherwise excellent article above. In fact, many JAWS users use it only at work, and only because their employer purchased it for them because it’s a standard accessibility tool in many jobs, particularly those within government. I know of many cases personally where people use JAWS at work, but something else, or no computing at all, at home.
Another comment about anecdotal evidence: What was that old expression used in an ad campaign, “two million people can’t all be wrong,” or something similar? Another one that comes to mind is the old saw, “where there’s smoke, …”. IN other words, if so many people report similar situations, plus or minus a minor detail or two, there’s clearly something behind it all, whether it’s been properly researched and documented or not. In other words, because something hasn’t been proven and is not always reproducible doesn’t mean it doesn’t exist. Of course, this could be said of many things, God Himself included. I think the best conclusion to which one should come, not only from reading your article, but from one’s own experience, is that other famous expression: “your mileage may vary.” Some users may never find themselves in the particular circumstance(s) in which JAWS fails to operate accurately, in which case, said user will report the software being the greatest thing since sliced bread; whereas, a user who continually runs up against a circumstance that causes JAWS to not perform accurately and adequately, will say it’s buggier than a Roach Motel.
What I would really like to see happen is for more JAWS users to report specific situations in which they find themselves when problems come up. While this would not be a definitive method of potential bug-tracking, it would tighten up the “loose” talk a great deal, and maybe begin to build a base of experiential data that could lead to some hard research into some of the hearsay problems, and, of course, their solutions. If this sort of problem-reporting and -discussing already exists, I’d surely like to know where that is, as I have been a dedicated JAWS user since version 1.05–a long time now–and have collected several reproducible problem instances over the years, some of which have been fixed, some which haven’t, and would love a to know a place I could go to add these things to my previously described user-provided detailed problem base. Of course, we both know the big problem here, and that is, most people don’t bother enough about problem-reporting, particularly in the details aspect, and the only people I’ve ever met who care enough to test and re-test things that don’t work are, well, testers, which most users decidedly are not. They just want to get their work done, and if something doesn’t work, they find a workaround and go on with their stuff.
There’s two interesting points about the WebAIM as it pertains to NVDA that are worth mentioning. First, the question should be asked if it’s overly representative of English speakers, as a great deal of NVDA’s popularity is with non-English speakers–it’s the only screen reader you can translate to your language of choice, even if that language is something completely unheard of. The other point is that it’s voluntary, and I’ve never personally seen anyone in meatspace pointing users toward it.
As for why Jaws holds such a huge market share, it’s the training problem. I’m not going to debate bug quantity here-the point I want to make stands alone. If you are not computer literate or otherwise capable of self-teaching, you have to have training. Apart from the other problems with this–the tendencies to be route-based instead of skill-based, as it were, and your sighted trainer not knowing the software well (or at all sometimes)–the only option typically given is Jaws. This is why NVDA got the technical users: we are the group who looks at learning a new screen reader, shrugs, and goes “give me 2 days”. I have personally introduced at least two people, one of them my DBS counselor, to NVDA–the people in the organizations who are supposed to “help” us (by which, unfortunately, they normally mean force you into a job as fast as possible) seem, at least in my experience, to only know and care about Freedom Scientific. Couple this with the fact that most blind people have trouble learning one screen reader, and bugginess doesn’t matter anymore because you have to go where the training is.
Yes, more data! It is clear that it is very much needed. One especially crucial number not mentioned in your article is the number of persons who rely on screen readers who could not get a specific job because no screen reading package could be adequately scripted to handle the software requirements of that job. All I have is anecdotal evidence, but I’ve now heard far too many stories of blind individuals who had been offered a position, but who encountered some accessibility snag that could not be overcome. Sometimes this is due to programs which are built in-house with no thought to accessibility; sometimes its a commercially available package that isn’t accessible to screen readers. We need the data on the percentage of jobs out there that persons who rely on screen reading technology could not hold because the applications used in those positions are not accessible. With or without this data, to my mind it is clear that 1. there needs to be real incentives to build in accessibility from the ground up; and 2. screen reading technology needs a major overhaul in order to be far more flexible than it currently is.
And, while I enjoyed reading this piece, it seems to me that the main point here is something like “JAWS may be sucky, but WindowEyes is probably suckier.” Maybe we need a sucky index.
I agree that Real data about screen reader quality, usage etc. is a good idea, and well worth investing in. I would say though that there are good and bad points with any screen reader, and many people use more than one on a daily bases in order to be as productive as possible. I do not think that using JAWS really has an impact on either level of education or how much one earns. Those are based on ability and mbition.
quote: “Specifically, I’d like to learn what is the median income of an employed JAWS users versus the median income of users of other screen readers. I’d also like to learn the average level of education accomplished by users of JAWS versus the other screen readers.”
With this data would need to be the type or area of work: a software worker is completely different from a real estate agent, a bookkeeper, a librarian or a school administrator.
Which touches on the excellent point Lisa made: while a general screen reader or other general AT can get a user able to use a computer (in the general sense), it does not mean that user can interact with specialised software in some area (finance, programming, records… or Google products). That Austin has to worry about being able to use the IDE of choice at any paricular employer is a major issue, and as with many “Orca bugs”, it’s also very much the accessibility of the software that interacts with the screen reader.
At my work, we have a back-end client (written by others in Gtk) and a web shop (with some adminny stuff in it). While I did and still do whatever I can to make the webshop work for as many users as possible, I can’t fix the fact that one of our clients (our clients are businesses) has a blind employee using System Access on Windows who cannot use the client (because Gtk doesn’t talk to Windows MSAA). Currently this guy’s option is to switch OSes entirely (Linux) so he can learn a whole other screen reader (Orca) just so he can do his job. Or not, if that client of ours decides they can’t employ someone to do a job that they are currently simply incapable of doing, which neither they, not we, can fix. Basically, the options for our end are:
– get the writers of the client interface to rewrite it in something else, like Wx or Qt
– someone or some group writes a bridge between Gtk and MSAA (this would benefit many many others, incidentally… Dragon users for example?)
Neither seem likely anytime soon. And this sort of thing simply transcends individual screen readers, or other individual AT.
I cannot comment on JAWS versus window-eyes, as I have never been a power user of the GW Micro product. That having been said, I have had several recent experiences with JAWS that lead me to believe many bug reports by other users are far more than anecdotal. I can say that on my work machine JAWS has some serious glitches supporting Microsoft Office 2010 never mind any custom software. Rather than failing to provide evidence for my assertions, I’ll now tell you how to reproduce at least three bugs that continuously hamper my productivity.
1. Using JAWS 15 on windows 7 64 bit and Outlook 2010, try replying to a message with contents in the subject line and a blank body. JAWS causes outlook to crash. Speech also goes silent until a sighted person can close the program and relaunch it.
2. While using JAWS 14 and JAWS 15 on windows 7 64 bit, receive a meeting invite while composing an email in Outlook 2010. JAWS 15.0 will read a false calendar appointment that is one minute long i.e. from 08-58 AM to 08-59 AM. JAWS will keep repeating this as you try to review your message. The only way to fix it is to alt tab to another program and then bring focus back to outlook 2010.
3. With the same operating system and software versions mentioned above, JAWS seems to be unable to detect automatic formatting changes that are made to your documents in Microsoft Word. To reproduce this, open a blank document in Word. Indent the first paragraph by hitting the tab key. If you read the text attributes JAWS says the paragraph is formatted with an indent.
Next, hit return and start typing a new paragraph. After you have written something read the text attributes. JAWS will tell you that no indent exists even though MS Word automatically inserted one. The problem extends to all attributes such as bold, italics, etc. If you make a certain chunk of text bold using control B, JAWS accurately reports the presence of the attribute. However, if you insert some characters in the bold text at a later time or create a paragraph in between the bold and regular text, JAWS will not tell you that Word has automatically made your new paragraph bold. I know automatic formatting features can be disabled, but isn’t document formatting a basic feature for a screen reader to handle? To be fair NVDA has problems with this too.
These are just some examples. If you want I can provide you with more glaringly obvious glitches such as JAWS claiming to support touch on windows 8.1 tablets but not allowing a blind user to invoke the touch type keyboard in Metro style apps. I know products cannot be perfect, but bugs of this magnitude should not exist in a product that calls itself “Job access”.
This is just my $0.02. Thanks for reading.
John
I easily duplicated these three bugs, using Office 2010, Windows 7 64-bit and of course JFW 14 and 15.
NVDA has bugs too, and as a user, I know it’s part of my responsibility in exchange for getting it for free, to report bugs.
But FS has no bug reporting facility. I propose they create a detailed web form that a potential tester can fully fill, so at least they’ll have the feedback. Then, for the bugs they fix, they should reward the testers who reported it with thoroughness by giving them a free year or two on their SMA. JAWS would become more stable; unemployed smart people with more time than money would benefit most, and those of us with good jobs and little time to test could continue to pay for the SMA as we always have.
It would be good to have some objective data here. I think JAWS may have a larger market share than NVDA in part because Voc Rehab counselors, trainers, and education specialists know about JAWS and provide it for their clients and students. FS is much better at marketing its products, especially in the States. FS is also a known vendor for our government agencies while NVDA generally is not.
I think one area where you can find some objective data showing a reduction in the quality of JAWS is by looking at the number and variety of applications for which there are scripts. For example, JAWS used to support MS Project 2000. However, those of us who need access to Project 2010 or 2013 for work are left with the need to hire someone to come in and do some custom scripting, scripting that will only belong to the company I work for and not benefit the JAWS user base as a whole. If you compare a list of applications JAWS supported in 2001 to what they actively support now, you will see that they’ve dropped support from many applications while focusing on most of Freedom’s efforts are in supporting popular Microsoft applications, especially Word, Outlook, Excel, and Powerpoint. Yet OneNote, Publisher, Access, and Project have very little support and don’t seem to be a priority for FS. I’m currently shoehorning Outlook to do some project management tasks for me since Project doesn’t work very well with JAWS.
I do use JAWS when I use Windows because until recently, it provided the best support for Office applications. It still does with Outlook. NVDA is steadily catching up though, and that could change the complexion of the JAWS user base in the future. For now, JAWS still has a definite edge for professionals. If NVDA ever gets good access to Citrix, JAWS is going to have a fight for survival on its hands.
Honestly I only celebrate a product’s success at making things more accessible. I switched to Android for about 6 months, and in the beginning I loved it because I felt like a rebel, and was able to prove that Android could work for a blind person if they cared to make some sacrifices to use it. My main beefs with Android accessibility are its web-browsing support, along with the wide variation between device manufacturers and even within devices made by the same company. I believe that if a device improves accessibility it should be celebrated. If you don’t like it just because it is not made by the company you have been using and have gotten into bed with, you shouldn’t bad-mouth it. I have never quite gotten the point of bashing a device that is not made by your favourite company, and will probably never be able to grasp that idea.
test numbers
1. First things first
2. And then another thing
3. Third thing
4. Fourth thing
end test