Introduction
Last week, I published a story here highly critical of NFBCS, NFB and Curtis Chong as leaders in technology related to blindness. The piece, “Accessibility And NFBCS” described a number of incredibly important issues in technological accessibility for people with vision impairment in which the largest advocacy organization in the world of blindness remains absent and asks how they can be effective leaders if they ignore the most important events of the day.
The article also discussed the question of whether or not it would have been better if Microsoft had made its own “end to end” screen reader. I believe that, as Apple provides on iOS and Macintosh and Google includes on Android, that all OS should have a fully functional screen reader shipping out-of-the-box. Sighted people don’t need to pay extra for the graphical UI they use, blind people should not need to pay extra for the UI we use either.
In last week’s article, I discuss the NFB role in pressuring Microsoft into not doing its own screen reader, favoring instead the high priced, third party solutions from Freedom Scientific, GW Micro, Dolphin and other companies. Last week’s article was specifically about NFBCS and Curtis Chong’s writings in Braille Monitor. It, therefore, described the NFB role in the third party screen reader story with little context. In the early drafts of that piece, I did include much more historical context but those early drafts of the article contained more than 6000 words and the final version that I actually published still had more than 3600 and was “too long” for some of my readers.
After publishing the story last week, I spent a few hours talking on the phone with NFB insiders who, like me and the other sources I used for that article, were actually present for some of the meetings with Microsoft and were observers to this history. While I feel that the story I told last week about NFBCS and its role is true, I also think it’s important that I tell the rest of the story.
I try to publish here every Tuesday. In some weeks, I have a lot of time to do a lot of research and write fairly formal pieces. Some weeks, like this one, I’ve less time to devote to the blog and, therefore, will be telling this story largely from memory. Unlike some articles here, it will not contain a lot of links to outside sources and such because I just haven’t the time to do so this week.
Ted Henter’s Speech
In the late nineties, Ted Henter, founder of Henter-Joyce and the inventor of JAWS, took the stage at an NFB convention general assembly and made a speech detailing exactly why he felt that screen readers will always be best if developed by small companies dedicated entirely to access technology. One may believe that Ted said these things out of a purely cynical desire to protect the profits of his own company but, while tis may be partially so, having worked for, talked to, hung out with and been friends with him for more than a decade now, I’m confident that Ted was speaking honestly to what he felt was a greater good.
When Ted made that speech, there were no fully functional screen readers built into operating systems. IBM had made two screen readers, Screen Reader/DOS and Screen Reader/2 but neither had ever gathered much popular appeal. Vocal-Eyes from GW Micro had been the most popular DOS screen reader among American users, followed by the DOS version of JAWS. When Windows came along, JAWS for Windows (JFW) and Window-Eyes would together dominate the market. Thus, when Ted made his speech, there were no examples of a fully functional screen reader having been accepted broadly by the user community, thus, no evidence of an OS vendor making a tool that our community would actually enjoy using.
The Others In The Room
Starting around 1995, MS held meetings on its campus up in Redmond to which as many accessibility oriented stakeholders were invited. This, of course, included the screen reader vendors, advocacy organizations including NFB, ACB and AFB and other notables from the world of technology and disability. As I state in the introduction, I am writing this from memory and the two NFB insiders to whom I spoke last week were also telling the story from their memories so, please realize, the following may be a bit foggy as that’s how human memory works.
As far as I can tell, everyone in the room at those meetings, the AT companies trying to protect our profits and the advocacy organizations speaking on behalf of their constituents, agreed that the third party screen reader system would provide the greatest access for the vast majority of users. I will contend that, then and for a number of years into this century, this model was probably the right path to take.
Life Before Accessibility API
If you are one of the many blind people who enjoy using an iOS device from Apple (iPhone, iPad, iPod Touch), you are benefitting from Apple’s very well designed accessibility API and compliance with it in the apps you use. In the late nineties, though, there were no examples of a functional API driven screen reader anywhere.
The Big Hack
In the days before modern accessibility API had been invented, Windows screen readers used a technique called “driver hooking.” In brief, a part of JAWS, HAL, Window-Eyes and the others pretended it was a video card driver and gathered its data in what we called an “off screen model” (OSM). In brief, the OSM was a database of information sorted primarily on where things appear on the screen. Using some other techniques, the screen readers knew which Window, hence, application they were in and, along wit painstakingly developed heuristics for each program they hoped to support, they would then speak and/or braille the information for their users.
A little example of how this worked that I can recall of the top of my head is how JAWS tracked focus in Excel. To “know” which cell a user had in focus, the JAWS Excel support would “see” that the software had issued a series of “LineTo” graphical drawing commands to the video driver. JAWS had no actual idea about what was going on in Excel, so it had to jump through heuristic hoops to do something as simple as tracking focus.
Needless to say, a system built on having human programmers spend so much time figuring out the strange details of how a rectangle is drawn on the screen to track focus had severe limitations. Anytime an application made the slightest change, it would likely break the screen reader.
The OSM and screen scraping techniques also introduced major stability problems as it was such a non-standard way of doing things that neither Windows nor the applications running on it were aware of the screen reader as a device driver and, for their own purposes, also used non-standard techniques to put information on the screen. An application would try to do its own optimizations and would, as a result, cause a screen reader to crash.
MSAA 1.0
Microsoft’s was the first OS vendor to attempt to build an accessibility API. It was called Microsoft Active Accessibility (MSAA) and, to be frank, it was little more than a demo of things to come. MSAA, even in its 2.0 version, did not provide a screen reader with enough information to provide its users with a usable solution. Thus, all screen readers using MSAA back then also had to include non-standard techniques to provide information properly to their users.
This fact alone is a large part of what made it virtually impossible for MS to make its own screen reader, until the advent of UIA (later in this piece), it was impossible to use anything resembling standard techniques to gather information.
The VBA Hack
At Henter-Joyce, Glen Gordon, still the top developer on JAWS, realized that using the automation API designed for VisualBasic programmers to extend Microsoft Office and other applications could also be used to get data into a screen reader. The JAWS team took Glen’s idea and ran with it. This was what the JAWS developers used to do all of the amazing things we did in MS Office back then.
The good thing about using the VBA approach was that we could gather very specific information in the context of the application being employed. We no longer had to follow graphics commands to determine focus, we could get the precise coordinates by simply asking Excel. There were two major downsides to this approach, it required that each application supported in this way have hand coded scripts written for it, hence, it also required that the screen reader had a scripting facility, something that, back then, Window-Eyes didn’t have and proudly boasted that they didn’t even want.
The Window-Eyes Versus JAWS Approach
At this point in our story, I must acknowledge that GW Micro, in what felt to me like business suicide, chose to embrace the MSAA path while we, at Freedom Scientific, continued to reject it. History shows us that GW Micro was on the right side of the technological discussion then but, in my mind, they arrived at that conclusion too early.
While GW Micro would provide no cost consulting advice to billion dollar corporations on how to get MSAA implemented in their software, we, at FS continued down the path of non-standard solutions. In many ways, this is what put JAWS on top of the heap, it could do things no other screen reader could and it provided the best experience for its users as a result.
The Hole In The JAWS Approach
In order to provide the best possible collection of data to its users, JAWS employed v very non-standard techniques mixing its OSM with the VBA support and MSAA where available. The outcomes were terrific for the end uses but, inside FS, continuing to support each application using custom code (scripts and internal) was an expensive proposition. In the world of generic accessibility API, a screen reader like VoiceOver gets all of its information from said interface, hence, when the app changes, no one needs to go in and change the code in the screen reader. This was not true then and to a lesser extent now for JAWS.
As I’ve written here many times before, by 2003, JAWS had reached a monopoly marketshare at around 85% of all new product sales. FS, therefore, had no incentive to continue investing in JAWS as it had won the race. Thus, with JAWS receiving less and less funding annually, these custom solutions started to deteriorate.
The First Real Accessibility API
While MSAA was a pretty poor bit of technology, Microsoft should be recognized for even giving it a try. As there were no accessibility API in existence before MSAA, they had no point of reference and MSAA stood as a solid prototype for things to come.
In my opinion, the first truly usable accessibility API was the one led by Peter Korn at Sun Microsystems. It not only provided specific information about a control, it had the facility to provide its context, hence, enable a screen reader to provide a more complete picture to its users. In effect, the Gnome Accessibility API was the first of its breed.
Apple would follow with its API and Microsoft would craft something called User Interface Automation (UIA) that serves as both an accessibility API and a test framework for the software. Today, with comprehensive accessibility API on all major OS, it’s reasonable to expect a fully featured screen reader also to be included with of them.
IAccessible2
On iOS, OSX and Gnome, there’s one accessibility API on each. On Windows, the OS with the largest number of users, blind or otherwise, there is a second one called iAccessible2 (iA2). Some experts would argue that iA2 is the best and most important of the various accessibility but, in all honesty, I’m not expert enough to describe either its benefits or any pitfalls within it. The Mozilla Foundation chose iA2 over UIA or MSAA in the Windows versions of it’s applications and it’s iA2 that you’re enjoying when you use FireFox with NVDA.
When iA2 was developed, it was intended to be a cross platform accessibility API. The goal was to permit application developers who build software on multiple OS to write their accessibility code once and be able to reuse it on other systems. Sadly, while iA2 is “owned” by the Linux Foundation, it has never been implemented on any system other than Windows, something I think is a shame.
Another huge question is whether or not Microsoft will support iA2 in its “end to end” Narrator. They may elect to only support UIA/MSAA and, therefore, leave FireFox and the other iA2 enabled applications out of the sphere of its support. This could be another reason third party screen readers like NVDA may stick around into the future.
Conclusions
The only conclusion I can draw about the entire third party screen reader debate is that the story is much more foggy than one might think. NFB did insist on promoting the third party, mom and pop company solutions but they didn’t do so alone. HJ/FS, GW Micro, AFB, ACB and everyone else in those meetings except for MS agreed that this was the best path forward. History and Apple have demonstrated that it is, indeed, possible to deliver a high quality, no cost screen reader along with the OS and, holding that high priced, proprietary third party screen readers are a favorable solution in the 21st century is purely anachronistic thinking.
I do not mean to suggest that third party screen readers will or even should disappear entirely, they will likely fulfill an important set of requirements for a lot of people, third party screen readers might even become the luxury products of the Windows world, supporting applications too old to have included MSAA/UIA or by providing a user experience different from and preferred by some to the generic Narrator. Of course, I’ve been predicting the demise of the commercial third party screen reader since I wrote an article slamming JAWS 7 on my old blog so I’m likely not the best prognosticator of things to come.
Afterward
I would like to express my thanks to the loyal NFB members to whom I talked on the phone and exchanged emails with last week. I appreciate the insight you guys gave me, a different perspective on the different meetings up in Redmond and the help you provided in writing this article. I appreciate honest and sincere dialogue and also thank the NFB faithful and everyone else who posted comments on last week’s article. In my opinion, the more discussion we can have about this community within this community the better.
Fortunately for all involved, technology progresses. At the AFB Leadership conference in Phoenix last week, Rob Sinclair announced that Narrator would, in Windows 10, be an “end to end” screen reader. Only the future can tell us how it will work out.
One might even say that it is unrealistic to expect anyone to develop a comprehensive, usable and programatically useful accessibility API unless they are forced to use it themselves to build their own A-T solution: how else are they supposed to come up with a solution that actually works in real life?
A cross-platform accessibility API would be a truly wondrous thing: do you think there is any way we can ever make this happen?
I always enjoy your insights into the history of A-T development, BTW, so keep them coming.
This was a great article. One issue which hasn’t been adequately resolved is Braille support. At least Apple has a half-baked Braille interface which is included in voiceOver. I say half-baked as the Braille drivers lack the full range of commands and functions we have been accustomed to from a notetaker or Windows screenreader. I also don’t like the way apple implemented their own computer Braille code. I have found many examples of translation errors with Grade Two Braille. As far as Windows Narrator is concerned there isn’t any support for Braille displays. I think the whole issue of Braille support could warrant an entire article. Despite what people say about the small braille market, there has been more attention lately with regard to the importance of Braille literacy. In addition, people are claiming that new Braille technology is on the horizon. I will believe this when I see it. If it does happen, this could provide more options for better Braille support in any operating system.
thanks again for raising these important issues.
as far as apple goes half baked is putting it nicely. in my book, it’s absolutely deplorable how we still, in 2015 have to rely on big bad companies like fs and human scare for braille support. I really hope with in my life we see a dramatic reduction in the cost of braille displays, and in all honesty get better braille support in the mainstream such as apple products, because I can say with out a certain doubt, the reason we have such high numbers in kids not being able to read braille is the cost of the braille display for one, vision teachers not being certified but only one time in all the years they teach, and the fact that apple and some others just seem to want to just throw out their own solutions for braille support. and don’t even get me started on UEB braille, this will not help kids and adults learn braille, it’ll be slower, much more costly, and it’s just plain wrong to force that code on people in america like my self who have used grade 2 braille all their lives.
I have not used Apple products with a Braille display, only played with a couple of them briefly using VoiceOver, but I’ve read similar observations to your own as to poor support in relation to computer Braille, translation etc. Couldn’t agree more re UEB. I was against it when it was being proposed over here in Australia and I still don’t use it!
Indeed, I am very much looking forward to this lower cost Braille display project Orbit Research has announced. I seriously hope this comes to pass. If it does, and it survives, then it will, I hope, lead the way for the future minds who develop Braille displays. To my knowledge, JFW is the only screen reader to fully support quality grade 2 Braille input via a Braille display. This to me, is very concerning, as one must have JFW or you just can’t write in Grade 2, no way, no how. I am surprised that Window Eyes didn’t come out with this years ago before they partnered with Microsoft. I seriously hope the Braille displays get cheaper soon, development for Grade 2 input advances and becomes available to more screen readers like NVDA, and we are less chained by the big companies who, in my experience have not adequately kept up with the modern world. Technology is advancing at a rapid pace, and more emphasis needs to be put on Braille development. I don’t want to here the excuse, well I don’t use it so I don’t care about it. Some of us do care about it, some of us use it every single day to get through school and work. Also, when someone says I don’t care about Braille, you are leaving out a very important group of folks, the deaf blind. They have no speech access. Just because you don’t need something, or don’t see a need for it, doesn’t mean it is needed.
that just gets me going, some in this community seem to have that stupid argument, well I don’t need it their for no one else does either. I’m sorry just because someone in this community doesn’t need something like braille doesn’t mean I don’t need it.
Superb and objective article.
I had the exceptional privilege of attending some of the meetings at Microsoft in the 90’s, and I have to say, your memory jives exactly with what I remember as well. All of the advocates were scared to death, believing that if MS did as bad a job with their screen reader as they did with accessibility in general at the time, we would be left with nothing. Competition and innovation are at the heart of capitalism and progress, and we all felt the same way about it.
And if it’s now Microsoft’s turn to shine in the sunlight for what they are doing, bring it on; let’s hope for good surprises in Windows 10.
Thanks Chris for providing your poignant insights and for stimulating all of our thinking on this fascinating and exciting subject.
Don
This article raises a lot of interesting points. Can someone explain to me what an end-to-end screen reader is? I’ve been trying to look up the term with no success. Brandon, I completely understand your frustration at the transition to UEB braille, and I believe that it does present a challenge to those who already know grade 2, although having used UEB on my iPhone for the past six months or so, I don’t see any huge differences except for a few missing contractions. I know that all of the parentheses and various special symbols and math punctuation have completely changed which is annoying. You stated, however, that you think it will make it more difficult for children to learn braille. Could you elaborate on this, because frankly I think it will make it much more simple considering that they won’t need anything more than one braille code. In school, I needed to master Grade II Braille, Nemeth code for math, and computer braille in case my notetaker forced me to use it. To be honest, I think my life would have been simpler being able to learn only the one code. This is just my opinion, but I’m curious to hear your thoughts on the matter.
Before I wrote this comment, I did a bit of poking around and also could not find a definition of an end-to-end screen reader. I will therefore say my piece on the subject based purely on experience and expectations. Linux has a screen reader called Speakup. This screen reader prides itself on being available from start up to shutdown. That means, when I start my computer, I have speech and/or Braille at the log in screen. By extension, I personally contend that this further applies to the parts of the operating system that facilitate recovery options, software updates and the like. Linux has had various ways of handling this. Voice over has come along and set a new gold standard as far as out of the box accessibility. It is for this reason that, the term end-to-end screen reader in my mind brings forth the expectation of having speech from boot up to shutdown, including recovery tools and Windows update. I have always wanted Narrator to take a head first dive into quality screen reading. Apple has proven this is possible, Linux has had it for a while, Google to a lesser extent has it, if you call Talk Back a quality screen reader, it is time that Microsoft come join the party.
As for UEB Braille, I don’t have an issue with a new Braille code being introduced. I however don’t like that it is almost being shoved down my throat. I can see the point that, given you learn nothing else from birth, UEB is probably easier than learning 3 Braille codes just to get through school. For me, I just don’t want to give up what I learned as a 2 and 3 year old kid. If you want to use UEB, I have no problem with that, but don’t tell me to use it just because.
Note that the OSM was originally developed for outSPOKEN for Mac, the first GUI screen reader. Let’s hope both the Windows 10 and post-beta Windows Phone versions of Narrator match or exceed Apple’s model of VoiceOver implementation.
I agree with tim on this subject matter. as far as UEB braille goes, I am absolutely amazed that some have just said well we need to use it because it’s just on our iPhone. sorry, but what if someone like me just doesn’t want to use it at all, am I going to have to be forced in to using it in elevators, on bathroom and classroom doors, and other places braille goes, just because a few simply refuse to learn the US code for braille? I’m sorry but this is the same thing to going to another country and forcing someone to learn english, they wouldn’t take kindly to that, so why should I have to be forced to learn UEB braille by a certain date in the united states of america. let me use grade 2, don’t take contractions away i have used all my life.
This is the first time I’ve heard of the windows 10 narrator. I hope it is a full blown screen reader like Jaws. Yes Apple is great as far as it goes but honestly folks, when was the last time a blind person got any work done with voiceover? If it wasn’t for third party readers, most blind persons wouldn’t be employed. Apple’s gadgets are great for amusement but I’ll be dammed if I can produce a professional document or spread sheet with them.
I’m going to say this and if some get angry at me oh well, but here it is. ever sense tim cook took over apple, it’s gone down hill real fast. voiceover on the mac hasn’t gotten fixes as quickly as it should, iOS has gotten steady worse over the three years sense he’s taken over, and it’s taken way to many updates to fix simple things that should have been fixed sooner. in short, tim cook is a better marketing person then a CEO. yes iOS 8.3 has gotten better, but the point still stands, it has taken three updates, three of them, just to fix 22 voiceover issues that under Steve, would have been fixed in one update.
I hope this pans out and that Ms does produce its own screen reader.
I thoroughly enjoyed this article and its plain-English explanations of the different accessibility APIS from a historical perspective. It is June 2017 now and the Creators update of Windows 10 with narrator working quite nicely has been out for several months. I think your predictions were true. Microsoft has a good start on tutorial web pages for developers to teach them accessibility testing using Narrator and there are many controls in Windows 10 that only Narrator can read. (Though Joseph Lee’s Win 10 app essentials add-on is really catching up.) If narrator continues to improve and documentation for Windows 10 developers increases, so that there is no excuse for an app being inaccessible, we might not need third-party screen reading in the future. But for now, the majority of Windows 10 apps I’ve tried have sadly not been accessible to any screen reader and it has me worrying that perhaps developers will simply ignore Microsoft’s efforts.