Thursday, December 8, 2011

Preliminary Findings From a Study to Evaluate Voice Note-taking: Using Dragon NaturallySpeaking in the Classroom


This blog post presents preliminary findings from a study that I am conducting into voice note-taking for the University of Calgary's Disability Resource Centre. It states the purpose of the study, describes the technologies being evaluated, and explains the methodology being followed including the evaluation criteria.
The study began in November 2009 but is still on-going. Thus, I will give an update on its current status and the preliminary findings in the form of criteria for success with this note-taking approach.


The purpose of the study is to evaluate technologies to enable voice recognition software to be used in the classroom for note-taking. This is a need for students who have a learning disability or motor impairment that may prohibit them from using traditional note-taking modalities e.g. handwriting, typing into a laptop. The study also aims to identify criteria necessary for success in the use of voice note-taking.


The three main technologies being evaluated are:
  1. Dragon NaturallySpeaking (DNS) voice recognition software: Converts voice into text
  2. Sylencer handheld microphone: Muffles speech, minimizes background noise
  3. Digital voice recorder (DVR) for later transcription with DNS.
DNS, when used in its conventional manner, is not suitable for taking notes in a classroom because:
  • it is speaker-dependent i.e. trained to recognize a single voice
  • a particular style of dictation must be followed
  • no background noise must be present
  • dictating while the lecturer or other students are speaking is disruptive.
The Sylencer, also known as a Stenomask, is not a new technology. It is widely used in court reporting for "voice writing.” It was invented by Horace Webb as a method for bypassing shorthand to record court proceedings; his son owns Talk Technologies, the developer of the Sylencer.
The approach that Nuance, the company behind DNS, recommends for students who want to use the software in the classroom is to record the lecture and using “echo-dictating” to convert it into text after the fact. But this is time consuming - the most proficient echo-dictator will need to listen to the entire lecture at least once to capture it all.

Technology Combinations

Students are offered two approaches for note-taking with these technologies:
  1. Dictating with DNS onto a laptop using the Sylencer
  2. Dictating with DNS onto a DVR using the Sylencer and later transcribing the dictation with DNS on the laptop.
Each of these approaches offers advantages over the other. The DVR is easier to carry and presents fewer distractions while taking notes i.e. with the laptop, one can view the output from DNS and misrecognitions may be distracting especially if there isn’t time to correct. But voice note-taking directly on the laptop saves time: the student does not have to perform a transcription after class nor manage another device (the DVR). Also, the student may have time to edit or format notes during the lecture.
Note: Those students that are unable to hold the Sylencer are offered a third option: using a DVR to record the lecture and then echo-dictating it afterwards to convert it into text on their computer.


The methodology includes the following phases:
  1. Initial Technology Evaluation: I tested the three technology combinations during two university lectures.
    • I was particularly interested in whether the Sylencer muffled my speech sufficiently and most students seated near to me reported that they did not hear me speaking. Some may have noticed my speech but could not understand it and were not distracted by it.
    • I learned that it is very important to be proficient in using DNS and the Sylencer, in order to keep up with the lecturer and maximize DNS’s recognition accuracy.
  2. Recruitment Session: When a student expresses interest in participating, we meet for a two hour session during which I:
    • demonstrate the technologies
    • explain the study purpose, procedure and participant responsibilities
    • if the student is interested in participating, complete a profile/pre-screening questionnaire to gather background information, assess their suitability as an evaluator, and assign them to one/more technology options to evaluate.
  3. One-on-one Training: to teach evaluator any technologies that are new to them
  4. Practice: administer a practice session to prepare the evaluator for the classroom
  5. Evaluation: evaluator takes voice notes in at least two lectures depending on the number of technology options they are evaluating and the time they require to provide an objective assessment; I attend the first session to observe technology use and assist with setup
  6. Notes Analysis: count number of corrections made to the transcription.
  7. Post-evaluation Interview: gather evaluator’s feedback about technology use and voice note-taking process; rate evaluation criteria.

Evaluation Criteria

The voice note-taking technologies are being evaluated for:
  • Training time
  • Ability to acquire the technology
  • Ease of use of the technology, especially in the classroom
  • Total time to produce text version of notes
  • Quality of notes
  • Effectiveness of entire note-taking process in achieving learning and retention of lecture material
  • Ability to use the technology independently i.e. motor skills
  • Ability to take notes independently i.e. dictation proficiency
  • Acceptance of technology by lecturer and other students

Current Status

Since the inception of the study in November 2009, eleven students have attended a recruitment session. Of these, five students agreed to participate and attended one/more technology training sessions for DNS, the Sylencer and/or the DVR. Two students took the Sylencer into the classroom: one recorded voice notes on a DVR while the other dictated onto both their laptop and a DVR. The latter student is continuing to evaluate voice note-taking. The reason for the drop in numbers between these three stages will be explained in the final section, Discussion.

Preliminary Findings

The preliminary findings can be grouped into 6 categories: personal characteristics, course selection, technology selection, training (both in technology use and for DNS) and DNS vocabulary customization. These findings are presented as criteria for success in the use of voice note-taking in the classroom.
Personal Characteristics
Personal characteristics covers a wide variety of factors that can determine whether the student is even a suitable candidate for voice note-taking, and can influence their success with it as well.
  • Comfortable using Sylencer in public (highlights disability)
  • Hold Sylencer
  • Operate the laptop or DVR
  • Suitable voice
  • Willing to learn the new technologies
  • Dictate well i.e. enunciate carefully, formulate phrase before speaking it
  • Adjust speaking style for Sylencer: Dictating using the Sylencer requires training and practice to master for two reasons.
    1. Its microphone is very sensitive so one must speak at a lower volume, in essentially an undertone as if one were speaking in a room where a small child was sleeping. A benefit of this quiet speech is that it makes the speaker less audible (and thus disruptive) to those nearby.
    2. Another design feature of the Sylencer that aids in muffling speech and minimizing background noise is the seal that its rubber mask makes when held to the face. But the mask also limits the movement of the facial muscles, making articulate speech more challenging and thus possibly compromising speech recognition as well.
  • Adjust note-taking style for DNS i.e. dictate in grammatically correct phrases versus the abbreviated note-taking style that suits physical task of writing at a quick pace
  • Multi-task i.e. listening and dictating compete for attention (versus listening and writing)
  • Manage switch between voice note-taking and class interaction e.g. removing Sylencer to answer a question, engage in laughter, etc.
Select Appropriate Course
Unfortunately, voice note-taking isn't suitable for all types of courses. DNS performs better under the following conditions:
  • Best initial DNS "out-of-the-box" results with a general vocabulary
  • Infrequent need to record special symbols, charts/graphs or diagrams that are best sketched by hand
  • Special versions of DNS are available for medical, legal, other languages (e.g. French, Spanish)
Select Appropriate Technologies
Voice recognition software is probably the most demanding application most users will ever run on their computer. This is why it is important to heed the system requirements. Nuance also rates devices used in conjunction with DNS such as DVRs and microphones. You'll achieve the best results if you observe the following:
  • Powerful computer: fast processor, lots of memory (try to exceed the recommended requirements for DNS)
  • DNS: latest version (recognition accuracy, performance and features continue to improve)
  • DVR: highly rated for transcription accuracy, easy to manipulate (see Recommended Digital Voice Recorders for Dragon NaturallySpeaking Transcription blog post)
  • Microphone: Sylencer has sensitive microphone, eliminates background noise, USB adapter for laptop (converts analog audio into digital audio, bypassing sound card which may not be the best quality)
Train To Become Proficient In Technology Use
The five evaluators that underwent one-on-one training presented with a range of training needs. Some were previous DNS users but had not received formal training nor engaged in much self-study through reading the DNS Help Topics, End-User Workbook, User Guide or online videos. Only one evaluator had used a DVR. All evaluators were new to the Sylencer.
One-on-one training was supplemented with step-by-step instruction guides for the different aspects and technologies used in the voice note-taking process. The evaluators were also made aware of the extensive online resources that Nuance makes available for DNS users. Finally, the evaluators were encouraged to practice and meet with me to work out any issues they were having.
Train Dragon NaturallySpeaking
As stated earlier, DNS is speaker-dependent, and while it recognizes speech quite well right out-of-the-box, its recognition accuracy improves after the new user trains it during the user enrollment process, and on an on-going basis. Training DNS has two effects. First, it refines the acoustic model which is how the user's voice sounds and how they pronounce words. Second, it modifies the language model, or word usage. The initial training where the user reads from a preset script adjusts the acoustic model. But DNS learns continuously after that especially if the user takes the time to properly correct misrecognitions and customize the vocabulary.
Customize DNS Vocabulary
Post-secondary students may use more unknown or uncommon words, or use words in rarer senses leading to increased misrecognitions. Therefore, to improve recognition accuracy, it is important for the student to:
  • Prior to lecture
    • Import unknown words into the DNS dictionary
    • Analyze lecture notes to adjust word usage frequency and context of use
  • During dictation or after transcription
    • Correct misrecognitions
Students who are taking courses in a variety of subjects will need to perform vocabulary customization for each specialized vocabulary they are dealing with, which may be a time-consuming process.


As stated earlier, many students did not continue past the recruitment or the training phases into classroom evaluation. The two evaluators are either not using voice note-taking at all or not on a regular basis.
There are a variety of reasons for this outcome, all relating to the preliminary findings and criteria for success. For example, to get out of the starting gate the student needs to feel comfortable using Sylencer in public. They need to be able to hold the Sylencer and operate a laptop or DVR. They also require access to the appropriate technology e.g. DNS and a computer that meets DNS's requirements.
The step from training to evaluation has not yet been made by all evaluators due other factors. They may require more training or are not taking an appropriate course. They may have an unsuitable voice (though DNS has evolved to accommodate a wide range of voice qualities).
The two evaluators who actually used voice note-taking in the classroom received poor recognition accuracy with DNS. Again, a variety of factors are likely at play. One evaluator needs to practice speaking in phrases so perhaps more experience in using DNS in a conventional manner would be advisable. Also, custom vocabulary needs to be imported prior to the lecture. Finally, the evaluators likely require more time and practice to adjust their speaking style with the Sylencer.
In conclusion, the preliminary findings of this study into voice note-taking highlight the importance of taking a holistic approach to assessing the wide variety of factors that can influence the successful adoption of an assistive technology. This approach needs to consider the characteristics of the user, the particular task(s) they need to perform, the setting(s) they will be in and the features of the technologies they will be using. Once these factors align, hopefully with the help of the criteria for success this study has identified, students who cannot use typical note-taking modalities may find voice note-taking helps them to achieve greater independence in their educational pursuits.

Wednesday, December 7, 2011

iOS Recording Apps For Dragon NaturallySpeaking Transcription: A Feature Comparison

My previous blog post analyzed how well recordings made on the iOS platform compared to a typical well-rated digital voice recorder for Dragon NaturallySpeaking (DNS) transcription. I tested the Voice Memos app that comes preinstalled on most iOS devices (the exception being the iPad) and the Andrea PureAudio Live Recorder app which was recommended by Nuance at the time.
But in October 2011 Nuance released its own recorder app for the iOS platform: Dragon Recorder. While I haven’t applied the analysis procedure to this app as outlined in my Recording on the iPhone for Dragon NaturallySpeaking Transcription blog post, the following feature comparison shows that Dragon Recorder nicely fits the requirements for DNS users looking to create recordings for transcription on the iOS platform.
Note however that Nuance says: "While the Dragon Recorder app introduces more customers to the freedom and flexibility of remote dictation, customers requiring advanced recording functionality can continue to rely on Dragon’s ability to transcribe the audio files of the industry’s leading digital voice recorders."
Feature / AppVoice MemoPureAudio Live RecorderDragon Recorder
File Type Accepted by DNSNo (M4A, but can use iTunes to convert to MP3)Yes (WAV)Yes (WAV)
Audio quality settings1 (64 kbps bit rate, 44.1 kHz sample rate)/td>4 sample rates: Low (8kHz), Med (11kHz), High (22kHz), CD (44kHz) (Nuance recommends High)1 (352 kbps bit rate, 22 kHz sample rate)
File sizeabout 4.6 MB for 10 min. recordingWAV is about 5 times larger than M4A / MP3WAV is about 5 times larger than M4A / MP3
Voice activated recordingNoYes (removes pauses in speech for shorter recordings but you may need to experiment with its “Activation threshold” slider to adjust the sensitivity so your voice isn't clipped)No
Background noise cancellationNoYes (but turn off if using the Sylencer to take voice notes)No
Pause during recordingYesYesYes
Append to end of recordingNoYesYes
Wireless file transfer to your computerNo (must use iTunes sync)YesYes
Rename fileNoYesYes
Create categories (i.e. folders) for recordingsNoYes (a good feature if you have lots of recordings but you should delete them from your iDevice after transferring / transcribing)No
Costfree (preinstalled in Utility folder on iPhone & iPod Touch)$2.99 (download from iTunes Store)free (download from iTunes Store)

More Info

Getting Started With Dragon Recorder App (video that demonstrates the end-to-end process of creating a recording, transferring it and transcribing it)
Dragon Recorder Application Home Page
Dragon Recorder App Press Release
PureAudio Live Recorder iTunes Preview

Tuesday, June 14, 2011

Recording on the iPhone for Dragon NaturallySpeaking Transcription


Dragon NaturallySpeaking (DNS) users have been purchasing digital voice recorders (DVRs) to take advantage of DNS's transcription feature. But the iPhone and iPod Touch (hereafter I'll use "iPhone" to refer to both devices) also allow you to record dictation that can be transcribed into text by DNS. Using a single device for multiple purposes has many advantages including being more cost-effective, and reducing the time and effort to learn how to operate and maintain another device. But the quality of the recording is critical to transcription accuracy; Nuance even tests and publishes quality ratings for a variety of recording devices.  So how do iPhone recordings compare to a typical, well-rated DVR?
I decided to try to answer this question as part of a study I'm conducting on voice note-taking for the University of Calgary's Disability Resource Centre. Voice note-taking allows students who cannot hand-write or type lecture notes to make notes by voice. They speak into a special hand-held microphone, called the Sylencer, which muffles their speech and acts as a portable sound booth to eliminate background noise. The Sylencer is connected to either a DVR or a laptop. The DVR option creates a recording that the student can later have DNS transcribe into text. With the laptop, notes are dictated into DNS in real-time.
For the voice note-taking study, I selected three DVRs that are rated for high transcription accuracy with DNS and offer a variety of features and form factors with low to mid-range affordability. The Sony ICD-SX68 DVR was available to me at the time I did this analysis, so I used it to compare to the iPhone.
Because most iPhone users won't be using a Sylencer, I also tested the iPhone with a conventional headset, the Andrea NC-185VM. It is rated 6 out of 6 on the DNS Hardware Compatibility List.

Options For Recording on the iPhone

I used two recording methods on the iPhone. It includes an app called "Voice Memo" in its Utility folder. The major disadvantage of this app is that the recording file type (M4A) is not accepted by DNS so it needs to be converted into a compatible format (e.g. MP3)  which is easy to do in iTunes once you've customized the import settings. Also, the recording is made with a 64 kbps bit rate and 44.1 kHz sample rate; you can downgrade these settings when you customize the import settings in iTunes (e.g. a 16 kbps bit rate and 11.025 kHz sample rate results in an MP3 file one-quarter the size of the original) but I've found that transcription accuracy really suffers.
In its evaluation of the iPhone as a recorder, Nuance recommends using the Andrea Pure Audio Live Recorder app. It is available at the iTunes Store for $2.99. The major advantage of this app is that it allows you to record in the WAV format which DNS accepts for transcription. There are other advantages including background noise cancellation, voice activated recording (VAR), 4 audio quality levels, naming a recording and assigning it to a category. However, Nuance recommends turning off the background noise cancellation feature, and Andrea recommends disabling both the noise cancellation and VAR features. Nuance recommends the "high" quality setting which produces files with a 22kHz sample rate. The "CD" quality setting is the highest and uses the same sample rate as the Voice Memo app: 44.1 kHz.
When I tested the Andrea headset with the iPhone, I used the Voice Memo app because at that point I had analyzed the results from using the Sylencer with these two apps, and I found that overall the Voice Memo app performed better than the Pure Audio app.

iPhone Microphone Adapter

To use the Sylencer microphone with the iPhone, I had to purchase a special adapter that converts from TR (microphone) to TRRS (iPhone). KVconnection sells a Y-shaped iPhone 1/8 inch electret condenser microphone and headphone adapter (3.5mm 4 conductor TRRS Male to 3.5mm TRS Jacks) that converts TR (mic) + TRS (headphone) to iPhone (TRRS).
The headphone plug is important because when the adapter is plugged into the iPhone, sound doesn't emit from the external speaker. The headphone plug will accept headphones, earbuds or desktop speakers.
Even more important is the fact that this adapter permits power from the iPhone to travel to the Sylencer which is necessary to power a "SmartMic" plug, found on some Sylencer models. The SmartMic plug requires power because the SmartMic enables the Sylencer to be tunable i.e. its sensitivity (loudness, softness) can be adjusted.
I also used the iPhone microphone adapter with the Andrea headset.

Analysis Procedure

On each platform, I made 3 recordings from the Success is a Journey DNS training script. Each recording was from a different section but contained the same number of words (881). I recorded on the following: Sony DVR, iPhone with Voice Memo app, iPhone with Pure Audio app. I recorded with the Sylencer microphone on each of these 3 device/app combinations but only recorded with the Andrea headset on the iPhone with the Voice Memo app. The transcripts were produced on the same computer using a DNS user profile (UP) that I had trained for each of the four test platforms. The UPs were either new or very recent so had not benefited from additional training through the transcript correction process.
I used the default or recommended audio settings for each device/app:
  • Sony DVR: recordings are made in the proprietary MSV format with highest quality (STHQ) and then converted in the Digital Voice Editor (DVE) application to the WAV format with 16 bit 11.025 kHz mono
  • iPhone with Voice Memo app: recordings are made in the M4A format with the only available quality setting and then converted in iTunes to the MP3 format with the equivalent audio settings of 64kbps 44.1kHz mono
  • iPhone with Andrea Pure Audio Live Recorder app: recordings are made in the WAV format with high quality (22kHz sample rate); they can be downloaded to your computer via Wi-Fi (browse to a particular web page) or iTunes via File Sharing in the Apps section.
To assess transcription accuracy, I used the Compare feature in Microsoft Word. I compared the transcript produced from a recording ("original document") to the correct version of the text ("revised document"). Because I was only interested in content differences, I selected only two comparison settings: Insertions and deletions, and Moves (infrequent but a good indication of a difference).

Analysis Results

Below is a table showing the Word 2010 Compare results for the 3 recordings made for the 4 platforms tested.  Usually an error will involve a deletion of the incorrect text and insertion of the correct text. But there tends to be more insertions than deletions because sometimes DNS omits words; you'll notice that accuracy may decrease as word count decreases.
"Success is a Journey" pages (881 words)MicDeviceFile TypeFile Size (MB)Rec. Time (min.)# wordsIns./ Del./ MovesTotal
6 - 7Andrea HeadsetiPod Touch, iOS 4, Voice Memo appMP3 44kHz 64kbps mono6.0113:0888227 / 22 / 049
6 - 7SylenceriPod Touch, iOS 4, Voice Memo appMP3 44kHz 64kbps mono5.6912:2589034 / 33 / 067
6 - 7SylencerSony ICD-SX68WAV 11kHz 16 bit mono16.413:0388743 / 36 / 079
6 - 7SylenceriPod Touch, iOS 4, Pure Audio appWAV 22kHz 16bit mono33.813:2389445 / 43 / 290
8 - 9Andrea HeadsetiPod Touch, iOS 4, Voice Memo appMP3 44kHz 64kbps mono5.0110:5788329 / 23 / 052
8 - 9SylenceriPod Touch, iOS 4, Pure Audio appWAV 22kHz 16bit mono29.311:3687741 / 40 / 081
8 - 9SylenceriPod Touch, iOS 4, Voice Memo appMP3 44kHz 64kbps mono4.6610:1087447 / 40 / 087
8 - 9SylencerSony ICD-SX68WAV 11kHz 16 bit mono13.910:4786156 / 44 / 0100
10 - 11Andrea HeadsetiPod Touch, iOS 4, Voice Memo appMP3 44kHz 64kbps mono4.7310:2087046 / 41 / 087
10 - 11SylenceriPod Touch, iOS 4, Voice Memo appMP3 44kHz 64kbps mono4.549:5587251 / 44 / 095
10 - 11SylencerSony ICD-SX68WAV 11kHz 16 bit mono1310:1887754 / 51 / 0105
10 - 11SylenceriPod Touch, iOS 4, Pure Audio appWAV 22kHz 16bit mono27.310:4988156 / 55 / 0111
Word 2010 Compare Feature Issues
For a number of reasons, the data reported by the Word 2010 Compare feature should only be used as a starting point for further analysis of the differences between the transcript and the original text. First, the number of insertions, deletions and moves reported may involve more than one word. For example, "can't trade him and made" is marked as a single deletion that is replaced by "concentrate on their main", a single insertion. So it is important to visually scan the compared document which uses colour highlighting and strikethrough font to show the differences.
Second, if there are major differences, large sections of text may be deleted and replaced, reducing the insertion and deletion count and thus making the transcription accuracy look more favourable. But a visual scan will show many colour changes while a closer analysis of these can reveal that the large chunks of inserted and deleted text have words in common. While I didn't encounter this situation with any of these transcriptions, I have seen this behaviour i.e. a compare reports the fewest insertions and deletions, yet most of the text is highlighted as being different while a closer look shows that some deleted chunks of text do have correctly transcribed content.
So, this comparison requires looking at the statistics as well as a close visual scan of the differences. And, at best, one can only use this tool to decide which platform is better or worse than another (versus quantifying by how much). Based upon my visual scan of the differences, the Compare results do accurately reflect which combinations of mic, device and app perform relatively better:
  • The Andrea headset gives better recognition accuracy than the Sylencer microphone.
  • Overall, iPhone Voice Memo app is likely better than the Pure Audio app.
  • Overall, iPhone offers better recognition accuracy than the Sony DVR.
  • iPhone Voice Memo file sizes are about 3 times smaller than Sony WAV files and 5-6 times smaller than Pure Audio WAV files.
Sylencer Microphone Issues
You might be surprised at the large number of errors when using the Sylencer, in even the Sony DVR recordings. This microphone has its pros and cons. It is a very sensitive microphone that by design also eliminates background noise which should give it a high signal-to-noise ratio. But in order to muffle one's voice, the user has to speak quietly, in an undertone, which means speaking more from the throat. Also, it is important to create a good seal with the mask around the face but this restricts the movement of the facial muscles. These two factors mean that the user must spend time mastering the art of dictating with the Sylencer to achieve good recognition accuracy with DNS.
DNS Transcription Issues
I've found that transcribing the same recording multiple times produces slightly different results even though no corrections were made to improve accuracy.
A Note On File Formats
I used the recommended (Sony DVR, Pure Audio) or default (Voice Memo) file format for the 3 device/app combinations tested. I also converted one of the Sony DVR recordings to 3 other formats offered by the DVE application and received similar results from the Compare analysis (WAV 44kHz stereo, MP3 44 kHz 160kbps stereo, MP3 44 kHz 128kbps stereo). Because the default sample rate is 44kHz for a Voice Memo recording, I also made a Pure Audio recording at 44kHz; its recognition accuracy was similar to the 22kHz recording.
Finally, note that MP3 is a compressed file format so minimize the number of times it is converted as this process reduces the quality of the audio.


While the iPhone Voice Memo app does not offer the features you can find on a DVR or in the Pure Audio app, it performed best in this analysis compared to the other methods tested in terms of recognition accuracy for DNS transcription. Using the iPhone for recording dictation is also cost-effective and convenient because you don't have to purchase, learn and maintain a separate device. If you understand iTunes well enough to customize the import settings and will mostly import recordings for transcription, then using the Voice Memo app is your best option. It's easy to create an MP3 version of the recording in iTunes, the file sizes are smaller and they seem to produce better accuracy than the Pure Audio recordings.
However, the bottom line is that there are several factors that affect DNS transcription recognition accuracy: recording device, recording settings, microphone, file format, DNS version, computer specifications, and DNS user profile training. Therefore, it's important for you to perform your own testing, and if necessary, adjust these factors until you achieve acceptable recognition accuracy.


PureAudio Live Recorder User Guide
Using PureAudio Live Recorder with Dragon NaturallySpeaking
Wikipedia article on TRS Connector

Monday, April 25, 2011

Removing Livescribe Smartpen Content

Removing pages that you are no longer interested in from a traditional notebook is easy - you simply tear them out and shred or recycle them. But there is much more involved in removing pages from a Livescribe notebook. The process depends on whether you’ve recorded audio along with the notes on those pages, whether you’ve transferred the content to your Livescribe Desktop (LD) application, and whether you plan to continue to use the notebook for study or note-taking purposes.
Note: This article pertains to version 2.3.4 of the Livescribe Echo Smartpen and Windows Livescribe Desktop application software.


Before we discuss each of these scenarios, let’s review the terms, definitions and processes involved in using a Livescribe smartpen to take and refer to notes.
Making Notes
Taking notes with your smartpen involves writing or drawing in a Livescribe notebook which contains dot paper i.e. plain paper printed with a microdot pattern that allows the smartpen to recognize areas on the paper so it can associate your handwritten notes with recorded audio. Your notes may or may not have audio associated with them; it depends on whether you chose to record sound with the smartpen microphone prior to writing your notes. A session is a single, complete audio recording that may optionally have notes. So you have 3 note-taking options: notes + audio, notes only and audio only. Also, keep in mind that you can add notes to an audio recording at a later time.
Accessing Content In Notebook / Smartpen
Once you have made your notes, you can access them in several ways. If you recorded audio while writing notes, tap any note to start play back from the moment you wrote that note. If you didn’t record audio, you just read the notes as you would if they were written on normal paper. To play audio that doesn’t have notes associated with it, go to the Paper Replay application in the smartpen’s Main Menu, select Play Session and then select the audio session according to its date and time.
Accessing Content In Livescribe Desktop
You can also access your notes from Livescribe Desktop, the software that you install on your computer to allow you to transfer, store, search and replay notes from your computer. When you connect your smartpen to your computer, any new notes and audio you’ve added since the last connection are copied to your computer. In LD, you can view the notes in a particular notebook. Notes that have audio associated with them appear in “active ink”. Ink refers to the markings the smartpen makes on dot paper and active ink displays in green by default. Notes not associated with recorded audio display in “inactive ink” which is black by default.
So now you understand why you want to turn the smartpen on when recording notes only; it allows those notes to be captured by the camera, so they transfer to LD. The smartpen contains a ballpoint ink cartridge, allowing you to write on any type of paper but if the pen is turned off when you take notes in a Livescribe notebook, those notes won’t be captured for viewing in the Desktop application.
There are several ways you can listen to a recording in LD. If the session has notes associated with it, and you want to view the notes while listening to the recording, go to Pages View, select the appropriate notebook in the Navigation Pane, select the page containing the notes, and click the note at which you want to begin play back of the audio recorded as you wrote the note. The note must display with active ink. You’ll notice that notes recorded prior to this point in the session are green, while “future” notes are greyed-out and traced in green as the audio plays.
If the session doesn’t have notes or you don’t want to view the notes while listening to the recording, you can play the recording from the Audio View. Locate the recording by its date and time from the Session Name column. Double click the list item to start play back.

Remove Content From Smartpen

Removing notes from a Livescribe notebook requires you to consider whether the notes were made with the pen on, are associated with a recording, were transferred to your computer and are in a notebook that you still want to use.
If you wrote in your Livescribe notebook without turning the pen on, your writing wasn’t recorded and you would not have been able to record audio as well. So the pen does not contain a session nor does it even have a copy of your ink marking for transfer and viewing in LD. In this case, tearing that page out of the notebook is okay as long as all of the notes on both sides were made with the pen off.
Delete Session From Smartpen
Recall that a session is an audio recording that can optionally be associated with notes. The Paper Replay application in your smartpen allows you to delete an individual session or all sessions (select Delete Session or Delete All Sessions). This has several implications. First, once you delete a session from your smartpen, your paper notes will no longer play back the audio that was associated with that ink. Second, if you had transferred the session to LD, the session will still exist there. Third, if you delete the session before transferring it, the ink will still appear in LD, and due to a bug:
  • you'll receive the following error message after connecting your smartpen: "The following items failed to transfer: Audio Recordings"
  • new sessions won't transfer to Livescribe Desktop unless you archive your notebook (see Archive Notebook below for the implications of this).
I've notified Livescribe about this bug so hopefully it will be resolved in a software update in the near future. In the meantime, I recommend that you do not delete a session from the smartpen until AFTER transferring it to LD.
Delete Audio Session From Livescribe Desktop (and Smartpen)
Livescribe Desktop allows you to delete audio either from the LD application or from the smartpen.
To delete a recorded audio session from LD only, select Audio View, click the audio session to delete, right-click and select “Remove Audio from Livescribe Desktop…”. Deleting a session from Livescribe Desktop does not affect your smartpen.
To delete a recorded audio session from the smartpen only, ensure that the smartpen is connected to your computer. Select Audio View, click the audio session to delete, right-click and select Remove Audio from Smartpen…. This has the same effect as using the Delete Session command in the smartpen’s Paper Replay application.
As stated above, the implications of deleting an audio recording from the smartpen are that your paper notes will no longer play back the audio that was associated with them. If you delete audio from LD, you’ll no longer be able to play it on your computer.
If you wrote with the pen on, but made no recording, your notes will be transferred to LD where you can view them as inactive ink.
Delete Pages From Notebook
If you want to remove pages from a notebook in LD, you must first archive the notebook. Only an archived notebook can have pages deleted from it. You can also delete an entire archived notebook.
Archive Notebook
When you have finished using a notebook, you should archive it by connecting the smartpen to your computer and selecting the notebook in the Library tab of the Navigation Pane, and selecting Archive Notebook... from the File menu. Archiving a notebook moves it from the Library folder to the Archived Notebooks folder in the Navigation Pane. It also deletes your notes (ink data) and audio from the smartpen. This frees up storage space on your smartpen but also means that you won’t be able to use the smartpen to interact with the notebook. That is, when you tap notes in the physical archived notebook, they won’t play back any audio. However, you can still use LD to access the notes and audio in an archived notebook in the same manner as a non-archived one. Another important reason to archive a notebook is that it allows you to start a new notebook that shares the same series number (and thus the same microdot pattern).
If the archived notebook still has blank pages in it, you can still write in it. (But only do this if you want to treat the unused pages as part of a new notebook and don’t do it if you have started another notebook with the same series/name.) The Livescribe Knowledge Base says, "When you connect your smartpen to transfer your new notes, that notebook will re-appear in the active notebook section of Livescribe Desktop, and the additional pages you wrote on will appear as thumbnails when the active notebook is selected." You can archive the same notebook as many times as you wish.
For example, in a school setting, a teacher may want to pass a smartpen and partially used notebook on to another student. Of course the sessions on the smartpen should be deleted. Archiving the notebook does this. Then the teacher should remove the written pages from the notebook so another student won’t see work from the previous student. Finally, if the teacher doesn't want to keep an electronic copy of those written pages, s/he can delete them from the archived notebook.
Note that if you write on archived pages (i.e. pages that had content), those pages will appear in the active notebook with the new ink that you applied.
Deleting Archived Notebooks and Pages
If you want to permanently remove an archived notebook or some of its pages, use the Delete command as follows: Select an Archived Notebook in the Library tab. Select Delete Notebook... from the File menu to remove the entire notebook. Or, select one or more page to be deleted (use the Ctrl / Shift to multi-select) and select Delete Page(s) from the File menu. Note: If there is audio associated, at the prompt choose if you also want to delete the notebook’s or page’s audio. Also, if you have custom notebooks that link to the pages you're deleting, these pages will also be deleted from the custom notebooks.


Livescribe Smartpen User Guide
Livescribe Desktop for Windows User Guide
Livescribe Knowledge Base

Thursday, April 21, 2011

Contextual Spell Checker Evaluation

Conventional spell checkers fix spelling mistakes (i.e. non-word errors) by essentially performing a dictionary look-up of a word, and fix grammar errors by applying a set of grammatical rules. Contextual spell checkers are a new class of software that go beyond this; they use context (i.e. a number of words surrounding the word, most typically a sentence) to identify and correct misused words, also known as real word errors.
While researching the contextual spell check feature first introduced into the 2007 edition of Microsoft Word, I discovered an evaluation of four contextual spell checkers including Word 2007 (on Windows), MS Word 2008 (on MacOS X), MacOS X Spell and Grammar Checker, and After the Deadline.
The evaluation was conducted by Raphael Mudge, the developer of After the Deadline, an open-source grammar, style, and misused word checker that can be added to various applications as an extension e.g. Firefox web browser, word processor. You can read his 2010/4/9 blog posting about this at Measuring the Real Word Error Corrector. Mudge has made available the data set and programs to perform this evaluation on other such tools. The data consists of 673 sentences containing 834 errors (of which Mudge determined 97.8% are real word errors) that were collected from writers with dyslexia by Dr. Jennifer Pedler for her PhD thesis. Pedler annotated the errors along with the expected corrections and Mudge wrote a program to compare a corrected version of Dr. Pedler’s error corpus to the original corpus with errors. The program outputs 2 results as a percentage:
  • recall: how many errors were found and changed to something
  • precision: how often these changes were correct
Note: Mudge's program does not measure the number of words outside the annotated errors that were changed correctly or incorrectly. This is unfortunate because I noticed many false positives when using Ghotit i.e. correct words that were identified as errors.
So I used Pedler's data and Mudge's script to evaluate Word 2007 (to attempt to replicate Mudge's results), Word 2010 (to compare it to Word 2007), and two relatively new online contextual spell checkers: Ginger Software and Ghotit. When doing so, I accepted the first suggestion offered by each checker. For Ginger, I ran the test twice: I first approved the corrected sentence exactly as offered and then on a fresh copy of the error corpus, I explicitly selected the first suggestion for those corrections annotated with the ? icon as these usually remain unchanged. So selecting the first suggestion would show that Ginger did identify a potential error (improving recall) and may correct the error (potentially improving precision).
Spell CheckerRecall % (identifies word)Precision % (corrects identified word)
Ginger Software v1.16.1 (select first suggestion for corrections labelled "?")56.688.8
Ginger Software v1.16.1 (approve default sentence)50.989.6
MS Word 200740.690.3
MS Word 201038.689.4
After the Deadline2888
My results show that Ghotit scores best at identifying errors with 72.2%. This is well ahead of the second highest recall value of 56.6% achieved by Ginger, when explicitly selecting the first suggestion; approving the corrected sentence exactly as offered identifies 50.9% of errors. The fourth highest recall is MS Word 2007 which identifies 40.6% of the errors.
But as I said above, I noticed that Ghotit generated a lot of false positives (the manner in which the Microsoft Word plugin interface is designed required me to select the first suggestion from a submenu for each highlighted error). This seems to have increased recall but notice that the precision score of 77.9% is significantly lower than the 88 - 90% that all of the other spell checkers achieved.
The error corpus does use UK English and I was able to specify that (or it was the default) for all the spell checkers I evaluated.
Note that the results for Ginger and Ghotit are subject to change as their algorithms and data sets are continuously being improved.

Tuesday, November 30, 2010

Using ZoomText With Dragon NaturallySpeaking

I am familiar with ZoomText but was not sure if it was compatible with Dragon NatInstalling and Using AIM for Facebook Chat With JAWS Screen ReaderurallySpeaking (DNS). So I tried both on a Dell laptop (ZoomText Reader/Magnifier 9.1; Dragon NaturallySpeaking Preferred 10.1; Dell Laptop with 2 GB RAM, Intel Core2 Duo CPU T5850 @ 2.16GHz 2.16GHz, Win XP SP3). They seemed to behave well together.
You do need to pay attention to the system requirements for these applications because they require more processing power and memory than conventional productivity applications. With my setup, DNS consumes about 300 MB of memory, ZoomText about 53 MB and Microsoft Word about 37 MB. I am also running McAfee which sometimes demands about 177 MB of memory and about 66 MB at other times.
I was able to dictate into a Word document and then invoke the ZoomText AppReader to highlight and read the text that I had dictated; I was also able to select some text and say, "Read That," to have DNS read it. What ZoomText didn't do was echo each word that was typed (I had the Typing Echo option set to Word) but that's understandable given that DNS outputs text in phrases/chunks, and by navigating through the text (e.g. word-by-word with Control Left/Right Arrow), ZoomText reads the text. Notice that the DNS Read That command does not highlight/identify the particular word being read as ZoomText does with its AppReader and DocReader tools.
I was also able to give ZoomText commands by voice (e.g. Press Control Numpad Plus to increase the AppReader's reading rate, or Press Alt Numpad Plus to increase magnification) and the commands were correctly recognized by DNS and processed by ZoomText.


While researching the compatibility of DNS and ZoomText, I learned that there is a software application called MagniTalk that supposedly acts as a bridge between DNS and ZoomText. I found several hits on MagniTalk using Google but they all referenced the same vague marketing blurb that states, "MagniTalk provides direct speech access to the ZoomText user interface (UI) with NaturallySpeaking Pro voice commands, serving as a bridge between these two exciting technologies. This means that users of the ZoomText Magnifier & Reader can more easily benefit from the advantages provided by Dragon NaturallySpeaking, continuous speech recognition software." I haven't been able to determine what features MagniTalk really provides and why it is needed on top of DNS and ZoomText. One additional piece of info I found said, "allows the coordinated use of the Dragon NaturallySpeaking voice input program and the Zoomtext screen magnification program."
Note that MagniTalk requires DNS Professional as it is likely using the scripting capabilities included in that version of DNS. The Preferred/Premium version lacks this feature.
Still, by learning a few ZoomText hotkeys, it is possible to work quite efficiently with both ZoomText and DNS. Many ZoomText users are likely already familiar with the hotkeys. Trainers tend to encourage their use because it makes using ZoomText more efficient. DNS processes keyboard commands well though I found found a few glitches that you should pay attention to; see the ZoomText Hotkeys section below.
So I don't think DNS Preferred/Premium users should despair. For the average user, Magnitalk isn't required. Note: Magnitalk costs $290US.

ZoomText and DNS Settings

Spend some time experimenting with the different ZoomText tracking settings. You might find that the following work for you:
  • Tracking
    • Track on: Mouse Pointer, Text Cursor, Menus, Controls Windows
    • Area to track: Entire Screen
  • Alignment
    • Mouse pointer: Within edge margins (15%)
    • Text Cursor: Within edge margins (15%)
    • Control and menu item: Centered within the zoom window
  • Mouse
    • Route pointer into view when it is moved
    • Route pointer over the active control
Note that after modifying ZoomText settings, you must select Save As Default from the File menu to save the settings for future sessions.
These DNS options are recommended:
  • View
    • DragonBar Mode: Docked To Top, Show Messages
    • Then create a Freeze Window for the DragonBar. This will allow the DragonBar to always to appear in the magnified window so you can observe e.g. microphone status, message line. The size, position and magnification level of the Freeze Window can be adjusted; it sits on top of the Full zoom window while the Full zoom window continues to scroll and display all areas of the screen. Create the Freeze window as follows:
      • ZT Magnifier menu, Freeze Window... dialog box
      • select the left-hand side so the microphone status and the message line will always be in view; if your magnification level doesn't permit both the mic status and message to remain in view, select only the left-hand side of the message area
      • use "Press Control Shift E" to enable/disable Freeze Window
      • note that the Freeze Window only works with the Full zoom window type and you cannot interact with it i.e. if a DragonBar menu is visible in the Freeze Window, you cannot bring the keyboard focus to it or click it with the mouse -- you need to navigate to the part of the screen that contains the actual DragonBar to interact with it.
    • Results Box: uncheck Anchor to allow Results Box to float around the window as you dictate
  • Commands
    • Require "click" to select hyperlinks in HTML windows
    • Require "click" to select menus and control
  • Miscellaneous
    • Use screen reader compatible menus
    • Use Active Accessibility for menu and dialog control

Using the DNS Correction Menu

To display the Correction menu, select the text you want to correct and say "correct that" or say "Press Num Pad Minus" or press Num Pad Minus key on the keyboard.
Voice commands that work in this dialog include:
  • "Press down arrow" to highlight an alternative and "Press Enter" to activate it OR "Press <number of alternative>" to both select and activate it e.g. "Press one"
  • "Press Alt P" to play back your dictation; the advantage of using this hotkey is that you won't hear ZoomText speak the name of the menu item and its tutor message
  • "Press Alt S" to open the spell dialog box; for best results, use the radio alphabet to spell the word(s)

ZoomText Hotkeys

Below are what I consider to be the most useful ZoomText hotkeys. You can issue these by voice with DNS. Keep the following in mind:
  • giving these commands to DNS by voice will increase the response time slightly over typing the commands directly via the keyboard
  • be sure to precede each of these hotkeys with the word "Press" (e.g. "Press Alt Shift Q")
  • do not pause in the middle of one of these commands; pause before and after
  • if DNS is not recognizing one of the modifier keys when speaking multiple keys, try switching the order e.g. I've had better success with "Press Alt Control W" than with "Press Control Alt W"
  • if DNS is misrecognizing a letter, use the radio alphabet e.g. "Press Alt Control Whiskey"
  • DNS won't accept a single modifier (e.g. "Press Control"); fortunately, you can use "Press Alt Shift Q" for the Quiet command
  • if you are using a laptop that lacks a Num Pad, you may have trouble speaking the ZoomText hotkeys that use "Num Pad 5"; you can work around this by modifying the hotkey for these in the Hotkeys dialog box that's found in the Settings menu e.g. change "Control Num Pad 5" to "Control ; (semicolon)" for the Say Current Word command.
  • AppReader: Alt Shift A
  • DocReader: Alt Shift D
  • Toggle Reading (Play/Pause): Enter
  • Exit AppReader/DocReader: Escape
  • Quiet Speech: Alt Shift Q
  • Increase Reading Rate: Control Num Pad Plus
  • Decrease Reading Rate: Control Num Pad Minus
  • Increase Magnification: Alt Num Pad Plus
  • Decrease Magnification: Alt Num Pad Minus
Document Navigation
  • Next/Previous Word: Control Right/Left Arrow
  • Next/Previous Line: Down/Up Arrow
  • Next/Previous Sentence: Alt Control Right/Left Arrow
  • Next/Previous Paragraph: Control Down/Up Arrow
  • Scroll Down/Up: Alt Down/Up Arrow
  • Scroll Right/Left: Alt Right/Left Arrow
  • Jump Up/Down: Alt Page Up/Down
  • Say Window Title: Alt Control W
  • Say All: Alt Control A
  • Say Time: Alt Control I
  • View Mode On/Off: Control Shift V
  • Speech On/Off: Alt Shift S
  • Enable/Disable ZoomText: Alt Insert/Delete
  • Display User Interface: Control Shift U
  • Open Magnifier Menu: Alt M (when UI has focus)

Wednesday, October 27, 2010

Installing and Using AIM for Facebook Chat With JAWS Screen Reader

As I stated in my blog post on Resources For Getting Started With Facebook Using JAWS Screen Reader, Facebook acknowledges that the current implementation of its Chat application may cause problems for screen readers and they recommend that a desktop instant messaging client such as AOL Instant Messenger (AIM) be used instead. In this blog post, I explain how to use JAWS 11 to download and install the current version of AIM (7.4), login with your Facebook user name, and perform the basic tasks that will allow you to chat with your Facebook friends. But before I get started on that, I will summarize two podcasts that already exist about using JAWS with AIM.

AIM and JAWS Podcasts

During the last 28 mintues of the March 2010 FSCast, Jonathan Mosen interviewed AOL Accessibility Director Tom Wlodkowski about how well JAWS is working with the new social networking features, including Facebook Chat, in AIM 7.2. Tom said that the objective of AIM is to become a universal communications platform. Previously, it permitted text, video and voice communication only with other AIM users but starting with version 7.2, AIM is now integrated with Facebook Chat. AIM also includes a new component called Lifestream which Tom describes as a "social network aggregator." Lifestream allows you to review the feeds from all of the social networks you've set up. You can update your status within AIM and have it posted to these social networks. Tom demonstrated how to sign in to AIM using your Facebook user name, locate a Facebook friend in your Buddy List and start a chat session with them. He also demonstrated some of the keyboard commands that make reviewing messages in the Conversation window easier. These commands are courtesy of the AIM scripts which were written by Doug Lee of SSB BART Group and are now included in the JAWS installation as of version 11.
In April 2010, Darragh Ó Héiligh of DigitalDarragh published a podcast about Using facebook chat with AIM and Jaws 11. Darragh wasn't able to sign into AIM 7.2 with his Facebook user name so he tried to create an AIM user name within the AIM application. He didn't find the form accessible (the only part I had difficultly with in AIM 7.4 was the audio CAPTCHA -- nothing happened when I activated the "audio" link) so he went to the AIM web site to create an AIM user name. That process plus the difficulties he had completing the audio CAPTCHA means that the first 26 minutes of the 44 minute podcast covers only downloading, installation and AIM account creation.

Installing AIM

The JAWS scripts for AIM make AIM more accessible to JAWS and are included in the JAWS installation after version 11.0.756 (see AIM 7.2 Integrates with Facebook Chat). Note that the AIM 7 and Accessibility option help page states that "For optimal performance, JAWS needs to run AIM-specific scripts when AIM is in use. Freedom Scientific includes these scripts in their standard JAWS installation but occasionally changes may be made to the AIM interface that require AOL to update the scripts. You can download and install AIM scripts at" I've compared the AIM scripts in the JAWS 11.0.1467 installation with those on the AOL web site and found that the scripts in the former are more up-to-date than those in the latter i.e. the AIM scripts installed with JAWS were last modified in February 2010 (and are located in C:\ProgramData\Freedom Scientific\JAWS\11.0\SETTINGS\enu) while the AOL version were last modified in August 2009 (are installed in C:\Users\<username>\AppData\Roaming\Freedom Scientific\JAWS\11.0\Settings\enu).
The steps for downloading and installing AIM are:
  1. Go to
  2. Locate and activate the Download AIM link
  3. Activate the Run button in the File Download and Internet Explorer Security Warning dialog boxes
  4. In the Installation Options dialog box, tab to and possibly uncheck the additional options; then tab to and activate the Install button
  5. Activate the Chat now button to start AIM
  6. You have 3 login options: Get an AIM user name, sign in with AIM or sign in with Facebook. Tab to the Facebook button and activate it.
  7. A Facebook login window opens. Type your Facebook user name / email and password. Then tab to and activate the Login button. [Note: If JAWS has difficulty with this form, exit from AIM (Alt Shift F4), restart AIM and try signing in again.]
  8. When you first sign in with your Facebook account, a Request for Permission form appears that describes what data from Facebook AIM plans to use. Tab to the Allow button and activate it.
  9. The Facebook login window closes and you're back at the AIM Login screen. The "Remember me" and "Automatically sign me in" checkboxes are enabled. The focus is on the Sign In button. Activate it.
  10. The Buddy List Builder window opens. By default it is set to import friends from your Facebook "address book." Enter your Facebook user name / email and password. Then tab to and activate the Find button.
  11. Activate the Continue link. To add Facebook to your Lifestream tab, locate and activate the Set up Lifestream link.
  12. The AOL Lifestream web page opens. Close it.
  13. Close the Buddy List Builder window e.g. activate the Close link.

Privacy Settings

To display the Privacy Settings:
  1. Open the Menu (Alt M) and select Privacy (press P).
  2. Tab through the various controls to review and possibly modify the following:
    • Who can IM me and see me online
    • Who can see my Buddy list
    • Allow others to see: there are 3 options here.
  3. Tab to and activate the Save button if you've made any changes to the settings.

AIM Settings

To review all of your settings:
  1. Open the Menu (Alt M) and select Settings (press S).
  2. A dialog box is displayed with a list of the types of settings on the left, and the controls to modify those settings on the right.
  3. Tab through the various controls to review and possibly modify your settings.
  4. When the focus is on the last control for that setting type (usually the Cancel button), press Tab to move to the list of settings; press up or down arrow to change the setting type; the controls on the right-hand side automatically appear so press Tab to access them.
  5. Particular settings you might want to review:
    • General IM: the JAWS Help recommends that you uncheck the "Flash the window when IM messages are received" check box
    • Sign In / Sign Out
      • When I sign in, display the following: select None radio button versus Welcome page by pressing down arrow
      • Start AIM when Windows starts
      • AIM Upgrades
  6. Tab to and activate the Save button if you've made any changes to the settings.

JAWS Verbosity Settings for AIM

When AIM has the focus, press Insert V to display the JAWS Verbosity settings dialog box. You'll find several AIM-related settings at the top. The default settings are probably okay.

Buddies, Lifestream and Me Tabs

When you login, you'll probably be positioned on the Buddies tab. The two other tabs you can access are Lifestream and Me. Press Control Tab to switch between these 3 tabs.
The Buddies tab is a tree view containing groups of people that you can communicate with. Use your up and down arrow keys to move up/down through the Buddies list. Press right arrow to open a group which displays a list of names beneath it. Press left arrow to close the group. If you have logged into AIM with your Facebook user name, you should now have a Facebook Friends group that lists all of your Facebook friends that are online. If your friend/buddy is not online, he/she will appear under the final group, called Offline. For quicker access, start typing a name to shorten the list to only those names containing the letters you typed. To display all of your buddies again, press Escape.
If you didn't sign in with your Facebook user name during the installation process, you can still do so from the Buddies tab. Tab to and activate the Facebook Connect link. A Facebook login window opens. Follow the instructions beginning with Step 7 in the Installing AIM section above.
The Lifestream tab brings together updates from social networking sites that you setup including AIM, Facebook, MySpace and Twitter. From here you can Comment, Like and update your status. You can filter which updates appear via the Expand Filter Options link; press Enter on it to open/close the list. There is also a search edit box which opens a new web browser window to display the results in the online version at Next you'll find a Refresh button. Tab through these controls and then through the updates from the sites you've integrated into AIM. Each update is a list item so you can use the navigation quick key, I, to move to the next update.
As described above, if you choose to sign in with your Facebook user name during the installation process, you'll be asked if you want to add Facebook to your Lifestream tab. You can add other social networks to Lifestream from the Me tab. Activate the appropriate link e.g. "Add Twitter to your Lifestream." You'll be taken to the Lifestream web site. A pop up dialog will describe what you'll be able to do on Lifestream and include two buttons: Go to service and Cancel. Depending on the service, you may need to sign in and/or allow Lifestream to have access to your data.
The Me tab shows a summary of information related to your AIM account. It also displays Facebook notifications and your updates. Press Tab to move through the links and controls. Press H to move to the Notifications or My Updates tabs as they are coded as heading level 3.
Updating Your Status
The "Set your status message" edit box appears above these 3 tabs so use Shift Tab to access it if you're in the Buddies list, etc. Type your status update here. Beneath is the "Post: button menu" to select which social networking site to post the update to; press Spacebar to open and Escape to close. Then tab to the "Set your status or hit Escape to cancel" button and activate to submit your update.

Chatting with a Facebook Friend

To start a chat with a Facebook friend, locate them in your Buddies list. Press Enter on their name. You can also press the Application key to display a context menu of actions you can perform. AIM refers to Facebook chat as IM (for Instant Message) so select that menu item. You'll notice that its shortcut is Alt + I.
A new "Conversation" window opens with the title "IM - <name>" and the focus is in the message edit box. Type your message, tab to and activate the Send message button. If you are within AIM but not in the Conversation window, JAWS alerts you when your friend replies -- you'll hear a sound and their name. The easiest method for reading a new message is to use the "review conversation history" command provided by the AIM scripts for JAWS. For example, press Alt+1 to hear the last message, Alt+2 to hear the second last, up to Alt+0 which reads the tenth previous message. Otherwise, you could always navigate through the Conversation History list. Press Shift Tab to move from the message edit box to the Conversation History list. Use Alt left and right arrow to move up and down through the list; JAWS will read your messages and the replies.
JAWS commands (courtesy of the AIM scripts) that you might want to try include:
  • Jump to the message edit box: CTRL+SHIFT+E
  • Jump to the Conversation History list: CTRL+SHIFT+H
  • Review Conversation History (Last 10 Messages): ALT+1, ALT+2, ALT+3, and so on through ALT+0, where ALT+1 is for the last message and ALT+0 is for the 10th previous message. Note that these commands leave the keyboard focus on the current control (most likely the message edit box!).
  • Move to a Specific Message: Press ALT+1 through ALT+0 twice quickly to move focus to a message. [Note: I've been unable to get this command to work properly; all variations of it move me to the second last message.]
When you are done with your chat, press Escape to close the Conversation window.

Learning More About AIM

The JAWS Help Topics contain information about AIM. You'll find it in the Popular Applications With JAWS section. The topic is entitled, AOL Instant Messenger. A quick way to display this is to press Insert F1 twice quickly when AIM has the focus.
To read the AIM Help, press F1. It displays in a new web browser window.
The AIM Help page about Chatting, Buddies & Away Messages contains a list of keyboard shortcuts. Many of these shortcuts appear in the Menu which you can open with Alt M.

Exit AIM

If you use the standard Windows close command, Alt F4, to close AIM, this merely closes the AIM window; it is still running in the background. You can reopen it from the Notification Area by opening its context menu and selecting Show Buddy List.
To completely exist from AIM, open the Menu (Alt M) and select Exit AIM (press X) or use the shortcut key (Alt + Shift + F4).