Thursday, December 8, 2011

Preliminary Findings From a Study to Evaluate Voice Note-taking: Using Dragon NaturallySpeaking in the Classroom

Introduction

This blog post presents preliminary findings from a study that I am conducting into voice note-taking for the University of Calgary's Disability Resource Centre. It states the purpose of the study, describes the technologies being evaluated, and explains the methodology being followed including the evaluation criteria.
The study began in November 2009 but is still on-going. Thus, I will give an update on its current status and the preliminary findings in the form of criteria for success with this note-taking approach.

Purpose

The purpose of the study is to evaluate technologies to enable voice recognition software to be used in the classroom for note-taking. This is a need for students who have a learning disability or motor impairment that may prohibit them from using traditional note-taking modalities e.g. handwriting, typing into a laptop. The study also aims to identify criteria necessary for success in the use of voice note-taking.

Technologies

The three main technologies being evaluated are:
  1. Dragon NaturallySpeaking (DNS) voice recognition software: Converts voice into text
  2. Sylencer handheld microphone: Muffles speech, minimizes background noise
  3. Digital voice recorder (DVR) for later transcription with DNS.
DNS, when used in its conventional manner, is not suitable for taking notes in a classroom because:
  • it is speaker-dependent i.e. trained to recognize a single voice
  • a particular style of dictation must be followed
  • no background noise must be present
  • dictating while the lecturer or other students are speaking is disruptive.
The Sylencer, also known as a Stenomask, is not a new technology. It is widely used in court reporting for "voice writing.” It was invented by Horace Webb as a method for bypassing shorthand to record court proceedings; his son owns Talk Technologies, the developer of the Sylencer.
The approach that Nuance, the company behind DNS, recommends for students who want to use the software in the classroom is to record the lecture and using “echo-dictating” to convert it into text after the fact. But this is time consuming - the most proficient echo-dictator will need to listen to the entire lecture at least once to capture it all.

Technology Combinations

Students are offered two approaches for note-taking with these technologies:
  1. Dictating with DNS onto a laptop using the Sylencer
  2. Dictating with DNS onto a DVR using the Sylencer and later transcribing the dictation with DNS on the laptop.
Each of these approaches offers advantages over the other. The DVR is easier to carry and presents fewer distractions while taking notes i.e. with the laptop, one can view the output from DNS and misrecognitions may be distracting especially if there isn’t time to correct. But voice note-taking directly on the laptop saves time: the student does not have to perform a transcription after class nor manage another device (the DVR). Also, the student may have time to edit or format notes during the lecture.
Note: Those students that are unable to hold the Sylencer are offered a third option: using a DVR to record the lecture and then echo-dictating it afterwards to convert it into text on their computer.

Methodology

The methodology includes the following phases:
  1. Initial Technology Evaluation: I tested the three technology combinations during two university lectures.
    • I was particularly interested in whether the Sylencer muffled my speech sufficiently and most students seated near to me reported that they did not hear me speaking. Some may have noticed my speech but could not understand it and were not distracted by it.
    • I learned that it is very important to be proficient in using DNS and the Sylencer, in order to keep up with the lecturer and maximize DNS’s recognition accuracy.
  2. Recruitment Session: When a student expresses interest in participating, we meet for a two hour session during which I:
    • demonstrate the technologies
    • explain the study purpose, procedure and participant responsibilities
    • if the student is interested in participating, complete a profile/pre-screening questionnaire to gather background information, assess their suitability as an evaluator, and assign them to one/more technology options to evaluate.
  3. One-on-one Training: to teach evaluator any technologies that are new to them
  4. Practice: administer a practice session to prepare the evaluator for the classroom
  5. Evaluation: evaluator takes voice notes in at least two lectures depending on the number of technology options they are evaluating and the time they require to provide an objective assessment; I attend the first session to observe technology use and assist with setup
  6. Notes Analysis: count number of corrections made to the transcription.
  7. Post-evaluation Interview: gather evaluator’s feedback about technology use and voice note-taking process; rate evaluation criteria.

Evaluation Criteria

The voice note-taking technologies are being evaluated for:
  • Training time
  • Ability to acquire the technology
  • Ease of use of the technology, especially in the classroom
  • Total time to produce text version of notes
  • Quality of notes
  • Effectiveness of entire note-taking process in achieving learning and retention of lecture material
  • Ability to use the technology independently i.e. motor skills
  • Ability to take notes independently i.e. dictation proficiency
  • Acceptance of technology by lecturer and other students

Current Status

Since the inception of the study in November 2009, eleven students have attended a recruitment session. Of these, five students agreed to participate and attended one/more technology training sessions for DNS, the Sylencer and/or the DVR. Two students took the Sylencer into the classroom: one recorded voice notes on a DVR while the other dictated onto both their laptop and a DVR. The latter student is continuing to evaluate voice note-taking. The reason for the drop in numbers between these three stages will be explained in the final section, Discussion.

Preliminary Findings

The preliminary findings can be grouped into 6 categories: personal characteristics, course selection, technology selection, training (both in technology use and for DNS) and DNS vocabulary customization. These findings are presented as criteria for success in the use of voice note-taking in the classroom.
Personal Characteristics
Personal characteristics covers a wide variety of factors that can determine whether the student is even a suitable candidate for voice note-taking, and can influence their success with it as well.
  • Comfortable using Sylencer in public (highlights disability)
  • Hold Sylencer
  • Operate the laptop or DVR
  • Suitable voice
  • Willing to learn the new technologies
  • Dictate well i.e. enunciate carefully, formulate phrase before speaking it
  • Adjust speaking style for Sylencer: Dictating using the Sylencer requires training and practice to master for two reasons.
    1. Its microphone is very sensitive so one must speak at a lower volume, in essentially an undertone as if one were speaking in a room where a small child was sleeping. A benefit of this quiet speech is that it makes the speaker less audible (and thus disruptive) to those nearby.
    2. Another design feature of the Sylencer that aids in muffling speech and minimizing background noise is the seal that its rubber mask makes when held to the face. But the mask also limits the movement of the facial muscles, making articulate speech more challenging and thus possibly compromising speech recognition as well.
  • Adjust note-taking style for DNS i.e. dictate in grammatically correct phrases versus the abbreviated note-taking style that suits physical task of writing at a quick pace
  • Multi-task i.e. listening and dictating compete for attention (versus listening and writing)
  • Manage switch between voice note-taking and class interaction e.g. removing Sylencer to answer a question, engage in laughter, etc.
Select Appropriate Course
Unfortunately, voice note-taking isn't suitable for all types of courses. DNS performs better under the following conditions:
  • Best initial DNS "out-of-the-box" results with a general vocabulary
  • Infrequent need to record special symbols, charts/graphs or diagrams that are best sketched by hand
  • Special versions of DNS are available for medical, legal, other languages (e.g. French, Spanish)
Select Appropriate Technologies
Voice recognition software is probably the most demanding application most users will ever run on their computer. This is why it is important to heed the system requirements. Nuance also rates devices used in conjunction with DNS such as DVRs and microphones. You'll achieve the best results if you observe the following:
  • Powerful computer: fast processor, lots of memory (try to exceed the recommended requirements for DNS)
  • DNS: latest version (recognition accuracy, performance and features continue to improve)
  • DVR: highly rated for transcription accuracy, easy to manipulate (see Recommended Digital Voice Recorders for Dragon NaturallySpeaking Transcription blog post)
  • Microphone: Sylencer has sensitive microphone, eliminates background noise, USB adapter for laptop (converts analog audio into digital audio, bypassing sound card which may not be the best quality)
Train To Become Proficient In Technology Use
The five evaluators that underwent one-on-one training presented with a range of training needs. Some were previous DNS users but had not received formal training nor engaged in much self-study through reading the DNS Help Topics, End-User Workbook, User Guide or online videos. Only one evaluator had used a DVR. All evaluators were new to the Sylencer.
One-on-one training was supplemented with step-by-step instruction guides for the different aspects and technologies used in the voice note-taking process. The evaluators were also made aware of the extensive online resources that Nuance makes available for DNS users. Finally, the evaluators were encouraged to practice and meet with me to work out any issues they were having.
Train Dragon NaturallySpeaking
As stated earlier, DNS is speaker-dependent, and while it recognizes speech quite well right out-of-the-box, its recognition accuracy improves after the new user trains it during the user enrollment process, and on an on-going basis. Training DNS has two effects. First, it refines the acoustic model which is how the user's voice sounds and how they pronounce words. Second, it modifies the language model, or word usage. The initial training where the user reads from a preset script adjusts the acoustic model. But DNS learns continuously after that especially if the user takes the time to properly correct misrecognitions and customize the vocabulary.
Customize DNS Vocabulary
Post-secondary students may use more unknown or uncommon words, or use words in rarer senses leading to increased misrecognitions. Therefore, to improve recognition accuracy, it is important for the student to:
  • Prior to lecture
    • Import unknown words into the DNS dictionary
    • Analyze lecture notes to adjust word usage frequency and context of use
  • During dictation or after transcription
    • Correct misrecognitions
Students who are taking courses in a variety of subjects will need to perform vocabulary customization for each specialized vocabulary they are dealing with, which may be a time-consuming process.

Discussion

As stated earlier, many students did not continue past the recruitment or the training phases into classroom evaluation. The two evaluators are either not using voice note-taking at all or not on a regular basis.
There are a variety of reasons for this outcome, all relating to the preliminary findings and criteria for success. For example, to get out of the starting gate the student needs to feel comfortable using Sylencer in public. They need to be able to hold the Sylencer and operate a laptop or DVR. They also require access to the appropriate technology e.g. DNS and a computer that meets DNS's requirements.
The step from training to evaluation has not yet been made by all evaluators due other factors. They may require more training or are not taking an appropriate course. They may have an unsuitable voice (though DNS has evolved to accommodate a wide range of voice qualities).
The two evaluators who actually used voice note-taking in the classroom received poor recognition accuracy with DNS. Again, a variety of factors are likely at play. One evaluator needs to practice speaking in phrases so perhaps more experience in using DNS in a conventional manner would be advisable. Also, custom vocabulary needs to be imported prior to the lecture. Finally, the evaluators likely require more time and practice to adjust their speaking style with the Sylencer.
In conclusion, the preliminary findings of this study into voice note-taking highlight the importance of taking a holistic approach to assessing the wide variety of factors that can influence the successful adoption of an assistive technology. This approach needs to consider the characteristics of the user, the particular task(s) they need to perform, the setting(s) they will be in and the features of the technologies they will be using. Once these factors align, hopefully with the help of the criteria for success this study has identified, students who cannot use typical note-taking modalities may find voice note-taking helps them to achieve greater independence in their educational pursuits.

Wednesday, December 7, 2011

iOS Recording Apps For Dragon NaturallySpeaking Transcription: A Feature Comparison

My previous blog post analyzed how well recordings made on the iOS platform compared to a typical well-rated digital voice recorder for Dragon NaturallySpeaking (DNS) transcription. I tested the Voice Memos app that comes preinstalled on most iOS devices (the exception being the iPad) and the Andrea PureAudio Live Recorder app which was recommended by Nuance at the time.
But in October 2011 Nuance released its own recorder app for the iOS platform: Dragon Recorder. While I haven’t applied the analysis procedure to this app as outlined in my Recording on the iPhone for Dragon NaturallySpeaking Transcription blog post, the following feature comparison shows that Dragon Recorder nicely fits the requirements for DNS users looking to create recordings for transcription on the iOS platform.
Note however that Nuance says: "While the Dragon Recorder app introduces more customers to the freedom and flexibility of remote dictation, customers requiring advanced recording functionality can continue to rely on Dragon’s ability to transcribe the audio files of the industry’s leading digital voice recorders."
Feature / AppVoice MemoPureAudio Live RecorderDragon Recorder
File Type Accepted by DNSNo (M4A, but can use iTunes to convert to MP3)Yes (WAV)Yes (WAV)
Audio quality settings1 (64 kbps bit rate, 44.1 kHz sample rate)/td>4 sample rates: Low (8kHz), Med (11kHz), High (22kHz), CD (44kHz) (Nuance recommends High)1 (352 kbps bit rate, 22 kHz sample rate)
File sizeabout 4.6 MB for 10 min. recordingWAV is about 5 times larger than M4A / MP3WAV is about 5 times larger than M4A / MP3
Voice activated recordingNoYes (removes pauses in speech for shorter recordings but you may need to experiment with its “Activation threshold” slider to adjust the sensitivity so your voice isn't clipped)No
Background noise cancellationNoYes (but turn off if using the Sylencer to take voice notes)No
Pause during recordingYesYesYes
Append to end of recordingNoYesYes
Wireless file transfer to your computerNo (must use iTunes sync)YesYes
Rename fileNoYesYes
Create categories (i.e. folders) for recordingsNoYes (a good feature if you have lots of recordings but you should delete them from your iDevice after transferring / transcribing)No
Costfree (preinstalled in Utility folder on iPhone & iPod Touch)$2.99 (download from iTunes Store)free (download from iTunes Store)

More Info

Getting Started With Dragon Recorder App (video that demonstrates the end-to-end process of creating a recording, transferring it and transcribing it)
Dragon Recorder Application Home Page
Dragon Recorder App Press Release
PureAudio Live Recorder iTunes Preview

Tuesday, June 14, 2011

Recording on the iPhone for Dragon NaturallySpeaking Transcription

Background

Dragon NaturallySpeaking (DNS) users have been purchasing digital voice recorders (DVRs) to take advantage of DNS's transcription feature. But the iPhone and iPod Touch (hereafter I'll use "iPhone" to refer to both devices) also allow you to record dictation that can be transcribed into text by DNS. Using a single device for multiple purposes has many advantages including being more cost-effective, and reducing the time and effort to learn how to operate and maintain another device. But the quality of the recording is critical to transcription accuracy; Nuance even tests and publishes quality ratings for a variety of recording devices.  So how do iPhone recordings compare to a typical, well-rated DVR?
I decided to try to answer this question as part of a study I'm conducting on voice note-taking for the University of Calgary's Disability Resource Centre. Voice note-taking allows students who cannot hand-write or type lecture notes to make notes by voice. They speak into a special hand-held microphone, called the Sylencer, which muffles their speech and acts as a portable sound booth to eliminate background noise. The Sylencer is connected to either a DVR or a laptop. The DVR option creates a recording that the student can later have DNS transcribe into text. With the laptop, notes are dictated into DNS in real-time.
For the voice note-taking study, I selected three DVRs that are rated for high transcription accuracy with DNS and offer a variety of features and form factors with low to mid-range affordability. The Sony ICD-SX68 DVR was available to me at the time I did this analysis, so I used it to compare to the iPhone.
Because most iPhone users won't be using a Sylencer, I also tested the iPhone with a conventional headset, the Andrea NC-185VM. It is rated 6 out of 6 on the DNS Hardware Compatibility List.

Options For Recording on the iPhone

I used two recording methods on the iPhone. It includes an app called "Voice Memo" in its Utility folder. The major disadvantage of this app is that the recording file type (M4A) is not accepted by DNS so it needs to be converted into a compatible format (e.g. MP3)  which is easy to do in iTunes once you've customized the import settings. Also, the recording is made with a 64 kbps bit rate and 44.1 kHz sample rate; you can downgrade these settings when you customize the import settings in iTunes (e.g. a 16 kbps bit rate and 11.025 kHz sample rate results in an MP3 file one-quarter the size of the original) but I've found that transcription accuracy really suffers.
In its evaluation of the iPhone as a recorder, Nuance recommends using the Andrea Pure Audio Live Recorder app. It is available at the iTunes Store for $2.99. The major advantage of this app is that it allows you to record in the WAV format which DNS accepts for transcription. There are other advantages including background noise cancellation, voice activated recording (VAR), 4 audio quality levels, naming a recording and assigning it to a category. However, Nuance recommends turning off the background noise cancellation feature, and Andrea recommends disabling both the noise cancellation and VAR features. Nuance recommends the "high" quality setting which produces files with a 22kHz sample rate. The "CD" quality setting is the highest and uses the same sample rate as the Voice Memo app: 44.1 kHz.
When I tested the Andrea headset with the iPhone, I used the Voice Memo app because at that point I had analyzed the results from using the Sylencer with these two apps, and I found that overall the Voice Memo app performed better than the Pure Audio app.

iPhone Microphone Adapter

To use the Sylencer microphone with the iPhone, I had to purchase a special adapter that converts from TR (microphone) to TRRS (iPhone). KVconnection sells a Y-shaped iPhone 1/8 inch electret condenser microphone and headphone adapter (3.5mm 4 conductor TRRS Male to 3.5mm TRS Jacks) that converts TR (mic) + TRS (headphone) to iPhone (TRRS).
The headphone plug is important because when the adapter is plugged into the iPhone, sound doesn't emit from the external speaker. The headphone plug will accept headphones, earbuds or desktop speakers.
Even more important is the fact that this adapter permits power from the iPhone to travel to the Sylencer which is necessary to power a "SmartMic" plug, found on some Sylencer models. The SmartMic plug requires power because the SmartMic enables the Sylencer to be tunable i.e. its sensitivity (loudness, softness) can be adjusted.
I also used the iPhone microphone adapter with the Andrea headset.

Analysis Procedure

On each platform, I made 3 recordings from the Success is a Journey DNS training script. Each recording was from a different section but contained the same number of words (881). I recorded on the following: Sony DVR, iPhone with Voice Memo app, iPhone with Pure Audio app. I recorded with the Sylencer microphone on each of these 3 device/app combinations but only recorded with the Andrea headset on the iPhone with the Voice Memo app. The transcripts were produced on the same computer using a DNS user profile (UP) that I had trained for each of the four test platforms. The UPs were either new or very recent so had not benefited from additional training through the transcript correction process.
I used the default or recommended audio settings for each device/app:
  • Sony DVR: recordings are made in the proprietary MSV format with highest quality (STHQ) and then converted in the Digital Voice Editor (DVE) application to the WAV format with 16 bit 11.025 kHz mono
  • iPhone with Voice Memo app: recordings are made in the M4A format with the only available quality setting and then converted in iTunes to the MP3 format with the equivalent audio settings of 64kbps 44.1kHz mono
  • iPhone with Andrea Pure Audio Live Recorder app: recordings are made in the WAV format with high quality (22kHz sample rate); they can be downloaded to your computer via Wi-Fi (browse to a particular web page) or iTunes via File Sharing in the Apps section.
To assess transcription accuracy, I used the Compare feature in Microsoft Word. I compared the transcript produced from a recording ("original document") to the correct version of the text ("revised document"). Because I was only interested in content differences, I selected only two comparison settings: Insertions and deletions, and Moves (infrequent but a good indication of a difference).

Analysis Results

Below is a table showing the Word 2010 Compare results for the 3 recordings made for the 4 platforms tested.  Usually an error will involve a deletion of the incorrect text and insertion of the correct text. But there tends to be more insertions than deletions because sometimes DNS omits words; you'll notice that accuracy may decrease as word count decreases.
"Success is a Journey" pages (881 words)MicDeviceFile TypeFile Size (MB)Rec. Time (min.)# wordsIns./ Del./ MovesTotal
6 - 7Andrea HeadsetiPod Touch, iOS 4, Voice Memo appMP3 44kHz 64kbps mono6.0113:0888227 / 22 / 049
6 - 7SylenceriPod Touch, iOS 4, Voice Memo appMP3 44kHz 64kbps mono5.6912:2589034 / 33 / 067
6 - 7SylencerSony ICD-SX68WAV 11kHz 16 bit mono16.413:0388743 / 36 / 079
6 - 7SylenceriPod Touch, iOS 4, Pure Audio appWAV 22kHz 16bit mono33.813:2389445 / 43 / 290
         
8 - 9Andrea HeadsetiPod Touch, iOS 4, Voice Memo appMP3 44kHz 64kbps mono5.0110:5788329 / 23 / 052
8 - 9SylenceriPod Touch, iOS 4, Pure Audio appWAV 22kHz 16bit mono29.311:3687741 / 40 / 081
8 - 9SylenceriPod Touch, iOS 4, Voice Memo appMP3 44kHz 64kbps mono4.6610:1087447 / 40 / 087
8 - 9SylencerSony ICD-SX68WAV 11kHz 16 bit mono13.910:4786156 / 44 / 0100
         
10 - 11Andrea HeadsetiPod Touch, iOS 4, Voice Memo appMP3 44kHz 64kbps mono4.7310:2087046 / 41 / 087
10 - 11SylenceriPod Touch, iOS 4, Voice Memo appMP3 44kHz 64kbps mono4.549:5587251 / 44 / 095
10 - 11SylencerSony ICD-SX68WAV 11kHz 16 bit mono1310:1887754 / 51 / 0105
10 - 11SylenceriPod Touch, iOS 4, Pure Audio appWAV 22kHz 16bit mono27.310:4988156 / 55 / 0111
Word 2010 Compare Feature Issues
For a number of reasons, the data reported by the Word 2010 Compare feature should only be used as a starting point for further analysis of the differences between the transcript and the original text. First, the number of insertions, deletions and moves reported may involve more than one word. For example, "can't trade him and made" is marked as a single deletion that is replaced by "concentrate on their main", a single insertion. So it is important to visually scan the compared document which uses colour highlighting and strikethrough font to show the differences.
Second, if there are major differences, large sections of text may be deleted and replaced, reducing the insertion and deletion count and thus making the transcription accuracy look more favourable. But a visual scan will show many colour changes while a closer analysis of these can reveal that the large chunks of inserted and deleted text have words in common. While I didn't encounter this situation with any of these transcriptions, I have seen this behaviour i.e. a compare reports the fewest insertions and deletions, yet most of the text is highlighted as being different while a closer look shows that some deleted chunks of text do have correctly transcribed content.
So, this comparison requires looking at the statistics as well as a close visual scan of the differences. And, at best, one can only use this tool to decide which platform is better or worse than another (versus quantifying by how much). Based upon my visual scan of the differences, the Compare results do accurately reflect which combinations of mic, device and app perform relatively better:
  • The Andrea headset gives better recognition accuracy than the Sylencer microphone.
  • Overall, iPhone Voice Memo app is likely better than the Pure Audio app.
  • Overall, iPhone offers better recognition accuracy than the Sony DVR.
  • iPhone Voice Memo file sizes are about 3 times smaller than Sony WAV files and 5-6 times smaller than Pure Audio WAV files.
Sylencer Microphone Issues
You might be surprised at the large number of errors when using the Sylencer, in even the Sony DVR recordings. This microphone has its pros and cons. It is a very sensitive microphone that by design also eliminates background noise which should give it a high signal-to-noise ratio. But in order to muffle one's voice, the user has to speak quietly, in an undertone, which means speaking more from the throat. Also, it is important to create a good seal with the mask around the face but this restricts the movement of the facial muscles. These two factors mean that the user must spend time mastering the art of dictating with the Sylencer to achieve good recognition accuracy with DNS.
DNS Transcription Issues
I've found that transcribing the same recording multiple times produces slightly different results even though no corrections were made to improve accuracy.
A Note On File Formats
I used the recommended (Sony DVR, Pure Audio) or default (Voice Memo) file format for the 3 device/app combinations tested. I also converted one of the Sony DVR recordings to 3 other formats offered by the DVE application and received similar results from the Compare analysis (WAV 44kHz stereo, MP3 44 kHz 160kbps stereo, MP3 44 kHz 128kbps stereo). Because the default sample rate is 44kHz for a Voice Memo recording, I also made a Pure Audio recording at 44kHz; its recognition accuracy was similar to the 22kHz recording.
Finally, note that MP3 is a compressed file format so minimize the number of times it is converted as this process reduces the quality of the audio.

Recommendation

While the iPhone Voice Memo app does not offer the features you can find on a DVR or in the Pure Audio app, it performed best in this analysis compared to the other methods tested in terms of recognition accuracy for DNS transcription. Using the iPhone for recording dictation is also cost-effective and convenient because you don't have to purchase, learn and maintain a separate device. If you understand iTunes well enough to customize the import settings and will mostly import recordings for transcription, then using the Voice Memo app is your best option. It's easy to create an MP3 version of the recording in iTunes, the file sizes are smaller and they seem to produce better accuracy than the Pure Audio recordings.
However, the bottom line is that there are several factors that affect DNS transcription recognition accuracy: recording device, recording settings, microphone, file format, DNS version, computer specifications, and DNS user profile training. Therefore, it's important for you to perform your own testing, and if necessary, adjust these factors until you achieve acceptable recognition accuracy.

Resources

PureAudio Live Recorder User Guide
Using PureAudio Live Recorder with Dragon NaturallySpeaking
Wikipedia article on TRS Connector

Monday, April 25, 2011

Removing Livescribe Smartpen Content

Removing pages that you are no longer interested in from a traditional notebook is easy - you simply tear them out and shred or recycle them. But there is much more involved in removing pages from a Livescribe notebook. The process depends on whether you’ve recorded audio along with the notes on those pages, whether you’ve transferred the content to your Livescribe Desktop (LD) application, and whether you plan to continue to use the notebook for study or note-taking purposes.
Note: This article pertains to version 2.3.4 of the Livescribe Echo Smartpen and Windows Livescribe Desktop application software.

Background

Before we discuss each of these scenarios, let’s review the terms, definitions and processes involved in using a Livescribe smartpen to take and refer to notes.
Making Notes
Taking notes with your smartpen involves writing or drawing in a Livescribe notebook which contains dot paper i.e. plain paper printed with a microdot pattern that allows the smartpen to recognize areas on the paper so it can associate your handwritten notes with recorded audio. Your notes may or may not have audio associated with them; it depends on whether you chose to record sound with the smartpen microphone prior to writing your notes. A session is a single, complete audio recording that may optionally have notes. So you have 3 note-taking options: notes + audio, notes only and audio only. Also, keep in mind that you can add notes to an audio recording at a later time.
Accessing Content In Notebook / Smartpen
Once you have made your notes, you can access them in several ways. If you recorded audio while writing notes, tap any note to start play back from the moment you wrote that note. If you didn’t record audio, you just read the notes as you would if they were written on normal paper. To play audio that doesn’t have notes associated with it, go to the Paper Replay application in the smartpen’s Main Menu, select Play Session and then select the audio session according to its date and time.
Accessing Content In Livescribe Desktop
You can also access your notes from Livescribe Desktop, the software that you install on your computer to allow you to transfer, store, search and replay notes from your computer. When you connect your smartpen to your computer, any new notes and audio you’ve added since the last connection are copied to your computer. In LD, you can view the notes in a particular notebook. Notes that have audio associated with them appear in “active ink”. Ink refers to the markings the smartpen makes on dot paper and active ink displays in green by default. Notes not associated with recorded audio display in “inactive ink” which is black by default.
So now you understand why you want to turn the smartpen on when recording notes only; it allows those notes to be captured by the camera, so they transfer to LD. The smartpen contains a ballpoint ink cartridge, allowing you to write on any type of paper but if the pen is turned off when you take notes in a Livescribe notebook, those notes won’t be captured for viewing in the Desktop application.
There are several ways you can listen to a recording in LD. If the session has notes associated with it, and you want to view the notes while listening to the recording, go to Pages View, select the appropriate notebook in the Navigation Pane, select the page containing the notes, and click the note at which you want to begin play back of the audio recorded as you wrote the note. The note must display with active ink. You’ll notice that notes recorded prior to this point in the session are green, while “future” notes are greyed-out and traced in green as the audio plays.
If the session doesn’t have notes or you don’t want to view the notes while listening to the recording, you can play the recording from the Audio View. Locate the recording by its date and time from the Session Name column. Double click the list item to start play back.

Remove Content From Smartpen

Removing notes from a Livescribe notebook requires you to consider whether the notes were made with the pen on, are associated with a recording, were transferred to your computer and are in a notebook that you still want to use.
If you wrote in your Livescribe notebook without turning the pen on, your writing wasn’t recorded and you would not have been able to record audio as well. So the pen does not contain a session nor does it even have a copy of your ink marking for transfer and viewing in LD. In this case, tearing that page out of the notebook is okay as long as all of the notes on both sides were made with the pen off.
Delete Session From Smartpen
Recall that a session is an audio recording that can optionally be associated with notes. The Paper Replay application in your smartpen allows you to delete an individual session or all sessions (select Delete Session or Delete All Sessions). This has several implications. First, once you delete a session from your smartpen, your paper notes will no longer play back the audio that was associated with that ink. Second, if you had transferred the session to LD, the session will still exist there. Third, if you delete the session before transferring it, the ink will still appear in LD, and due to a bug:
  • you'll receive the following error message after connecting your smartpen: "The following items failed to transfer: Audio Recordings"
  • new sessions won't transfer to Livescribe Desktop unless you archive your notebook (see Archive Notebook below for the implications of this).
I've notified Livescribe about this bug so hopefully it will be resolved in a software update in the near future. In the meantime, I recommend that you do not delete a session from the smartpen until AFTER transferring it to LD.
Delete Audio Session From Livescribe Desktop (and Smartpen)
Livescribe Desktop allows you to delete audio either from the LD application or from the smartpen.
To delete a recorded audio session from LD only, select Audio View, click the audio session to delete, right-click and select “Remove Audio from Livescribe Desktop…”. Deleting a session from Livescribe Desktop does not affect your smartpen.
To delete a recorded audio session from the smartpen only, ensure that the smartpen is connected to your computer. Select Audio View, click the audio session to delete, right-click and select Remove Audio from Smartpen…. This has the same effect as using the Delete Session command in the smartpen’s Paper Replay application.
As stated above, the implications of deleting an audio recording from the smartpen are that your paper notes will no longer play back the audio that was associated with them. If you delete audio from LD, you’ll no longer be able to play it on your computer.
If you wrote with the pen on, but made no recording, your notes will be transferred to LD where you can view them as inactive ink.
Delete Pages From Notebook
If you want to remove pages from a notebook in LD, you must first archive the notebook. Only an archived notebook can have pages deleted from it. You can also delete an entire archived notebook.
Archive Notebook
When you have finished using a notebook, you should archive it by connecting the smartpen to your computer and selecting the notebook in the Library tab of the Navigation Pane, and selecting Archive Notebook... from the File menu. Archiving a notebook moves it from the Library folder to the Archived Notebooks folder in the Navigation Pane. It also deletes your notes (ink data) and audio from the smartpen. This frees up storage space on your smartpen but also means that you won’t be able to use the smartpen to interact with the notebook. That is, when you tap notes in the physical archived notebook, they won’t play back any audio. However, you can still use LD to access the notes and audio in an archived notebook in the same manner as a non-archived one. Another important reason to archive a notebook is that it allows you to start a new notebook that shares the same series number (and thus the same microdot pattern).
If the archived notebook still has blank pages in it, you can still write in it. (But only do this if you want to treat the unused pages as part of a new notebook and don’t do it if you have started another notebook with the same series/name.) The Livescribe Knowledge Base says, "When you connect your smartpen to transfer your new notes, that notebook will re-appear in the active notebook section of Livescribe Desktop, and the additional pages you wrote on will appear as thumbnails when the active notebook is selected." You can archive the same notebook as many times as you wish.
For example, in a school setting, a teacher may want to pass a smartpen and partially used notebook on to another student. Of course the sessions on the smartpen should be deleted. Archiving the notebook does this. Then the teacher should remove the written pages from the notebook so another student won’t see work from the previous student. Finally, if the teacher doesn't want to keep an electronic copy of those written pages, s/he can delete them from the archived notebook.
Note that if you write on archived pages (i.e. pages that had content), those pages will appear in the active notebook with the new ink that you applied.
Deleting Archived Notebooks and Pages
If you want to permanently remove an archived notebook or some of its pages, use the Delete command as follows: Select an Archived Notebook in the Library tab. Select Delete Notebook... from the File menu to remove the entire notebook. Or, select one or more page to be deleted (use the Ctrl / Shift to multi-select) and select Delete Page(s) from the File menu. Note: If there is audio associated, at the prompt choose if you also want to delete the notebook’s or page’s audio. Also, if you have custom notebooks that link to the pages you're deleting, these pages will also be deleted from the custom notebooks.

References

Livescribe Smartpen User Guide
Livescribe Desktop for Windows User Guide
Livescribe Knowledge Base

Thursday, April 21, 2011

Contextual Spell Checker Evaluation

Conventional spell checkers fix spelling mistakes (i.e. non-word errors) by essentially performing a dictionary look-up of a word, and fix grammar errors by applying a set of grammatical rules. Contextual spell checkers are a new class of software that go beyond this; they use context (i.e. a number of words surrounding the word, most typically a sentence) to identify and correct misused words, also known as real word errors.
While researching the contextual spell check feature first introduced into the 2007 edition of Microsoft Word, I discovered an evaluation of four contextual spell checkers including Word 2007 (on Windows), MS Word 2008 (on MacOS X), MacOS X Spell and Grammar Checker, and After the Deadline.
The evaluation was conducted by Raphael Mudge, the developer of After the Deadline, an open-source grammar, style, and misused word checker that can be added to various applications as an extension e.g. Firefox web browser, OpenOffice.org word processor. You can read his 2010/4/9 blog posting about this at Measuring the Real Word Error Corrector. Mudge has made available the data set and programs to perform this evaluation on other such tools. The data consists of 673 sentences containing 834 errors (of which Mudge determined 97.8% are real word errors) that were collected from writers with dyslexia by Dr. Jennifer Pedler for her PhD thesis. Pedler annotated the errors along with the expected corrections and Mudge wrote a program to compare a corrected version of Dr. Pedler’s error corpus to the original corpus with errors. The program outputs 2 results as a percentage:
  • recall: how many errors were found and changed to something
  • precision: how often these changes were correct
Note: Mudge's program does not measure the number of words outside the annotated errors that were changed correctly or incorrectly. This is unfortunate because I noticed many false positives when using Ghotit i.e. correct words that were identified as errors.
So I used Pedler's data and Mudge's script to evaluate Word 2007 (to attempt to replicate Mudge's results), Word 2010 (to compare it to Word 2007), and two relatively new online contextual spell checkers: Ginger Software and Ghotit. When doing so, I accepted the first suggestion offered by each checker. For Ginger, I ran the test twice: I first approved the corrected sentence exactly as offered and then on a fresh copy of the error corpus, I explicitly selected the first suggestion for those corrections annotated with the ? icon as these usually remain unchanged. So selecting the first suggestion would show that Ginger did identify a potential error (improving recall) and may correct the error (potentially improving precision).
Spell CheckerRecall % (identifies word)Precision % (corrects identified word)
Ghotit72.277.9
Ginger Software v1.16.1 (select first suggestion for corrections labelled "?")56.688.8
Ginger Software v1.16.1 (approve default sentence)50.989.6
MS Word 200740.690.3
MS Word 201038.689.4
After the Deadline2888
My results show that Ghotit scores best at identifying errors with 72.2%. This is well ahead of the second highest recall value of 56.6% achieved by Ginger, when explicitly selecting the first suggestion; approving the corrected sentence exactly as offered identifies 50.9% of errors. The fourth highest recall is MS Word 2007 which identifies 40.6% of the errors.
But as I said above, I noticed that Ghotit generated a lot of false positives (the manner in which the Microsoft Word plugin interface is designed required me to select the first suggestion from a submenu for each highlighted error). This seems to have increased recall but notice that the precision score of 77.9% is significantly lower than the 88 - 90% that all of the other spell checkers achieved.
The error corpus does use UK English and I was able to specify that (or it was the default) for all the spell checkers I evaluated.
Note that the results for Ginger and Ghotit are subject to change as their algorithms and data sets are continuously being improved.