Comments on: HTML5 Audio Read-Along

By: Harry Pannu

Harry Pannu — Fri, 09 Dec 2011 18:07:07 +0000

Weston,

I have already skimmed through most of your posts as I was looking for a solution for myself. I just added a message of my own about how I got MP3 working with Sphinx for everyone’s good.

See https://sourceforge.net/projects/cmusphinx/forums/forum/382337/topic/4787012?message=10898990

By: Weston Ruter

Weston Ruter — Fri, 09 Dec 2011 17:57:41 +0000

@Harry:
The cue points are not reliable enough yet for me. Maybe things have improved since last time I tried, but compare the results of aligning John 1 and John 3:
https://github.com/westonruter/esv-text-audio-aligner/blob/master/reports/John.1
https://github.com/westonruter/esv-text-audio-aligner/blob/master/reports/John.3

John 3 gets extremely misaligned. See more information in the last post on the Sphinx forums: http://sourceforge.net/projects/cmusphinx/forums/forum/382337/topic/4503550

By: Harry Pannu

Harry Pannu — Fri, 09 Dec 2011 17:41:27 +0000

Did you ever need to tweak the cue points generated by Sphinx? I am working on a web app that is suppose to auto generate cue points for a given script and audio file on the click of a button in the web interface. On top it should have an “easy-to-use” interface for editors to manually correct any errors.

After struggling for several days, at the end, it turned out to be pretty simple to get the MP3 decoding working with Sphinx. I would be happy to provide you the specifics if interested in trying it. It will be comforting to know that it works for you as well.

By: Weston Ruter

Weston Ruter — Fri, 09 Dec 2011 16:13:29 +0000

@Harry:
In my Sphinx4 attempt, I did indeed convert from MP3 to WAV first. I wasn’t able to get Sphinx to work with MP3. However, my build script automatically converts the MP3s to WAV format before passing into Sphinx, so it’s no extra labor. See https://github.com/westonruter/esv-text-audio-aligner/blob/master/align.py#L120

By: Harry Pannu

Harry Pannu — Fri, 09 Dec 2011 16:09:43 +0000

Weston,

In your attempt to use Sphinx4, were you able to get word cue points directly from MP3 or did you convert MP3 to WAV format first? I am trying to make Sphinx4 use MP3 directly to save the labor of WAV conversion. I am getting good results. I would like to exchange information with you hoping it will benefit both of us. Looking forward to hear from you.

Thanks,
Harry Pannu

By: Weston Ruter

Weston Ruter — Thu, 01 Sep 2011 04:28:52 +0000

The CMU Sphinx project has improved their code to work with aligning long audio with text. I’ve created a new project which uses it to align the ESV text with the audio: https://github.com/westonruter/esv-text-audio-aligner

By: Weston Ruter

Weston Ruter — Fri, 20 May 2011 19:21:02 +0000

I filed a feature request for Google Chrome to extend their experimental TTS API to facilitate read-along apps: http://code.google.com/p/chromium/issues/detail?id=83404

I was excited to learn about Chrome’s experimental TTS API. An application that I am very keen to develop is a read-along, where the page highlights the text corresponding with the words as they are being spoken. To do this, an API would need to be exposed for determining when each word is spoken. Currently, the Chrome TTS API only has events for onSpeak and onStop. To do a read-along, however, something like a “onSayWord” or “onUtter” event would be needed, where the event handler would be passed an Event object indicating the actual word being spoken and maybe a word index (the original text passed in would need to get broken up into individual utterances). It would also be useful to be able to seek to a specific time position within TTS audio given a word index (or time index)—this would allow you to navigate the audio via selecting words in the text. See the URL example provided for the kind of application I’d love to build utilizing such extensions to the TTS API.

By: Weston Ruter

Weston Ruter — Sat, 14 May 2011 22:47:59 +0000

@Dmitry:
I think I used Audacity and a Google Spreadsheet to manually obtain the timings. I literally stepped-through the audio second-by-second finding the start and end time for each word in the audio and then added them to the spreadsheet. It took a few hours just for this passage.

I know it is possible to automatically obtain the time indices for the words in audio given a transcript. The closest I got to doing this myself was utilizing the CMU Sphinx project which includes the ability to align text and audio. I had some success, but hit a roadblock. You can read all about my efforts and see the code on a thread on the Sphinx forum: http://sourceforge.net/projects/cmusphinx/forums/forum/382337/topic/4503550

If you’re able to tweak my code to get the desired results, please let me know!

By: Dmitry

Dmitry — Sat, 14 May 2011 08:03:37 +0000

Hi, Weston,
You’ve mentioned that you manually traversed the audio and set all the timings. How exactly did you do that? What software did you use?

So, we have both sound file read by a human (not TTS) and the transcript. We want to do the slicing (that process of recovering word timings) automatically. Did you think if that is possible?
I was trying to do something similar and tried to employ Free-TTS… But I did manage to do smth usefull.
I’m doing some study-English software and that exciting read-along feature is what I’m really looking for. Can you advise?

By: Gerardo Capiel

Gerardo Capiel — Fri, 28 May 2010 22:32:07 +0000

You should check out a JavaScript based TTS implementation at http://scotland.proximity.on.ca/dxr/tmp/audio/tts/ . It would be interesting to link that work with your work. Keep me posted, if you do anything with this.