Dynamic dialects: integrating articulatory video to reveal the complexity of speech

Lead Research Organisation: University of Glasgow
Department Name: School of Critical Studies

Abstract

Our motivation for creating this resource was to bring Phonetics (the study of speech sounds) into the 21st century by making use of imaging technologies (MRI and ultrasound tongue imaging) and digital processing to allow people to see the hidden movements of the tongue and other articulatory organs inside the vocal tract.

We will create an online teaching resource called Dynamic Dialects that will provide easy access to multimedia material (videos and audio) showing how the vocal organs move during real speech. The website will be designed to aid phonetic training, language teaching and learning and speech therapy and will improve people's understanding of how speech sounds are produced.

Knowing how to make a sound with your tongue and other vocal organs is important when you are learning a foreign language with unfamiliar sounds, learning how to produce a sound in your own language that you have difficulty with, or if you need to teach other people how to make speech sounds, e.g. as a phonetician, language teacher or speech therapist, and also if you want to carry out research into how speech sounds change over time. Without being able to see what is going on in the vocal tract, it can be a struggle to understand what to do to achieve the correct sound quality.

Traditional phonetics teaching relies on visual cues, such as lip and jaw movement; diagrams of the vocal tract; focussing on and trying to interpret the movements of your own vocal organs during speech and, of course, listening to and trying to interpret the acoustic signal produced during speech. Analysis of speech that is based mainly on listening to other people's speech or feeling your own articulatory movement can lead to misunderstandings of how speech sounds are produced. Not everyone has the same strategy for producing sounds. We find, through articulatory-based research, that speakers often have very different strategies for producing a speech sound than what might be expected.

Ultrasound tongue imaging video, Magnetic Resonance Imaging video, lip movement video and animations (based on data gathered through all of these technologies) can offer a new insight into exactly how speech sounds are produced. Speech production can be taught with visual stimuli showing, or based on, real speech, to aid learning. Articulatory strategies can be copied and practised using a visual, as well as auditory, models.

The Dynamic Dialects articulatory teaching resource will allow contain a collection of pronunciations of words from speakers of different varieties of English, which will be an invaluable resource for the study of linguistic change in the future, preserving not only how speech sounded, but how it was produced.

Planned Impact

The principal beneficiaries of this resource will be a wide range of students an professionals with interests in Language study, for example students and teachers of phonetics; English as a Second Language (ESL) and English ad a Foreign Language (EFL) teachers; speech therapists; drama and voice coaches. This resource will provide these users with a range of modelled and natural speech and is the first resource of its kind. Users will be able to view the articulators in action, rather than relying on purely auditory stimuli, static diagrams or descriptions of speech. We expect this resource to have a significant impact on the accuracy of speech and language teaching as articulatory data can help overturn erroneous views of how speech is produced. The resource will be available freely online to users around the world.

The resource is also important in preserving a snapshot of language, from both an acoustic and articulatory perspective, as it is today. Future researchers will be able to use this resource to study sound change over time, by comparing how speech sounds were produced today with how they develop in the future, e.g. the study of consonantal vocalisation where consonants change into vowels over time.

The raw data in this resource will be preserved for student and postgraduate research projects into language articulation and will be made available to users at other research institutes with the appropriate permissions and ethical approval.
 
Description Our project developed two new freely-available online resources for the study of the movements of the vocal organs during speech. We used medical imaging technology and other techniques to capture (hidden) vocal organ movement, making use of headset-mounted lip cameras, Ultrasound Tongue Imaging (UTI) and Magnetic Resonance Imaging (MRI). Accurate 2-D animations of the human vocal tract, based on MRI recordings of real speech, were created and made available to users of our online resource.
During the project we developed:
(1) an MRI recording protocol with an improved frame rate.
(2) a set of 100 MRI videos of the moving vocal tract during speech.
(3) a method of integrating ultrasound tongue imaging video with lip-movement video.
(4) a set of 1,500 integrated videos of tongue and lip movement from 53 native speakers of English from 48 different localities in 16 different countries around the world
(5) a method for turning MRI video recordings into 2-D animation, improving on the deficiencies of the MRI recordings: low frame rate, blurring of articulatory surfaces, missing passive articulators.
(6) a set of empirically-based articulatory 2-D animations of speech sounds.
(7) methods and protocols to process large numbers of short videos (around 7,000) in order to make them optimally accessible to an online audience.
(8) a set of written materials that introduce users to vocal tract imaging techniques such as UTI and MRI and describe the recording processes.
Exploitation Route Our online resources have received interest from the following users: phonetics teachers and students; speech and language therapists; accent coaches; English Language learners. The resources give these users access to vocal organ movement that has previously been hidden. Members of each of these groups have expressed a willingness to work with us on future projects tailored to their specific needs.
Sectors Creative Economy,Digital/Communication/Information Technologies (including Software),Education,Healthcare
URL http://www.dynamicdialects.ac.uk
 
Description Our websites became available on 1 May 2015, and were launched with a press release on 28 May 2015. We have been contacted by speech and language therapists, accent coaches, ESL teachers and phonetics teachers and students, expressing their interest in the resources. We have also conducted several public engagement activities (Meet the Expert, Glasgow Science Centre; Seeing Speech, Glasgow Science Festival), as well as specific workshops for speech and language therapists. Our resources are used to teach and learn about speech articulation (i.e. how speech is produced by the moving vocal organs). The movements of many of the vocal organs are hidden within the head and throat, but the medical imaging techniques that we used reveals this hidden movement. Our resource allows the easy comparison of accents of English across the World, either through an interactive map or through a chart. Accent coaches can select a particular word and compare its pronunciation across accents, hearing not only how it sounds, but seeing also the movement of the tongue and the speaker's lips.
First Year Of Impact 2015
Sector Digital/Communication/Information Technologies (including Software),Education,Healthcare
Impact Types Cultural,Societal
 
Title 3D MRI head scan database 
Description 3D head scans obtained at the Clinical Research Imaging Centre (CRIC) at Edinburgh Royal Infirmary as part of the MRI data-collection phase of Dynamic Dialects. 
Type Of Material Database/Collection of data 
Year Produced 2014 
Provided To Others? Yes  
Impact 3D head scans have been provided to Pertti Palo and colleagues at the Helsinki University of Technology to allow modelling of the human vocal tract including the production of 3D printed physical models. 
 
Title ArticulatoryIPA Youtube channel 
Description A Youtube channel that predates this project, but has been augmented with material collected as part of this project. The channel contains MRI, ultrasound and lip-camera video of speech produced by speakers with different accents. The channel has 199 subscribers many of whom are students of Linguistics and Phonetics, English Language learners or those interested in acting. To date we have had 128,000 views of material on the channel. Top locations for users are: UK, US, Canada, Germany, Ireland, Australia, Poland and Brazil. 
Type Of Material Database/Collection of data 
Provided To Others? No  
Impact The ArticulatoryIPA channel allows us to reach a wider range of users than our Seeing Speech website www.seeingspeech.ac.uk which is designed to be used by those studying speech. The resource provides a useful tool for self-study for those studying Linguistics, those learning English pronunciation (including regional variation) and those learning accent variants for performance (e.g. acting). Our videos show how the tongue moves within the mouth as well as lip movement for a wide variety of accents of English. 01/06/2015 - As of this date, 238 subscribers to the ArticulatoryIPA youtube channel and 158, 808 views. February 2015 Comment sent via Youtube from L.A.-based dialect coach Pamela Vanderway "Just want to say thank you for these videos. They are WONDERFUL." 18/03/2015 - Skype call with Pamela Vanderway to talk about the Dynamic Dialects resource and how a future resource could be further tailored to the needs of dialect coaches. 
URL https://www.youtube.com/channel/UCuOKJqD00W2EiC3DHmOuu0g
 
Title Dynamic Dialects 
Description Dynamic Dialects is a Web resource that allows users to listen to and watch the tongue and lip movements of 53 native speakers of English from 48 different localities in 16 different countries around the world: the UK, Ireland, USA, Canada, India, New Zealand, Australia, Singapore, Sri Lanka, Hong Kong, Malaysia, Zimbabwe, Nigeria, Tanzania, Trinidad and the Philippines. Tongue movements are imaged using ultrasound. The Web site hosts around 1,500 videos including sets of 27 reference words that allow users to compare the vowel systems of different accents of English and a brief sample of spontaneous conversational speech from each speaker. This project is a collaboration between the University of Glasgow, Queen Margaret University, University College London and Napier University. 
Type Of Material Database/Collection of data 
Year Produced 2015 
Provided To Others? Yes  
Impact 28/05/2015 Prof. Jane Stuart-Smith (P.I.) interview on Radio Scotland on Dynamic Dialects 28/05/2015 Email from Glasgow-based speech therapist Jane O'Brien, enquiring about resource and about project launch event at the Glasgow Science Centre (to be held on 30/05/2015) 28/05/2015 Prof. Jim Scobbie (Co-I) interview on BBC radio 4's PM programme on Dynamic Dialects 01/06/2015 Contacted by Paul Meier of Paul Meier Dialect services, who congratulated us on our resource and said that he had added a link to it on his website. 
URL http://www.dynamicdialects.ac.uk
 
Title Seeing Speech 
Description Seeing Speech is a Phonetics teaching resource containing 300 moving 2-D videos showing the inside of the vocal tract using medical imaging techniques. The videos were obtained using ultrasound tongue image and magnetic resonance imaging technology. In addition, we have created 60 2-D vocal tract animations, based on MRI recordings, with moving tongue, lips, larynx, epiglottis and velum. The website is a resource for both students and teachers of linguistics and speech therapy. Seeing Speech is the product of a collaboration between the University of Glasgow, Queen Margaret University, University College London, Napier University, Edinburgh University, Strathclyde University and the University of Aberdeen. 
Type Of Material Database/Collection of data 
Year Produced 2012 
Provided To Others? Yes  
Impact SLTs attending a clinical workshop at Queen Margaret University, Edinburgh on 23/05/2015 said in evaluation forms handed out at the end of a talk on Seeing Speech that they felt that they could have benefitted from having the Seeing Speech resource during their training. 
URL http://www.seeingspeech.ac.uk
 
Description Block Seminar on sound change, Hannover 2014 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Undergraduate students
Results and Impact Invited intensive Block Seminar (entire seminar course, 28 hours, taught over 4 days), on sound change. Included a segment on speech production and sound change, trialling the improved Seeing Speech website (was being augmented at that time), and highlighting findings from the previous two ESRC grants on the role of socially stratified articulatory variation in sound change.
Year(s) Of Engagement Activity 2014
 
Description Glasgow Science Centre "meet the expert" event to launch Seeing Speech and Dynamic Dialects 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach Regional
Primary Audience Public/other audiences
Results and Impact To be completed by Jane and Jim
Year(s) Of Engagement Activity 2015
URL http://www.glasgowsciencecentre.org/Search.html?ordering=&searchphrase=all&searchword=seeing+speech
 
Description Invited talk at the Phonetics Teaching and Learning Conference 2015, University College London, 6/08/2015. 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact Talk was well received and directed more users towards the Web resources created during the project.
Year(s) Of Engagement Activity 2015
URL http://www.dynamicdialects.ac.uk
 
Description Jane Stuart-Smith Radio Scotland interview on Dynamic Dialects 28/05/2015 
Form Of Engagement Activity A press release, press conference or response to a media enquiry/interview
Part Of Official Scheme? No
Geographic Reach Regional
Primary Audience Media (as a channel to the public)
Results and Impact 28/05/2015 Email from Glasgow-based speech therapist Jane O'Brien, enquiring about resource and about project launch event at the Glasgow Science Centre (to be held on 30/05/2015)


28/05/2015 Email from Glasgow-based speech therapist Jane O'Brien, enquiring about resource and about project launch event at the Glasgow Science Centre (to be held on 30/05/2015). We asked if she would be available to be contacted in the future about designing a potential clinically-focussed resource.
Year(s) Of Engagement Activity 2015
 
Description Jim Scobbie interview on Radio 4's PM programme 28/05/2015 
Form Of Engagement Activity A press release, press conference or response to a media enquiry/interview
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Media (as a channel to the public)
Results and Impact Prof. Scobbie contacted by an English teacher working in Brazil about his interest in the project 28/05/2015.

Resource noticed by J.C. Wells who alerted the members of VASTA (Voice and Speech Trainers Association) about the resource.

01/06/2015 Eleanor Lawson (R.A.) contacted by Contacted by Paul Meier of Paul Meier Dialect services, who congratulated us on our resource and said that he had added a link to it on his website. Further discussion between Paul Meier and Eleanor Lawson and sharing of resources.
Year(s) Of Engagement Activity 2015
 
Description Plenary and workshop at Sociolinguistics Summer School 6, Trinity College Dublin, 8/08/2015 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact A plenary on the use of ultrasound in sociolinguistic study. A practical workshop on best practice use of ultrasound in linguistic study with ultrasound demonstration.
Year(s) Of Engagement Activity 2015
 
Description Postgraduate Training (São Paulo) 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Postgraduate students
Results and Impact one week of masters / doctoral seminars at Pontifícia Universidade Católica de São Paulo, Brazil, primarily to a group of Speech Therapists and Therapy-oriented researchers.

Additional request for CAPES-funded international post-grad visit to QMU for 2015.
Year(s) Of Engagement Activity 2014
 
Description Press release for related project Sounds of the City (PI Prof Stuart-Smith) 
Form Of Engagement Activity A press release, press conference or response to a media enquiry/interview
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Media (as a channel to the public)
Results and Impact Press release on the related Sounds of the City project (PI Prof. Stuart-Smith) was picked up by the Mail online, who reported on the Dynamic Dialects resource in their article 16/11/2015

Public will have been reminded of this resource.
Year(s) Of Engagement Activity 2015
URL http://www.dailymail.co.uk/sciencetech/article-3320379/I-cannae-believe-Scottish-accents-flourishing...
 
Description SLT UTI Study Day (06/14) 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach Regional
Primary Audience Professional Practitioners
Results and Impact Visualising Articulation: A Study Day for Speech & Language Therapists on Visual Biofeedback and Articulatory Models for treating Speech Sound Disorders. (5th June 2014), Queen Margaret University, Edinburgh.
Attended by NHS Speech and Language Therapists who wanted to find out about research findings on ultrasound tongue imaging fundamentals, what it reveals about typical populations and development, and how it can be used for biofeedback therapy.
Input from researchers on multiple projects for a full day workshop for local SLT managers and therapists, upper undergraduate and masters-level students.
a. EPSRC ULTRAX (Renals PI)
a. ESRC Mimicry (Scobbie PI)
c. Carnegie Trust Seeing Speech (Stuart-Smith PI)
d. AHRC Dynamic Dialects (Stuart-Smith PI)
(An associated advanced training workshop in UTI was also run at the same time - it is recorded as a separate event.)

Increased awareness of
a. the importance and use of visual biofeedback methods in speech therapy, protocols used in current successful therapy at QMU and elsewhere,
b. datasets of and research into typical production of a range of sounds, including social variation in adulthood and typical childhood articulation patterns
Year(s) Of Engagement Activity 2014
 
Description Seeing Speech event at Glasgow Science Festival (14/06/15) 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? Yes
Geographic Reach Regional
Primary Audience Public/other audiences
Results and Impact We ran two workshops called Seeing Speech. Each one attracted around 20 participants of all ages, from 2 to mid-50s. We gave a short presentation explaining speech production and ultrasound tongue imaging, and then participants volunteered to have their tongue imaged in order to carry out a collaborative quiz task. Everyone enjoyed the presentation, and especially the ultrasound imaging.

We had a range of participants, from families with young children to early teenaged schoolboys (who loved the ultrasound headset) to university students and couples interested in science. The evaluation sheets rated the workshops as 'excellent' or 'very good', saying that they were 'very interesting'. Two afterschool teachers had brought the group of teenage boys to the university, and were very enthusiastic about the impact of being in a phonetics lab and taking part in 'real' research. All the boys said that they would like to come to university and work in a lab like ours.
Year(s) Of Engagement Activity 2014,2015
URL https://glasgowsciencefestival.wordpress.com/2015/05/07/seeing-speech/
 
Description Speech and Language Therapy Paediatric Study Day, 9th June 2015, QMU 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach Regional
Primary Audience Professional Practitioners
Results and Impact Positive comments by SLTs on the Seeing Speech and Dynamic Dialects resources.

After talk, SLTs commented on the usefulness of the resource.
Year(s) Of Engagement Activity 2015
 
Description Speech and Language therapy workshop 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach Regional
Primary Audience Professional Practitioners
Results and Impact Comments on feedback sheets handed out to SLTs suggested that they felt the resource would have been very helpful to them during their training.

One attendee asked if she could learn more about the resource later.
Attendees were asked for their input on creating a clinically-focussed resource similar to Seeing Speech and Dynamic Dialects
Year(s) Of Engagement Activity 2015
 
Description Speech articulation event for the Glasgow Science Festival at the University of Glasgow followed by a hands-on UTI demonstration 
Form Of Engagement Activity Participation in an open day or visit at my research institution
Part Of Official Scheme? No
Geographic Reach Local
Primary Audience Public/other audiences
Results and Impact An hour long slot at the Glasgow Science Festival consisting of a 40 minute presentation about speech production with video of vocal fold vibration, MRI video of vocal organ movement and ultrasound tongue imaging (UTI) video of tongue movement, followed by 20 minutes of hands-on demonstration of UTI to the talk attendees with two ultrasound machine.

The event was rated as excellent by all attendees, who also reported that they felt they had learned things they didn't know about speech production in general and about their own speech.
Year(s) Of Engagement Activity 2014
 
Description Speech articulation event for the Glasgow Science Festival at the University of Glasgow followed by a hands-on UTI demonstration 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach Local
Primary Audience Public/other audiences
Results and Impact An hour long slot at the Glasgow Science Festival consisting of a 40 minute presentation about speech production with video of vocal fold vibration, MRI video of vocal organ movement and ultrasound tongue imaging (UTI) video of tongue movement, followed by 20 minutes of hands-on demonstration of UTI to the talk attendees with two ultrasound machine.

The event was rated as excellent on feedback forms by all attendees, who also reported that they felt they had learned things they didn't know about speech production in general and about their own speech.
Year(s) Of Engagement Activity 2014
URL http://www.glasgowsciencefestival.org.uk/
 
Description Student training, LSA Summer Institute, Chicago 2015 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Undergraduate students
Results and Impact Jane Stuart-Smith was invited to give a course on the influence of the media on speech at the LSA Summer Institute, Chicago, July 2015. This had two segments which used materials from the Dynamic Dialects project (Seeing Speech), and reported results from the two ESRC projects on UTI and socially-stratified variation and change.
Year(s) Of Engagement Activity 2015
 
Description UTI researcher training (06/14) 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Other academic audiences (collaborators, peers etc.)
Results and Impact Advanced and Intermediate Training in UTI data analysis for post-doc, PhD and active researchers from around the UK, June 2014.
(See also a related Speech and Language Therapist Training event).

Academic colleagues requested further, regular (eg annual) events. Small grant application written and submitted to British Academy with one of the participants on the basis of their attendance (unsuccessful).
Year(s) Of Engagement Activity 2014
 
Description Ultrasound Tongue Imaging hands-on demo at the Science in the Community events 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach Local
Primary Audience Public/other audiences
Results and Impact Over 60 adults and children (of all ages) took part in hands-on ultrasound tongue imaging (UTI) demos at Anniesland Public Library and Drumchapel Community Centre as part of a community science even centring on the Queen's Baton Relay.

It is difficult to quantify the impact of this event; however, some children were very interested in the ultrasound machine and returned again and again to look at their tongue moving, experimenting with producing different speech sounds. At one venue, a young boy took over the demonstration and began showing people how to apply gel to the probe, where to place it under their chin and described their speech movements to them. I believe that the demonstration stimulated an interest in speech articulation for the children who returned again and again to the demo. As I said, it is very difficult to quantify what the long-term impact of this interest will be.
Year(s) Of Engagement Activity 2014
 
Description Using ultrasound tongue imaging to study variation in the GOOSE vowel across accents of English 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Other academic audiences (collaborators, peers etc.)
Results and Impact This paper was an invited contribution to a special session of the Methods in Dialectology XV conference in Groningen, Holland. The session was entitled: Instrumental articulatory phonetics and dialectological fieldwork. Strange bedfellows? The paper set out the reasons for studying vowel variation from an articulatory perspective. It described some methodological innovations to improve inter-speaker comparison and a method for normalising data. It also identified issues and drawbacks relating to ultrasound use in the study of vowels that are still to be resolved.

The paper was reviewed by a discussant after it was delivered and was well-received in general. The paper led to talks with a postgraduate student who wanted to use ultrasound tongue imaging in her research. We set a date for a pilot recording session.
Year(s) Of Engagement Activity 2014
URL http://methodsxv.webhosting.rug.nl/
 
Description project demo at Digging into Data conference event, January 2016 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Other audiences
Results and Impact Invited demostration of Dynamic Dialects web resources (Seeing Speech and Dynamic Dialects) to members of the Digging into Data showcase event, held at the Glasgow School of Art, 29 January 2016, as part of the Digging into Data conference for award holders, council administrators, and senior international research council leaders.
Year(s) Of Engagement Activity 2016
 
Description ultrasound demonstration at Applicants Day 31/03/15 
Form Of Engagement Activity Participation in an open day or visit at my research institution
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Public/other audiences
Results and Impact We discussed and demonstrated speech production using ultrasound tongue imaging and through looking at Seeing Speech website, and discussing recent findings from the ESRC Seeing the Links in the Speaker-Hearer Chain project, and the Looking Variation in the Mouth project. Applicants were excited and interested and asked many questions.

Applicants and their parents enjoyed their visit to the lab as a result of these hands-on activities.
Year(s) Of Engagement Activity 2015