Sustainable Software for Digital Music and Audio Research

Lead Research Organisation: Queen Mary, University of London
Department Name: Sch of Electronic Eng & Computer Science

Abstract

The aim of this proposal is to provide a Service to support the development and use of software, data and metadata to enable high quality research in the thriving UK audio and music research community.Researchers in this area come from a wide range of backgrounds including: signal processing, electronics, computer science, music, information sciences, dance & performance, and data sonification, but have a common interest in the use of audio and music in their work. Through this project, we want to create a climate where researchers think beyond the research they are undertaking at the present, towards the impact that it could have on other researchers, but right from the start. Through helping researchers to strongly connecting their work with that of others, each will see that their work should be reusable: my research helps your research. In many of the fields within this community, researchers do not have the skills or desire to write their own code, or piece together other code, but benefit greatly from straight out of the box software. Even where they do write their own code, they work on different platforms and use a wide variety of batch and real-time environments (Matlab, Python, Max/MSP, SuperCollider, VST plugins). While there may be potentially useful software in other fields that could help them with their research, they are often not be aware of its existence, and it is often not available for the platform or environment they use.Other specific software- and data-related difficulties faced by researchers in this community include:* Software designed for legacy platforms (Sun SPARC, NeXTSTEP) and/or no longer maintained;* PhD students graduate or staff move: their web pages containing original software and/or data lost;* Code for some systems never released (not considered a priority for researchers, or code not considered ready );* Software and/or datasets in slightly different versions (due to error corrections or enhancements);* Copyright issues of datasets (e.g. audio of Beatles tracks cannot be placed on the web).The proposed Service to the UK music and audio research community will consist of several elements: * A software engineering resource to engineer cross-platform robust software from UK research prototypes; * A software tailoring programme to modify or adapt existing tools for use by other UK researchers; * A source repository with version control, to hold source code for software tools being developed in the community; * A dataset repository to hold and maintain key experimental datasets and metadata (e.g. human-generated labels); * Advice & training in the use of available software, and best practice in developing sustainable software. * Engagement and outreach to the research community and beyondAn important aspect of the Service will be planning for sustainability, so that the software developed during its lifetime, and the Service itself, can be sustained beyond the end of the funded period.

Planned Impact

Potential beneficiaries of this project outside of the academic research community include anyone who could benefit from access to high quality software based on the latest research in audio and music. Examples from various sectors are given below. Commercial private sector: * Commercial companies designing audio equipment, through easier access to new audio research; * Musicians and sound artists, through their ability to use new research methods and processes for new creative outputs. * Computer games companies, through better access and use audio and music research outputs for improved game music and audio, or for new types of audio or music-based computer games * Hearing aid and cochlear implant manufacturers, through access to new research applicable to hearing improvement * Audio archiving companies, through access to the latest algorithms and methods for music information retrieval * Television and radio companies, through ability to use latest research in the design of new technology-based programmes Policy-makers and others in government and government agencies: * Research funders, through improved flow of research outputs to research users and other researchers, and thus more effective use of research funds; and increased across-the-board impact of research in audio and digital music * Police and security services, through access to algorithms for analysing and separating audio signals Public sector, third sector and others: * Healthcare workers who work with music and audio, such as music therapists, through new music and audio visualization tools * Museums, through software to measure and visualise acoustic properties of objects * Computing museums and digital libraries, through sharing of experiences with emulation and virtualization of software * Libraries, through improved open access to research results (including software) for the benefit of their users; and access to improved ways to catalogue and search audio and music information based on the latest research * Standards organizations, through access to high quality examples of research methods on which to based forthcoming standards * Science promotion organizations, through availability of high-quality usable research software attractive to people who may be interested in science Wider public: * People interested in exploring music or other audio recordings at home, school, college or university, using the latest research, either for educational or general interest purposes * Teachers in schools, colleges or universities who want to use research-generated software for teaching audio or music. * People such as programming hobbyists, who wish to participate in open-source research software development and maintenance, through contribution to bug fixing, upgrading, etc. of software development activities promoted by the Service. These may have an interest in audio and music, but that would not be a precursor to involvement. * Audiences of creative output involving audio and music, through availability of new creative outputs or technology facilitated by audio and digital music research Researchers employed on the project, and others undertaking training or other interaction: * Improved skills in research software development and research methodologies, which may be transferred into e.g. the commercial private sector on completion of the project

Publications


10 25 50
Bittner, R. (2014) MedleyDB: A Multitrack Dataset for Annotation-Intensive MIR Research in 15th International Society for Music Information Retrieval Conference
Ewert S (2014) Score-Informed Source Separation for Musical Audio Recordings: An overview in IEEE Signal Processing Magazine
Figueira L.A. (2013) Software techniques for good practice in audio and music research in 134th Audio Engineering Society Convention 2013
Stowell D (2015) Detection and Classification of Acoustic Scenes and Events in IEEE Transactions on Multimedia
Wilson G (2014) Best practices for scientific computing. in PLoS biology
 
Description This project produced several key outcomes, including:
* We were instrumental in bringing Software Carpentry to the UK, as a programme and community for training researchers to develop reliable software. This is relevant to academics outside our own field and of significance to UK research in general;
* We developed the SoundSoftware code site, which continues to be a centre for audio and music research software
* We developed and maintained numerous audio analysis plugins for wider use, including submitting them for the annual MIREX evaluation, managing builds and distribution, and providing language interfaces for using them in Java and Python as well as C++
* We developed the broadly useful application Tony for singing-pitch tracking and analysis
Exploitation Route Researchers in the UK and internationally who use audio and music research can benefit from the research software produced by the project, and the code development approaches that we have been promoting.
The commercial private sector, public sector and wider public can also benefit from the availability of digital music tools.
Sectors Creative Economy,Digital/Communication/Information Technologies (including Software),Education
URL http://www.soundsoftware.ac.uk/
 
Description The SoundSoftware code site has assisted with the development of a range of software such as Sonic Visualiser. Sonic Visualiser is available for Linux, Mac OS/X and Windows, and has been downloaded over 330,000 times. In a survey of over 800 users (11 Oct 2010 to 25 Apr 2013), 49% reported that they were non-academic users, using it as a professional (9%) or for personal use (40%). Usage is truly international, with users from 66 countries including the USA (32%), UK (25%), Germany (10%) and France (8%).
Sector Creative Economy,Digital/Communication/Information Technologies (including Software),Leisure Activities, including Sports, Recreation and Tourism
Impact Types Cultural,Economic
 
Description FP7 Marie Curie Initial Training Network
Amount € 2,800,000 (EUR)
Funding ID 607290 
Organisation European Commission 
Sector Public
Country European Union (EU)
Start 10/2014 
End 09/2018
 
Description H2020 Marie Sklodowska-Curie Action (MSCA) Innovative Training Network
Amount € 3,800,000 (EUR)
Funding ID 642685 
Organisation European Commission 
Sector Public
Country European Union (EU)
Start 01/2015 
End 12/2018
 
Description H2020-ICT-2015 Audio Commons
Amount € 2,980,000 (EUR)
Funding ID 688382 
Organisation European Commission 
Sector Public
Country European Union (EU)
Start 02/2016 
End 01/2019
 
Description Making Sense of Sounds
Amount £1,275,401 (GBP)
Funding ID EP/N014111/1 
Organisation Engineering and Physical Sciences Research Council (EPSRC) 
Sector Academic/University
Country United Kingdom of Great Britain & Northern Ireland (UK)
Start 01/2016 
End 12/2018
 
Title EasyMercurial 
Description EasyMercurial is a simple user interface for the Mercurial distributed version control system. It's free, open source, and cross-platform. 
Type Of Technology Software 
Year Produced 2011 
Open Source License? Yes  
Impact Supported learning of software development techniques and easy access to version control repositories, particularly those hosted at the SoundSoftware code site. This application has seen some tens of thousands of downloads and has been positively received by users we have engaged with. 
URL http://easyhg.org/
 
Title SMALLbox: Sparse Representation and Dictionary Learning evaluation toolbox 
Description SMALLbox is a foundational framework for processing signals using adaptive sparse structured representations. The main aim of SMALLbox is to become a test ground for exploration of new provably good methods to obtain inherently data-driven sparse models, which are able to cope with large-scale and complicated data. The main focus of research in the area of sparse representations is in deveoping reliable algorithms with provable performance and bounded complexity. Yet, such approaches are simply inapplicable in many scenarios for which no suitable sparse model is known. Moreover, the success of sparse models heavily depends on the choice of a "dictionary" to reflect the natural structures of a class of data. Inferring a dictionary from training data is a key to the extension of sparse models for new exotic types of data. SMALLbox was developed under an EU grant, with development support and facilitation from the SoundSoftware project (EPSRC). 
Type Of Technology Software 
Year Produced 2012 
Open Source License? Yes  
Impact Through this work, the SoundSoftware project supported the publication and collection in a reusable form of substantial research work in sparse representation techniques. 
URL https://code.soundsoftware.ac.uk/projects/smallbox
 
Title Tony: a tool for melody annotation 
Description Tony is a software program for high quality scientific pitch and note annotation in three steps: automatic visualisation/sonification, easy correction, and export. First, Tony automatically analyses the audio to visualise pitch tracks and notes from monophonic recordings . As a Tony user, you can play back pitches and notes alongside the recording, which makes it easy to spot the inevitable extraction errors. Then you can use several tools that make correction of such errors a breeze, including alternative pitch track selection, octave shift, and easy note split, merge and deletion. Finally, you can export pitch track and note track in .csv (comma-separated values) format, or simply save a session file to continue annotating another time. 
Type Of Technology Software 
Year Produced 2014 
Open Source License? Yes  
Impact Has supported development and publication of an annotated multi-track test audio dataset, and is likely to be used in future work of this kind. 
URL https://code.soundsoftware.ac.uk/projects/tony
 
Description Annual SoundSoftware Workshop 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Postgraduate students
Results and Impact An annual series of one-day workshops on Software and Data for Audio and Music Research, held on Queen Mary University of London.
Year(s) Of Engagement Activity 2012,2013,2014
URL http://www.soundsoftware.ac.uk/soundsoftware2014
 
Description Software Carpentry Autumn School 2010 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Postgraduate students
Results and Impact 22 PhD students and researchers attended a one-week Autumn School (London, UK, 1-5 November 2010) to learn about software development skills required to build reliable research software quickly and with a minimum of effort, and so maximize the impact of their research.
Year(s) Of Engagement Activity 2010
URL http://www.soundsoftware.ac.uk/autumnschool2010review