An affordable stereoscopic camera array system for capturing real-time 3D responses to vegetation dense environments

Lead Research Organisation: University of Edinburgh
Department Name: Sch of Biological Sciences

Abstract

To feed our growing global population, the productivity of staple crops will require an increase in yields of ca. 60% by 2050. Realisation of this significant challenge is currently hampered by insufficient capacity of the plant science and Agri-Tech communities to analyse existing plant genetic resources for their interaction with the environment. Plant phenotyping is an emerging science that links genomics with plant ecophysiology and agronomy. The collection of architectural traits of a plant throughout the life cycle (the phenotype) is a result of dynamic interactions between the genetic background (the genotype) and the physical world in which the plant develops (the environment). These interactions determine plant performance and productivity measured as accumulated biomass and commercial yield, and resource use efficiency. A critical parameter of a plant's phenotype is the shape in three dimensions - the plant architecture - which reflects the adaption of a plant to environmental conditions, such as light quantity and quality, and temperature. We will bring together plant scientists, and image capture and machine vision experts to develop an affordable and user-friendly system for progressing on one of the most important issues in biology, our capacity to enhance the productivity of plants. We will combine a robust photometric stereo approach with software using optimised learning algorithms that will produce standardised outputs that can be readily integrated across scales with other biological data sets.

Technical Summary

There is an urgent need for new technologies that allow us to monitor and predict the impact of abiotic and biotic stresses on seasonal plant growth. Such tools could provide quantitative data on crop yield traits and plant fitness markers in natural habitats. The current project is a collaboration between plant researchers, modellers, engineers and image analysis specialists that aims to take a new approach, by producing i) an affordable and robust photometric stereo platform for 3D capture of rosette growth in Arabidopsis that can be extended to other crops, such as related Brassica species (e.g. Brassica rapa), and ii) a standardised open source software based in optimised machine learning algorithms and fronted by a biologist-friendly user interface for simplifying the extraction of traits of interest. To test the system we will target an important plant adaptive response that can negatively impact plant biomass and crop yield - the shade avoidance response (SAR). Initial computational work will focus on processing image data to gather 3D information for plant reconstruction through intelligent integration over the surface normal field. Subsequent image/model analysis will centre on the optimisation of leaf segmentation algorithms using features commonly used in machine vision and new approaches rooted in unsupervised feature learning. Following segmentation, we will quantify the plant phenotypic traits required for evaluating the key features of SAR, including: leaf emergence rates, positioning, orientation, as well as alterations in leaf length, width, shape and expansion. The novel algorithms resulting from this research will be integrated into the existing UoE PhenoTiki package (http://phenotiki.com) so that we reach a wide base of users. We aim to combine 3D phenotype data with mathematical modelling to predict growth outcomes in differing growth scenarios and in unseasonal weather, including controlled environments and outside in field or natural habitats.

Planned Impact

Who will benefit from this research?
1. Academics and researchers in all fields of plant research and machine learning.
2. UK and international science base.
4. The Agro-Tech industry including biotechnologists and plant breeders seeking to increase plant productivity, and metabolic engineers.
5. Agricultural community and advisors.
6. The University of Edinburgh.
6. The postdoctoral researcher (PDRA), and research associate (RA).
7. Public.

How will they benefit?
1. The research will have a major impact on understanding of light signalling and shade avoidance, and enhance the UK's international standing in plant science. Currently no other equipment can perform these measurements elsewhere in the world, nor integrate them with the wealth of molecular data and integrated modelling expertise available at UoE.
2. The technology is designed to enable labs in universities, research institutes and the Agro-Tech industry to pursue phenotyping approaches that have previously been largely inaccessible to them. To acquire the technology following publication, researchers will access an already established open source online resource (PhenoTiki) with an already existing user community to download a user-friendly software and have the option to reproduce the phenotyping system in their own labs, using our published designs that where possible will be made available for 3D printing.
3. Our system will provide a means of standardisation across the phenotyping field, to inform research strategies towards enhanced food security, which will be a platform for direct translation to achieving this goal in food crop plants through yield improvements. The public interest in food security and sustainable crop production, the broad biological scope of the project, and the predictive, systems approach already highlight areas of Impact.
4. The PDRA and RA will receive a wide training in plant integrative biology, hardware design and machine vision approaches, full access to professional skills and wider training courses, and the opportunity to work with two different research institutes.

What will be done to ensure they benefit from this research?
1. Publish results in high-impact journals in a timely fashion, with open access.
2. Present research results at UK and international meetings and institutions.
3. Exploit existing contacts with other UK and international academics with relevant research interests as soon as any exploitable results/materials are generated.
4. Make informal contacts with industrialists as soon as exploitable results/materials are generated; recognise and protect intellectual property to ensure wise and fruitful exploitation.
5. Provide PDRA and RA training and mentoring available at UoE and CMV, including regular reviews to monitor progress and a career development plan. We will also encourage participation in all aspects of the dissemination of research results, and understanding of the wider implications and applications of the research.
7. Use results as part of our regular engagement with non-academic audiences, e.g. local interest groups, schools, local and national shows, science showcases, media.

Publications


10 25 50
 
Description The project has been running for 5 months. During this time we have achieved the first part of WP2 - the design and construction of two photoemetric stereo image capture rigs. These have been delivered from Bristol to Edinburgh, and are currently capturing data for the second part of WP2 and WP3.
Exploitation Route We have only just started collecting data. However, the technology shows promise in terms of the data delivered thus far. We foresee that this technology will have an impact in the plant science and agriculture sectors. Furthermore, due to the low-cost of the technology, we envisage an impact in the education sector.
Sectors Agriculture, Food and Drink,Education
 
Description Stereoscopic camera array system - BRL, Centre for Machine Vision - UWE Bristol 
Organisation University of the West of England
Department Psychology
Country United Kingdom of Great Britain & Northern Ireland (UK) 
Sector Academic/University 
PI Contribution Currently collaborating on the construction and software development of a stereoscopic camera array system for capturing real-time 3D images of plants with the BRL, Centre for Machine Vision (CMV) (Smith Lab).
Collaborator Contribution The CMV has designed and built the rigs, and delivered them to Edinburgh for testing. We collaborate closely with the CMV for building software to extract growth traits.
Impact The project commenced in Oct 2016. Thus far the rigs have been assembled and data is currently being generated and analysed. This is a multidisciplinary collaboration between computer scientists (CMV), biologists and engineers (Ed).
Start Year 2016
 
Description Stereoscopic camera array system - Edinburgh, Eng 
Organisation University of Edinburgh
Department School of Biological Sciences Edinburgh
Country United Kingdom of Great Britain & Northern Ireland (UK) 
Sector Academic/University 
PI Contribution Currently collaborating on the construction and software development of a stereoscopic camera array system for capturing real-time 3D images of plants with UoE School of Engineering (Tsaftaris group).
Collaborator Contribution The Tsaftaris group brings specific expertise in machine learning and algorithm development for Arabidopsis plants. McCormick co-supervises a PhD student in the Tsaftaris group. We have shared access of data, protocols and biological materials between labs.
Impact The project commenced in Oct 2016. Thus far the rigs have been assembled and data is currently being generated and analysed. This is a multidisciplinary collaboration between computer scientists (CMV), biologists and engineers (Ed).
Start Year 2016
 
Description Stereoscopic camera array system - Edinburgh, SBS 
Organisation University of Edinburgh
Country United Kingdom of Great Britain & Northern Ireland (UK) 
Sector Academic/University 
PI Contribution Currently collaborating on the construction and software development of a stereoscopic camera array system for capturing real-time 3D images of plants with UoE School of Engineering (Tsaftaris group).
Collaborator Contribution The Halliday group brings specific expertise in shade avoidance Arabidopsis mutants (Phy mutants) - the primary targets for data generation in this project. McCormick co-supervises a PhD student in the Halliday group. We have shared access of data, protocols and biological materials between labs.
Impact The project commenced in Oct 2016. Thus far the rigs have been assembled and data is currently being generated and analysed. This is a multidisciplinary collaboration between computer scientists (CMV), biologists and engineers (Ed).
Start Year 2015
 
Title Plant Magic software for leaf rosette area analysis 
Description Plant Magic is a software tool for analysing images of plants and calculating the leaf/rosette area under both night and day conditions. An associated hardware tool is used for night imaging. 
Type Of Technology New/Improved Technique/Technology 
Year Produced 2016 
Impact The software was developed by a Masters student (Andrei Dobrescu). Andrei has now been accepted as a PhD student co-supervised by McCormick and Sotirios Tsaftaris (UoE, Eng) to continue work on plant image analysis. The software is currently used by several labs in UoE and will shortly be released on the open Plant Image Analysis website (http://www.plant-image-analysis.org/). An additional hardware tool (a low cost far-red LED array with a raspberry Pi camera) was developed to capture images throughout the diel cycle. This method for building this setup is being prepared for publication.