Plant Vision

Plants are the major organisms that can absorb the light energy from the sun to produce biomass and oxygen. One key problem in studying plant growth is to understand the photosynthetic activities of plants under various external stimuli or genetic variations. Because leaves at different developmental ages may response to the change of the environmental conditions and gene mutations in very different ways, it is important to conduct a leaf-level analysis of the photosynthetic efficiency. Motivated by the needs in plant biology, given images and videos captured by visual sensors, the goal of the plant vision project is to develop advanced computer vision algorithms to automatically, accurately and efficiently estimate the structure of a plant, which includes 2D multi-leaf alignment and tracking, and 3D reconstruction of all leaves in a plant. We have developed a framework based on the well-known Chamfer Matching algorithm. The input to our system is a fluorescence image or video of a plant, which is captured in a growth chamber (Fig. 1). Multi-leaf alignment aims to segment/align all leaves with pre-defined leaf templates and estimate the two tip points of each leaf. Multi-leaf tracking aims to track all leaves over time based on the alignment results of one frame. The leaf alignment and tracking results can directly benefit the study of leaf behavior in plant biology, such as leaf growth, leaf-level photosynthesis, leaf-level variations in plant mutant, etc. We have recently extended our system to process RGB videos of plants as well.

Multi-leaf Alignment

As shown in Fig. 2, the multi-leaf tracking algorithm consists of two steps. Firstly, a set of templates are applied to the test image to generate the same amount of leaf candidates. Secondly, we develop a multi-objective optimization process to select a subset of leaf candidates. The objective is to select a minimal number of leaf candidates with smaller Chamfer distances to cover the test image mask as much as possible.

Multi-leaf Tracking

Multi-leaf tracking is an extension of the leaf alignment algorithm. Given a fluorescence plant video taken over time, we first apply the alignment algorithm to the last frame of the video, and then continuously apply template transformation to the current leaf candidates in order to fit to the previous frame. We develop an objective function considering the Chamfer Matching distances, test image mask, and the rotation angels of all leaves. An example of multi-leaf tracking is shown below.

Plant vision source code may be downloaded from here.

If you use the code, please cite to the papers:

Publications

Joint Multi-Leaf Segmentation, Alignment, and Tracking from Fluorescence Plant Videos
Xi Yin, Xiaoming Liu, Jin Chen, David M. Kramer
IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 40, No. 6, pp.1411-1423, , Jul. 2017
Bibtex | PDF | arXiv | Code

Multi-modality Imagery Database for Plant Phenotyping
Jeffrey Cruz, Xi Yin, Xiaoming Liu, Saif Imran, Daniel Morris, David Kramer, Jin Chen
Machine Vision and Applications, Vol. 27, No. 5, pp.735-749, , Jul. 2016 (equal contribution by first two authors)
Bibtex | PDF

Leaf Segmentation in Plant Phenotyping: A Collation Study
Hanno Scharr, Massimo Minervini, Andrew P. French, Christian Klukas, David M. Kramer, Xiaoming Liu, Imanol L. Muntion, Jean-Michel Pape, Gerrit Polder, Danijela Vukadinovic, Xi Yin, Sotirios A. Tsaftaris
Machine Vision and Application, Vol. ['27', '7'], No. 4, pp.585-606, , May. 2016
Bibtex | PDF | Code

Multi-Leaf Tracking from Fluorescence Plant Videos
Xi Yin, Xiaoming Liu, Jin Chen, David M. Kramer
In Proceedings of IEEE Conference on Image Processing (ICIP 2014), Paris, France, Oct. 2014 (Top 10% Paper Award)
Bibtex | PDF

Multi-Leaf Alignment from Fluorescence Plant Images
Xi Yin, Xiaoming Liu, Jin Chen, David M. Kramer
Proc. IEEE Winter Conference on Application of Computer Vision (WACV 2014), Steamboat Springs, USA, Mar. 2014 (Best Student Paper Award)
Bibtex | PDF