Plants are the major organisms that can absorb the light energy from the sun to produce biomass and oxygen. One key problem in studying plant growth is to understand the photosynthetic activities of plants under various external stimuli or genetic variations. Because leaves at different developmental ages may response to the change of the environmental conditions and gene mutations in very different ways, it is important to conduct a leaf-level analysis of the photosynthetic efficiency. Motivated by the needs in plant biology, given images and videos captured by visual sensors, the goal of the plant vision project is to develop advanced computer vision algorithms to automatically, accurately and efficiently estimate the structure of a plant, which includes 2D multi-leaf alignment and tracking, and 3D reconstruction of all leaves in a plant. We have developed a framework based on the well-known Chamfer Matching algorithm. The input to our system is a fluorescence image or video of a plant, which is captured in a growth chamber (Fig. 1). Multi-leaf alignment aims to segment/align all leaves with pre-defined leaf templates and estimate the two tip points of each leaf. Multi-leaf tracking aims to track all leaves over time based on the alignment results of one frame. The leaf alignment and tracking results can directly benefit the study of leaf behavior in plant biology, such as leaf growth, leaf-level photosynthesis, leaf-level variations in plant mutant, etc. We have recently extended our system to process RGB videos of plants as well.

Multi-leaf Alignment

As shown in Fig. 2, the multi-leaf tracking algorithm consists of two steps. Firstly, a set of templates are applied to the test image to generate the same amount of leaf candidates. Secondly, we develop a multi-objective optimization process to select a subset of leaf candidates. The objective is to select a minimal number of leaf candidates with smaller Chamfer distances to cover the test image mask as much as possible.

Multi-leaf Tracking

Multi-leaf tracking is an extension of the leaf alignment algorithm. Given a fluorescence plant video taken over time, we first apply the alignment algorithm to the last frame of the video, and then continuously apply template transformation to the current leaf candidates in order to fit to the previous frame. We develop an objective function considering the Chamfer Matching distances, test image mask, and the rotation angels of all leaves. An example of multi-leaf tracking is shown below.

Publications

  • Leaf Segmentation in Plant Phenotyping: A Collation Study
    Hanno Scharr, Massimo Minervini, Andrew P. French, Christian Klukas, David M. Kramer, Xiaoming Liu, Imanol L. Muntion, Jean-Michel Pape, Gerrit Polder, Danijela Vukadinovic, Xi Yin, Sotirios A. Tsaftaris
    Machine Vision and Application, Vol. 7, Oct. 2015
    Bibtex | PDF | Project Webpage
  • @article{ leaf-segmentation-in-plant-phenotyping-a-collation-study,
      author = { Hanno Scharr and Massimo Minervini and Andrew P. French and Christian Klukas and David M. Kramer and Xiaoming Liu and Imanol L. Muntion and Jean-Michel Pape and Gerrit Polder and Danijela Vukadinovic and Xi Yin and Sotirios A. Tsaftaris },
      title = { Leaf Segmentation in Plant Phenotyping: A Collation Study },
      booktitle = { Machine Vision and Application },
      volume = { 7 },
      month = { October },
      year = { 2015 },
    }
  • Multi-modality Imagery Database for Plant Phenotyping
    Jeffrey A. Cruz, Xi Yin, Xiaoming Liu, Saif M. Imran, Daniel D. Morris, David M. Kramer, Jin Chen
    Machine Vision and Applications, Vol. 7, pp.1-15, Oct. 2015 (equal contribution by first two authors)
    Bibtex | PDF | Project Webpage
  • @article{ multi-modality-imagery-database-for-plant-phenotyping,
      author = { Jeffrey A. Cruz and Xi Yin and Xiaoming Liu and Saif M. Imran and Daniel D. Morris and David M. Kramer and Jin Chen },
      title = { Multi-modality Imagery Database for Plant Phenotyping },
      booktitle = { Machine Vision and Applications },
      volume = { 7 },
      month = { October },
      year = { 2015 },
      pages = { 1--15 },
    }
  • Joint Multi-Leaf Segmentation, Alignment, and Tracking from Fluorescence Plant Videos
    Xi Yin, Xiaoming Liu, Jin Chen, David M. Kramer
    Revision review in IEEE Transactions on Pattern Analysis and Machine Intelligence, May. 2015
    Bibtex | PDF | Project Webpage
  • @article{ joint-multi-leaf-segmentation-alignment-and-tracking-from-fluorescence-plant-videos,
      author = { Xi Yin and Xiaoming Liu and Jin Chen and David M. Kramer },
      title = { Joint Multi-Leaf Segmentation, Alignment, and Tracking from Fluorescence Plant Videos },
      booktitle = { Revision review in IEEE Transactions on Pattern Analysis and Machine Intelligence },
      volume = { },
      month = { May },
      year = { 2015 },
    }
  • Multi-Leaf Tracking from Fluorescence Plant Videos
    Xi Yin, Xiaoming Liu, Jin Chen, David M. Kramer
    in Proceedings of IEEE Conference on Image Processing (ICIP 2014), Paris, France, Oct. 2014 (Top 10% Paper Award)
    Bibtex | PDF | Project Webpage
  • @inproceedings{ multi-leaf-tracking-from-fluorescence-plant-videos,
      author = { Xi Yin and Xiaoming Liu and Jin Chen and David M. Kramer },
      title = { Multi-Leaf Tracking from Fluorescence Plant Videos },
      booktitle = { in Proceedings of IEEE Conference on Image Processing },
      address = { Paris, France },
      month = { October },
      year = { 2014 },
    }
  • Multi-Leaf Alignment from Fluorescence Plant Images
    Xi Yin, Xiaoming Liu, Jin Chen, David M. Kramer
    Proc. IEEE Winter Conference on Application of Computer Vision (WACV 2014), Steamboat Springs, USA, Mar. 2014 (Best Student Paper Award)
    Bibtex | PDF | Project Webpage
  • @inproceedings{ multi-leaf-alignment-from-fluorescence-plant-images,
      author = { Xi Yin and Xiaoming Liu and Jin Chen and David M. Kramer },
      title = { Multi-Leaf Alignment from Fluorescence Plant Images },
      booktitle = { Proc. IEEE Winter Conference on Application of Computer Vision },
      address = { Steamboat Springs, USA },
      month = { March },
      year = { 2014 },
    }