Apple Harvesters Use Vision To “See” Fruit

Editor’s Note: This past October, Tony Koselka, Jillian Cannons, and Ryan Carlon of Vision Robotics Corporation (VRC) returned to Washington for further field trials of the Newton Scout, a vision-based system for automatic crop load assessment. The following report highlights their results.

Developed in part through the USDA-funded project Comprehensive Automation for Specialty Crops (CASC), the Newton Scout will revolutionize orchard operations by providing growers with a map that accurately maps fruit placement, fruit size, and number throughout the growing season. Fruit maps will be used to determine crop load, which in turn will give growers the information to develop harvest and marketing strategies. And this is just the beginning. Developing reliable vision systems that are robust enough for use in orchards opens up all kinds of possibilities in the area of pest and disease detection, plant health assessment, and the use of robotics at harvest.

The Newton Scout integrates stereo cameras with a custom image processing algorithm. The algorithms use classifiers to identify the apples, where the simplest classifier uses color and shape. For red apples, color is the primary classifier. For detecting green fruit, the Scout detects using statistical classifiers generated by boosting algorithms. The most challenging aspects of detecting fruit are occlusions, shadows, shape and color coincidences, and ambient lighting conditions such as bright sunlight. Once an apple is identified, the stereo cameras enable the system to determine its location and size.

The Vision Robotics field trial was conducted at Allan Brothers Othello Orchard in Washington. Prior to VRC’s arrival, Washington State University (WSU) Extension Educator Karen Lewis, and interns with the Washington Tree Fruit Research Commission, hand tagged, mapped, and sized fruit on several rows of fruit. This data allows for the groundtruthing or validation component of the test. Koselka took time out to speak about the development of a similar unit that counts oranges with the interns.

On one of the field trials for that unit, Koselka asked the grower to estimate his crop. The farm manager responded with 1,200 bushels. For this test, Vision Robotics scouted and counted 1,900 — a sizable difference and one that would have a large financial impact.

As Koselka stated, “not only will it [Scout] be able to accurately count a crop and locate each piece of fruit, it will ultimately feed into a robot harvester that will pick all the apples.” And if that was not enough, Koselka added, “Oh, and a pruner is not far off either!”

Even More Potential

The Newton Scout utility will be enhanced in the future with the addition of other possible technologies, including near infrared (NIR). As Newton evolves, it will have multiple uses throughout the fruit growing cycle — for example, assessment of sugar and acid content and internal damage. CASC is also making progress in other automation fronts that bridge with Vision Robotics’ work.

Carnegie Mellon University (CMU) researchers, led by Sanjiv Singh, are developing an Autonomous Prime Mover (APM) capable of towing the Newton Scout safely through entire orchard blocks. In the future, they will also automate an agricultural harvesting platform so that workers no longer need to climb on ladders to pick fruit. The APM’s multifunctionality, autonomous drive, and wireless networking could move it into the center of precision management of specialty crops.

The CASC team, led by CMU, includes engineers, scientists, Extension educators, growers, equipment manufacturers, and industry representatives. This plurality of expertise has led to synergistic endeavors that push the envelope in both the engineering and the science components of the project. For example, Ben Grocholsky at CMU is developing a geospatial system to map and locate each tree in an orchard; in cooperation with the plant science group, he was able to augment this information with local temperature and NDVI data that has the future possibility of being incorporated to detect plant stress and disease.

In another example of the synergy of the CASC team, during a field trial of an automated caliper measurement device, growers told the CASC team that a simpler tree counter would also be very useful. Singh’s group returned to CMU and produced a prototype tree counter ready to be tested in weeks; they will return to nurseries and test the counter this fall. The total time from conception to development will be less than four months, and if successful, this device could go into production within a year.

It will only be a matter of time before specialty crop growers also start using vehicles, sensors, devices, and software that automate farm operations, increase farm efficiency, and meet the ultimate goal of delivering to the consumer a consistently high quality affordable eating experience while returning money to the farming operation.

For more information on Comprehensive Automation for Specialty Crops, please visit www.cascrop.com.

Leave a Reply

Featured Stories
the sunset on a hot day
Citrus
January 18, 2017
NASA, NOAA Concur 2016 Was World’s Warmest Year on Record
For the third time in three years, the bar is raised on surface temperature statistics. Read More
Vegetables
January 18, 2017
Southwest Growers Are Best Prepared for Succession
Almost half of all Southwest operations are grooming its next generation of leadership — an alarmingly low statistic, but one that is the highest in the country, according to American Vegetable Grower magazine's 2017 State of the Vegetable Industry survey. Read More
Vegetables
January 18, 2017
The Midwest Is Nurturing the Next Generation of Operations
It has more young businesses, percentage wise, than other regions: 39% of responding businesses are less than 10 years old. Read More
Vegetables
January 18, 2017
The West Had the Highest Increase in U.S. Vegetable Production in 2016
The West not only has the largest vegetable operations, it also saw the strongest growth in production in the U.S., according to American Vegetable Grower magazine's 2017 State of the Vegetable Industry survey. Read More
Vegetables
January 18, 2017
One in Five Southeast Growers Use H-2A
American Vegetable Grower magazine's 2017 State of the Vegetable Industry survey revealed quite a few qualities about the Southeast that may surprise you, including it being more likely to use H-2A than other regions. Read More
Vegetables
January 18, 2017
The 2016 Drought Is Having a Big Impact on Northeast Growers
The 2017 State of the Vegetable Industry gave us insight into each region of the U.S. Sifting through the data, we flagged those responses that were markedly higher or lower than other regions. Responses from Northeast growers made it clear that the 2016 drought had taken a toll. Read More
GenNext Growers
January 18, 2017
$858,000 in Grants to Encourage Careers in Agriculture Science
Funding to invest in programs that educate, promote science in the classroom. Read More
Vegetables
January 18, 2017
Mississippi Greenhouse Tomato Short Course Registration Is Now Open
Online registration is now open for the Greenhouse Tomato Short Course, which takes place March 7-8 in Raymond, MS. This Read More
Fruits
January 18, 2017
Spotted Wing Drosophila Control the Topic of Webinar
Research updates, recommendations to be presented on devastating pest. Read More
'Gold nugget' seedless tangerine
Varieties & Rootstocks
January 18, 2017
5 Florida Citrus Nursery Trends Worth Watching
Gleanings from recent grower gatherings expose opportunities and possibilities in new variety development and management. Read More
The Latest
Citrus
January 18, 2017
NASA, NOAA Concur 2016 Was World’s Warme…
For the third time in three years, the bar is raised on surface temperature statistics. Read More
Vegetables
January 18, 2017
Southwest Growers Are Best Prepared for …
Almost half of all Southwest operations are grooming its next generation of leadership — an alarmingly low statistic, but one that is the highest in the country, according to American Vegetable Grower magazine's 2017 State of the Vegetable Industry survey. Read More
Vegetables
January 18, 2017
The Midwest Is Nurturing the Next Genera…
It has more young businesses, percentage wise, than other regions: 39% of responding businesses are less than 10 years old. Read More
Vegetables
January 18, 2017
The West Had the Highest Increase in U.S…
The West not only has the largest vegetable operations, it also saw the strongest growth in production in the U.S., according to American Vegetable Grower magazine's 2017 State of the Vegetable Industry survey. Read More
Vegetables
January 18, 2017
One in Five Southeast Growers Use H-2A
American Vegetable Grower magazine's 2017 State of the Vegetable Industry survey revealed quite a few qualities about the Southeast that may surprise you, including it being more likely to use H-2A than other regions. Read More
Vegetables
January 18, 2017
The 2016 Drought Is Having a Big Impact …
The 2017 State of the Vegetable Industry gave us insight into each region of the U.S. Sifting through the data, we flagged those responses that were markedly higher or lower than other regions. Responses from Northeast growers made it clear that the 2016 drought had taken a toll. Read More
GenNext Growers
January 18, 2017
$858,000 in Grants to Encourage Careers …
Funding to invest in programs that educate, promote science in the classroom. Read More
Vegetables
January 18, 2017
Mississippi Greenhouse Tomato Short Cour…
Online registration is now open for the Greenhouse Tomato Short Course, which takes place March 7-8 in Raymond, MS. This Read More
Fruits
January 18, 2017
Spotted Wing Drosophila Control the Topi…
Research updates, recommendations to be presented on devastating pest. Read More
Varieties & Rootstocks
January 18, 2017
5 Florida Citrus Nursery Trends Worth Wa…
Gleanings from recent grower gatherings expose opportunities and possibilities in new variety development and management. Read More
GenNext Growers
January 17, 2017
9 Financial Resolutions to Boost Your Fa…
If a goal for the new year includes increasing profitability, there are ways you can better manage your business. Read More
Farm Management
January 17, 2017
Will Big Data Yield Big Returns for Farm…
Modern tools of hort tech are ripe to inspire the next generation of productivity and profitability. Read More
Citrus
January 17, 2017
The Future of Agriculture is in Your Han…
Can farmers actually reach the point of having too much information? Read More
Equipment
January 17, 2017
Agricultural Robots No Longer Science Fi…
New automated technologies could help specialty crop growers deal with labor crisis. Read More
Citrus
January 16, 2017
First Bee in Continental U.S. Listed as …
Rusty patched bumble bee receives protection from activities that could cause it to go extinct. Read More
Fruits
January 16, 2017
Pollination Experts Host Webinar Series
Fruit and vegetable growers can prepare for spring by hearing about recent pollination research. Read More
Citrus
January 16, 2017
Precision Agriculture and Big Data Gaini…
Specialty crop adoption of hort tech to usher in new efficiencies and transparency. Read More
Crop Protection
January 16, 2017
Monsanto, NRGene Enter Agreement for Big…
Technology platform is designed to help Monsanto analyze, store, and mine its genomic data sets to enhance breeding and R&D opportunities. Read More