Apple Harvesters Use Vision To “See” Fruit

Editor’s Note: This past October, Tony Koselka, Jillian Cannons, and Ryan Carlon of Vision Robotics Corporation (VRC) returned to Washington for further field trials of the Newton Scout, a vision-based system for automatic crop load assessment. The following report highlights their results.

Developed in part through the USDA-funded project Comprehensive Automation for Specialty Crops (CASC), the Newton Scout will revolutionize orchard operations by providing growers with a map that accurately maps fruit placement, fruit size, and number throughout the growing season. Fruit maps will be used to determine crop load, which in turn will give growers the information to develop harvest and marketing strategies. And this is just the beginning. Developing reliable vision systems that are robust enough for use in orchards opens up all kinds of possibilities in the area of pest and disease detection, plant health assessment, and the use of robotics at harvest.

The Newton Scout integrates stereo cameras with a custom image processing algorithm. The algorithms use classifiers to identify the apples, where the simplest classifier uses color and shape. For red apples, color is the primary classifier. For detecting green fruit, the Scout detects using statistical classifiers generated by boosting algorithms. The most challenging aspects of detecting fruit are occlusions, shadows, shape and color coincidences, and ambient lighting conditions such as bright sunlight. Once an apple is identified, the stereo cameras enable the system to determine its location and size.

The Vision Robotics field trial was conducted at Allan Brothers Othello Orchard in Washington. Prior to VRC’s arrival, Washington State University (WSU) Extension Educator Karen Lewis, and interns with the Washington Tree Fruit Research Commission, hand tagged, mapped, and sized fruit on several rows of fruit. This data allows for the groundtruthing or validation component of the test. Koselka took time out to speak about the development of a similar unit that counts oranges with the interns.

On one of the field trials for that unit, Koselka asked the grower to estimate his crop. The farm manager responded with 1,200 bushels. For this test, Vision Robotics scouted and counted 1,900 — a sizable difference and one that would have a large financial impact.

As Koselka stated, “not only will it [Scout] be able to accurately count a crop and locate each piece of fruit, it will ultimately feed into a robot harvester that will pick all the apples.” And if that was not enough, Koselka added, “Oh, and a pruner is not far off either!”

Even More Potential

The Newton Scout utility will be enhanced in the future with the addition of other possible technologies, including near infrared (NIR). As Newton evolves, it will have multiple uses throughout the fruit growing cycle — for example, assessment of sugar and acid content and internal damage. CASC is also making progress in other automation fronts that bridge with Vision Robotics’ work.

Carnegie Mellon University (CMU) researchers, led by Sanjiv Singh, are developing an Autonomous Prime Mover (APM) capable of towing the Newton Scout safely through entire orchard blocks. In the future, they will also automate an agricultural harvesting platform so that workers no longer need to climb on ladders to pick fruit. The APM’s multifunctionality, autonomous drive, and wireless networking could move it into the center of precision management of specialty crops.

The CASC team, led by CMU, includes engineers, scientists, Extension educators, growers, equipment manufacturers, and industry representatives. This plurality of expertise has led to synergistic endeavors that push the envelope in both the engineering and the science components of the project. For example, Ben Grocholsky at CMU is developing a geospatial system to map and locate each tree in an orchard; in cooperation with the plant science group, he was able to augment this information with local temperature and NDVI data that has the future possibility of being incorporated to detect plant stress and disease.

In another example of the synergy of the CASC team, during a field trial of an automated caliper measurement device, growers told the CASC team that a simpler tree counter would also be very useful. Singh’s group returned to CMU and produced a prototype tree counter ready to be tested in weeks; they will return to nurseries and test the counter this fall. The total time from conception to development will be less than four months, and if successful, this device could go into production within a year.

It will only be a matter of time before specialty crop growers also start using vehicles, sensors, devices, and software that automate farm operations, increase farm efficiency, and meet the ultimate goal of delivering to the consumer a consistently high quality affordable eating experience while returning money to the farming operation.

For more information on Comprehensive Automation for Specialty Crops, please visit

Leave a Reply

Featured Stories

All Vegetables Stories >All Fruits Stories >All Nuts Stories >All Citrus Stories >

The Latest

Crop ProtectionGetting To The Root Of Good Soil Health Requires Some D…
August 21, 2014
Dave Gilliam of Horticultural Alliance says more citrus growers are paying attention to what's happening below the ground in their groves. Read More
Citrus‘Super Cold’ Winter, Sizzling Summer On Tap In 2015…
August 21, 2014
The 223rd edition of the folksy regional forecast manual predicting extremes. Read More
Insect & Disease UpdateMountains Of Data Being Used To Combat Citrus Greening
August 20, 2014
As HLB research evolves, new methods need to be developed to effectively utilize information. Read More
VegetablesAnnual Santa Maria Vegetable Meeting To Cover Progressi…
August 20, 2014
University of California Cooperative Extension will host annual vegetable meeting focusing on nutrient management, plant growth, weed management, and more. Read More
Leafy VegetablesTanimura & Antle Introduces New Lettuce Variety 
August 20, 2014
The California grower now offers George T’s Colossal Romaine Heart, in honor of the company's founder, George Tanimura. Read More
CitrusBorder Crisis Not Helping Farmers
August 19, 2014
As thousands cross into the U.S. seeking refuge, calls for ag labor reform are lost in the uproar. Read More
Cucurbits24 Sweet Watermelon Varieties [Slideshow]
August 19, 2014
Browse the slideshow below for information on 24 watermelon varieties from the nation’s leading seed breeders and distributors. Read More
FruitsHandheld Produce Quality Meter Debuts At 2014 Internati…
August 19, 2014
Researchers to present data measuring dry matter, color, and sugar content of cherries and other product pre- and postharvest. Read More
GrapesNew York State Assists Grape Growers Hit By Harsh Winte…
August 19, 2014
State to open winegrape market to grapes grown outside the state. Read More
FruitsEuropean Fruit And Vegetable Growers Hit By Russian Ban…
August 19, 2014
Angry at European Union/United States sanctions over Ukraine, Russia has banned many food imports. Read More
Apple Grower of the YearGet The Latest News On The Nation’s Apple Crop
August 19, 2014
American and Western Fruit Grower editors will be tweeting in real time this week from the Apple Crop Outlook & Marketing Conference. Read More
OrangesSour Forecast For 2014-2015 Florida Orange Crop
August 18, 2014
Paltry prediction signals lowest output in 50 years. Read More
Crop ProtectionBioConsortia Inc. Bolsters Executive Team
August 18, 2014
Industry veterans Christina Huben and Dr. Susan Turner bring experience to plant biotechnology firm. Read More
Stone Fruit‘Ladderless’ Peach, Nectarine Orchards Explored
August 15, 2014
University of California researchers explore the concept of so-called pedestrian orchards. Read More
CitrusU.S. Sugar Buying South Florida Sugar Cane And Vegetabl…
August 15, 2014
Purported deal worth $100 million to purchase farmland and assets of Knight Management Inc. Read More
CitrusFlorida’s Future Farming Leaders Dig Up Knowledge…
August 15, 2014
Class 3 of FFVA's Emerging Leader Development Program learn a lot from road trip to California's Salinas Valley. Read More
CitrusWater Bond Will Appear On California Ballot
August 14, 2014
Voters will decide if thirsty state will spend $7.5 billion, including $2.7 billion for storage. Read More
BerriesFamiliar Face Settles In As New Florida Strawberry Asso…
August 14, 2014
Kenneth Parker’s deep roots in the community and knowledge of production challenges make a good combination for executive director role. Read More