Apple Harvesters Use Vision To “See” Fruit

Editor’s Note: This past October, Tony Koselka, Jillian Cannons, and Ryan Carlon of Vision Robotics Corporation (VRC) returned to Washington for further field trials of the Newton Scout, a vision-based system for automatic crop load assessment. The following report highlights their results.

Developed in part through the USDA-funded project Comprehensive Automation for Specialty Crops (CASC), the Newton Scout will revolutionize orchard operations by providing growers with a map that accurately maps fruit placement, fruit size, and number throughout the growing season. Fruit maps will be used to determine crop load, which in turn will give growers the information to develop harvest and marketing strategies. And this is just the beginning. Developing reliable vision systems that are robust enough for use in orchards opens up all kinds of possibilities in the area of pest and disease detection, plant health assessment, and the use of robotics at harvest.

The Newton Scout integrates stereo cameras with a custom image processing algorithm. The algorithms use classifiers to identify the apples, where the simplest classifier uses color and shape. For red apples, color is the primary classifier. For detecting green fruit, the Scout detects using statistical classifiers generated by boosting algorithms. The most challenging aspects of detecting fruit are occlusions, shadows, shape and color coincidences, and ambient lighting conditions such as bright sunlight. Once an apple is identified, the stereo cameras enable the system to determine its location and size.

The Vision Robotics field trial was conducted at Allan Brothers Othello Orchard in Washington. Prior to VRC’s arrival, Washington State University (WSU) Extension Educator Karen Lewis, and interns with the Washington Tree Fruit Research Commission, hand tagged, mapped, and sized fruit on several rows of fruit. This data allows for the groundtruthing or validation component of the test. Koselka took time out to speak about the development of a similar unit that counts oranges with the interns.

On one of the field trials for that unit, Koselka asked the grower to estimate his crop. The farm manager responded with 1,200 bushels. For this test, Vision Robotics scouted and counted 1,900 — a sizable difference and one that would have a large financial impact.

As Koselka stated, “not only will it [Scout] be able to accurately count a crop and locate each piece of fruit, it will ultimately feed into a robot harvester that will pick all the apples.” And if that was not enough, Koselka added, “Oh, and a pruner is not far off either!”

Even More Potential

The Newton Scout utility will be enhanced in the future with the addition of other possible technologies, including near infrared (NIR). As Newton evolves, it will have multiple uses throughout the fruit growing cycle — for example, assessment of sugar and acid content and internal damage. CASC is also making progress in other automation fronts that bridge with Vision Robotics’ work.

Carnegie Mellon University (CMU) researchers, led by Sanjiv Singh, are developing an Autonomous Prime Mover (APM) capable of towing the Newton Scout safely through entire orchard blocks. In the future, they will also automate an agricultural harvesting platform so that workers no longer need to climb on ladders to pick fruit. The APM’s multifunctionality, autonomous drive, and wireless networking could move it into the center of precision management of specialty crops.

The CASC team, led by CMU, includes engineers, scientists, Extension educators, growers, equipment manufacturers, and industry representatives. This plurality of expertise has led to synergistic endeavors that push the envelope in both the engineering and the science components of the project. For example, Ben Grocholsky at CMU is developing a geospatial system to map and locate each tree in an orchard; in cooperation with the plant science group, he was able to augment this information with local temperature and NDVI data that has the future possibility of being incorporated to detect plant stress and disease.

In another example of the synergy of the CASC team, during a field trial of an automated caliper measurement device, growers told the CASC team that a simpler tree counter would also be very useful. Singh’s group returned to CMU and produced a prototype tree counter ready to be tested in weeks; they will return to nurseries and test the counter this fall. The total time from conception to development will be less than four months, and if successful, this device could go into production within a year.

It will only be a matter of time before specialty crop growers also start using vehicles, sensors, devices, and software that automate farm operations, increase farm efficiency, and meet the ultimate goal of delivering to the consumer a consistently high quality affordable eating experience while returning money to the farming operation.

For more information on Comprehensive Automation for Specialty Crops, please visit

Leave a Reply

Featured Stories

All Vegetables Stories >All Fruits Stories >All Nuts Stories >All Citrus Stories >

The Latest

GenNext GrowersGrowers Need To Be Mindful When Dealing With The Media
October 1, 2014
When being interviewed for a story, preparation is the foundation to help you successfully get your point across. Read More
Citrus Achievement AwardSharing Is Caring When It Comes To Curing Citrus Greeni…
October 1, 2014
Mike Sparks, 2014 Citrus Achievement Award winner, says being open with peers about what's working and not working is crucial in managing HLB. Read More
Florida Ag ExpoKnow How To Knock Back Nematodes
October 1, 2014
The 2014 Florida Ag Expo will provide critical tips in soilborne pest management. Read More
CEU SeriesCEU Series: Vow To Vanquish Weeds Among Vegetables
October 1, 2014
Herbicidal neglect can and will kill your crops. Read More
Insect & Disease UpdateUF Names Interim Director For Citrus Research And Educa…
September 30, 2014
Michael Rogers has been a central figure in Florida’s battle to survive greening. Read More
BerriesPractice Persistence When Battling Botrytis In Blueberr…
September 30, 2014
Cognizance of resistance is key to managing formidable fungus. Read More
CitrusFarming Is Quite The Scary Prospect For Some [Opinion]
September 30, 2014
Florida Grower managing editor Paul Rusnak says economic realities might frighten off future leaders from noble professions. Read More
FruitsNew Suppress Herbicide Gets Green Light From EPA
September 30, 2014
Approval gives organic growers new weed management tool. Read More
BerriesNew Fill-By-Weight Clamshell Filler For Blueberries
September 30, 2014
Lakewood Process Machinery's equipment offers a simplified user interface, minimized drop heights, a new dribble-gate system designed for an increased level of accuracy. Read More
Stone FruitLarge Cherry Crop Expected In Washington
September 30, 2014
Approximately 23.5 million boxes of cherries were sold during season. Read More
LaborWestern Growers President Speaks Out About Passage Of C…
September 29, 2014
California Governor Jerry Brown recently vetoed a Senate Bill 25 and approved Assembly Bill 1897. Read More
BerriesDrought Could Impact Spring California Strawberry Crop
September 29, 2014
Ventura County expects reduction of acres planted as surface water availability takes a hit. Read More
Apples & PearsUnusual Weather Slows Michigan Apple Maturity
September 29, 2014
Fruit may have lower Brix levels, soluble solids lower than normal. Read More
BerriesFlorida Blueberry Growers Ready To Rise To The Occasion…
September 29, 2014
Florida Blueberry Growers Association president Dudley Calfee provides state of the industry insight. Read More
CitrusFlorida Announces Its 2014 Woman Of The Year In Agricul…
September 29, 2014
Longtime award recognizes women who have made outstanding contributions to the state’s farming sector. Read More
FruitsNational Security May Depend On A Healthy Diet [Opinion…
September 29, 2014
Your bottom line as well as the future of our country will depend on promoting healthy eating to the next generation. Read More
CitrusFlorida Produce Industry Embracing Progress, Confrontin…
September 29, 2014
Production pressures and politics hot topics at FFVA’s 71st Annual Convention. Read More
BerriesDon’t Let Your Guard Slip When Taking On Thrips
September 27, 2014
Tiny insect pests can present big problems for blueberry growers. Read More