Apple Harvesters Use Vision To “See” Fruit

Editor’s Note: This past October, Tony Koselka, Jillian Cannons, and Ryan Carlon of Vision Robotics Corporation (VRC) returned to Washington for further field trials of the Newton Scout, a vision-based system for automatic crop load assessment. The following report highlights their results.

Developed in part through the USDA-funded project Comprehensive Automation for Specialty Crops (CASC), the Newton Scout will revolutionize orchard operations by providing growers with a map that accurately maps fruit placement, fruit size, and number throughout the growing season. Fruit maps will be used to determine crop load, which in turn will give growers the information to develop harvest and marketing strategies. And this is just the beginning. Developing reliable vision systems that are robust enough for use in orchards opens up all kinds of possibilities in the area of pest and disease detection, plant health assessment, and the use of robotics at harvest.

The Newton Scout integrates stereo cameras with a custom image processing algorithm. The algorithms use classifiers to identify the apples, where the simplest classifier uses color and shape. For red apples, color is the primary classifier. For detecting green fruit, the Scout detects using statistical classifiers generated by boosting algorithms. The most challenging aspects of detecting fruit are occlusions, shadows, shape and color coincidences, and ambient lighting conditions such as bright sunlight. Once an apple is identified, the stereo cameras enable the system to determine its location and size.

The Vision Robotics field trial was conducted at Allan Brothers Othello Orchard in Washington. Prior to VRC’s arrival, Washington State University (WSU) Extension Educator Karen Lewis, and interns with the Washington Tree Fruit Research Commission, hand tagged, mapped, and sized fruit on several rows of fruit. This data allows for the groundtruthing or validation component of the test. Koselka took time out to speak about the development of a similar unit that counts oranges with the interns.

On one of the field trials for that unit, Koselka asked the grower to estimate his crop. The farm manager responded with 1,200 bushels. For this test, Vision Robotics scouted and counted 1,900 — a sizable difference and one that would have a large financial impact.

As Koselka stated, “not only will it [Scout] be able to accurately count a crop and locate each piece of fruit, it will ultimately feed into a robot harvester that will pick all the apples.” And if that was not enough, Koselka added, “Oh, and a pruner is not far off either!”

Even More Potential

The Newton Scout utility will be enhanced in the future with the addition of other possible technologies, including near infrared (NIR). As Newton evolves, it will have multiple uses throughout the fruit growing cycle — for example, assessment of sugar and acid content and internal damage. CASC is also making progress in other automation fronts that bridge with Vision Robotics’ work.

Carnegie Mellon University (CMU) researchers, led by Sanjiv Singh, are developing an Autonomous Prime Mover (APM) capable of towing the Newton Scout safely through entire orchard blocks. In the future, they will also automate an agricultural harvesting platform so that workers no longer need to climb on ladders to pick fruit. The APM’s multifunctionality, autonomous drive, and wireless networking could move it into the center of precision management of specialty crops.

The CASC team, led by CMU, includes engineers, scientists, Extension educators, growers, equipment manufacturers, and industry representatives. This plurality of expertise has led to synergistic endeavors that push the envelope in both the engineering and the science components of the project. For example, Ben Grocholsky at CMU is developing a geospatial system to map and locate each tree in an orchard; in cooperation with the plant science group, he was able to augment this information with local temperature and NDVI data that has the future possibility of being incorporated to detect plant stress and disease.

In another example of the synergy of the CASC team, during a field trial of an automated caliper measurement device, growers told the CASC team that a simpler tree counter would also be very useful. Singh’s group returned to CMU and produced a prototype tree counter ready to be tested in weeks; they will return to nurseries and test the counter this fall. The total time from conception to development will be less than four months, and if successful, this device could go into production within a year.

It will only be a matter of time before specialty crop growers also start using vehicles, sensors, devices, and software that automate farm operations, increase farm efficiency, and meet the ultimate goal of delivering to the consumer a consistently high quality affordable eating experience while returning money to the farming operation.

For more information on Comprehensive Automation for Specialty Crops, please visit www.cascrop.com.

Leave a Reply

Featured Stories
OrganicDavid Peri of Peri & Sons Receives Outstanding Agriculturalist Award
May 22, 2015
David Peri, president of Peri & Sons Farms, has been named the 2015 Nevada Outstanding Agriculturalist Award by the University Read More
FruitsQ&A With Penn State’s New Young Grower Alliance Coordinator
May 22, 2015
Erin Dugan recently joined Penn State University extension as a specialty crop innovations program manager and Young Grower Alliance coordinator. Read More
FruitsHow To Start A Young Grower Association
May 22, 2015
You have recently returned to your family’s business. You have friends and family you can tap for their knowledge and Read More
Tomato bug
Insect ControlStop Tomato Bugs From Sucking Your Crops Dry
May 21, 2015
Learn how to identify, the survival and spread, as well as management methods of this vegetable pest. Read More
UF/IFAS developmental biologist Dennis Gray looks at the progress of grapevines in a vineyard.
Crop ProtectionResearcher Using ‘Precision Breeding’ To Create Disease-Resistant Winegrapes
May 20, 2015
Breakthrough would mean significant savings on pesticide and fungicide sprays. Read More
Spraying indoor citrus
Protected AgricultureInterest Continues To Grow In Protected Citriculture
May 20, 2015
Inside move could put producers in prime scoring position. Read More
Expansion groundbreaking for Southwest Florida Research and Education Center
CitrusSouthwest Florida Research And Education Center Embracing New Expansion
May 20, 2015
A 7,000-square-foot addition to the UF/IFAS facility will house labs and offices for potential new faculty members. Read More
storm clouds
CitrusSouth Florida Rainy Season Could Wind Up On Drier Side
May 20, 2015
National Weather Service anticipating El Niño to play a hand in possible below-normal conditions. Read More
Food SafetyProduce Safety Alliance Offers Course To Become A Certified Trainer
May 20, 2015
The Produce Safety Alliance (PSA) has announced dates for their first two Train-the-Trainer courses this June. The first of the Read More
GrapesMatthew Fidelibus To Be Recognized For Extension Work
May 20, 2015
The American Society of Enology and Viticulture (ASEV) recently announced that Western Fruit Grower™ magazine contributor Matthew W. Fidelibus, of Read More
The Latest
OrganicDavid Peri of Peri & Sons Receives Outstanding Agri…
May 22, 2015
David Peri, president of Peri & Sons Farms, has been named the 2015 Nevada Outstanding Agriculturalist Award by the University Read More
FruitsQ&A With Penn State’s New Young Grower Alliance Coo…
May 22, 2015
Erin Dugan recently joined Penn State University extension as a specialty crop innovations program manager and Young Grower Alliance coordinator. Read More
FruitsHow To Start A Young Grower Association
May 22, 2015
You have recently returned to your family’s business. You have friends and family you can tap for their knowledge and Read More
Tomato bug
Insect ControlStop Tomato Bugs From Sucking Your Crops Dry
May 21, 2015
Learn how to identify, the survival and spread, as well as management methods of this vegetable pest. Read More
UF/IFAS developmental biologist Dennis Gray looks at the progress of grapevines in a vineyard.
Crop ProtectionResearcher Using ‘Precision Breeding’ To Cr…
May 20, 2015
Breakthrough would mean significant savings on pesticide and fungicide sprays. Read More
Spraying indoor citrus
Protected AgricultureInterest Continues To Grow In Protected Citriculture
May 20, 2015
Inside move could put producers in prime scoring position. Read More
Expansion groundbreaking for Southwest Florida Research and Education Center
CitrusSouthwest Florida Research And Education Center Embraci…
May 20, 2015
A 7,000-square-foot addition to the UF/IFAS facility will house labs and offices for potential new faculty members. Read More
storm clouds
CitrusSouth Florida Rainy Season Could Wind Up On Drier Side
May 20, 2015
National Weather Service anticipating El Niño to play a hand in possible below-normal conditions. Read More
Food SafetyProduce Safety Alliance Offers Course To Become A Certi…
May 20, 2015
The Produce Safety Alliance (PSA) has announced dates for their first two Train-the-Trainer courses this June. The first of the Read More
GrapesMatthew Fidelibus To Be Recognized For Extension Work
May 20, 2015
The American Society of Enology and Viticulture (ASEV) recently announced that Western Fruit Grower™ magazine contributor Matthew W. Fidelibus, of Read More
Apples & PearsGrants Available For Farmworker Education
May 19, 2015
Grants are available for farmworkers and their families interested in attending adult educational programs through the Washington Apple Educational Foundation Read More
Fruits$10 Million Available For California Water Conservation…
May 19, 2015
Applications for the California Department of Food and Agriculture’s (CDFA) California State Water Efficiency and Enhancement Program (SWEEP) are now Read More
CitrusBad Weather Or Not, Preparation Always On Radar For Flo…
May 19, 2015
You cannot prevent a natural disaster from taking everything you have, but you can lessen the blow if and when it happens. Read More
Sweet CornGrower, Marketer Team Up To Launch Line Of Single-Serve…
May 19, 2015
Next Level Fresh, LLC announced the introduction of Fresh Life Harvest Whole Ear Gourmet Sweet Corn. The concept was developed Read More
CitrusSave Your Greenhouse Structures From Storms
May 19, 2015
Make sure your protected agriculture components are prepared for whatever may blow this way. Read More
FruitsGreen Fruitworm Numbers High In Pennsylvania
May 19, 2015
In their latest insect report, David Biddinger and Grzegorz Krawczyk, tree fruit entomologists discuss the timing of pest control applications Read More
Apples & PearsApple Scab Alert
May 18, 2015
Kari Peter of Penn State University Extension warned growers of possible apple scab and fire blight infections following the rainy Read More
FruitsUSDA To Invest $21M To Help Growers Cope With Drought
May 18, 2015
USDA’s National Resources Conservation Service (NRCS) will allocate $21 million for growers and ranchers to apply conservation practices in the Read More