Apple Harvesters Use Vision To “See” Fruit

Editor’s Note: This past October, Tony Koselka, Jillian Cannons, and Ryan Carlon of Vision Robotics Corporation (VRC) returned to Washington for further field trials of the Newton Scout, a vision-based system for automatic crop load assessment. The following report highlights their results.

Developed in part through the USDA-funded project Comprehensive Automation for Specialty Crops (CASC), the Newton Scout will revolutionize orchard operations by providing growers with a map that accurately maps fruit placement, fruit size, and number throughout the growing season. Fruit maps will be used to determine crop load, which in turn will give growers the information to develop harvest and marketing strategies. And this is just the beginning. Developing reliable vision systems that are robust enough for use in orchards opens up all kinds of possibilities in the area of pest and disease detection, plant health assessment, and the use of robotics at harvest.

The Newton Scout integrates stereo cameras with a custom image processing algorithm. The algorithms use classifiers to identify the apples, where the simplest classifier uses color and shape. For red apples, color is the primary classifier. For detecting green fruit, the Scout detects using statistical classifiers generated by boosting algorithms. The most challenging aspects of detecting fruit are occlusions, shadows, shape and color coincidences, and ambient lighting conditions such as bright sunlight. Once an apple is identified, the stereo cameras enable the system to determine its location and size.

The Vision Robotics field trial was conducted at Allan Brothers Othello Orchard in Washington. Prior to VRC’s arrival, Washington State University (WSU) Extension Educator Karen Lewis, and interns with the Washington Tree Fruit Research Commission, hand tagged, mapped, and sized fruit on several rows of fruit. This data allows for the groundtruthing or validation component of the test. Koselka took time out to speak about the development of a similar unit that counts oranges with the interns.

On one of the field trials for that unit, Koselka asked the grower to estimate his crop. The farm manager responded with 1,200 bushels. For this test, Vision Robotics scouted and counted 1,900 — a sizable difference and one that would have a large financial impact.

As Koselka stated, “not only will it [Scout] be able to accurately count a crop and locate each piece of fruit, it will ultimately feed into a robot harvester that will pick all the apples.” And if that was not enough, Koselka added, “Oh, and a pruner is not far off either!”

Even More Potential

The Newton Scout utility will be enhanced in the future with the addition of other possible technologies, including near infrared (NIR). As Newton evolves, it will have multiple uses throughout the fruit growing cycle — for example, assessment of sugar and acid content and internal damage. CASC is also making progress in other automation fronts that bridge with Vision Robotics’ work.

Carnegie Mellon University (CMU) researchers, led by Sanjiv Singh, are developing an Autonomous Prime Mover (APM) capable of towing the Newton Scout safely through entire orchard blocks. In the future, they will also automate an agricultural harvesting platform so that workers no longer need to climb on ladders to pick fruit. The APM’s multifunctionality, autonomous drive, and wireless networking could move it into the center of precision management of specialty crops.

The CASC team, led by CMU, includes engineers, scientists, Extension educators, growers, equipment manufacturers, and industry representatives. This plurality of expertise has led to synergistic endeavors that push the envelope in both the engineering and the science components of the project. For example, Ben Grocholsky at CMU is developing a geospatial system to map and locate each tree in an orchard; in cooperation with the plant science group, he was able to augment this information with local temperature and NDVI data that has the future possibility of being incorporated to detect plant stress and disease.

In another example of the synergy of the CASC team, during a field trial of an automated caliper measurement device, growers told the CASC team that a simpler tree counter would also be very useful. Singh’s group returned to CMU and produced a prototype tree counter ready to be tested in weeks; they will return to nurseries and test the counter this fall. The total time from conception to development will be less than four months, and if successful, this device could go into production within a year.

It will only be a matter of time before specialty crop growers also start using vehicles, sensors, devices, and software that automate farm operations, increase farm efficiency, and meet the ultimate goal of delivering to the consumer a consistently high quality affordable eating experience while returning money to the farming operation.

For more information on Comprehensive Automation for Specialty Crops, please visit www.cascrop.com.

Leave a Reply

Featured Stories
Brown marmorated stink bug. (Photo credit: USDA)
Crop Protection
May 6, 2016
Scouting Tips For Growing Grape Pest Threat
Grape growers should be on the lookout this fall for two pests that are popping up in vineyards along the Read More
Laurel wilt-damaged avocado tree leaves
Disease Control
May 6, 2016
Avocado Tree Killer On The Move In Florida
Devastating laurel wilt disease now reported in 61 of 67 counties. Read More
western food for safety and security course
Farm Management
May 6, 2016
$8.7 Million Available In Risk Management Education Funding
The funds are for programs to help underserved growers and ranchers to learn how to manage long-term risks and challenges. Read More
photo credit: USDA
Food Safety
May 6, 2016
Recall Expands For Frozen Fruit And Vegetables
CRF Frozen Foods of Pasco, WA, is expanding its voluntary recall of frozen organic and traditional fruits and vegetables due to potential Listeria contamination. Read More
Potatoes
Potatoes
May 6, 2016
National Potato Council Summer Meeting Slated For July
The summer meeting is open to growers and industry leaders interested in advancing the potato industry’s most pressing policy debates. Read More
Harris Seeds
Vegetables
May 6, 2016
Head Of Harris Seeds To Retire; New CEO Appointed
President Dick Chamberlin will be retiring after almost 29 years of continuous service to the company. Read More
potato field_featured
Crop Protection
May 6, 2016
Late Blight Hotlines For Potato Growers Up And Running For 2016 
Offering management information for five key potato-growing regions, the hotlines provide data on weather threats, disease pressures, and scouting. Read More
sweet corn field shot Gordon
Production
May 6, 2016
A Close Look At The GMO Breeding Process
A plant breeding expert details the scientific procedures required to bring GMO crops to market. Read More
Cosmis Crisp apples planting FEATURE
GenNext Growers
May 5, 2016
Washington Apple Education Foundation Awards $1 Million In Scholarships
Funded through donations, scholarship fund reaches major milestone. Read More
Downy Mildew in Spinach for web
Crop Protection
May 5, 2016
Michigan Researcher Leads $2.3-Million Grant To Combat Downy Mildew In Vegetables
Michigan State’s Mary Hausbeck along with a multistate research team will work to enhance and refine early detection methods, allowing growers to identify the pathogen’s presence in the air -- before it infects fields. Read More
The Latest
Crop Protection
May 6, 2016
Scouting Tips For Growing Grape Pest Thr…
Grape growers should be on the lookout this fall for two pests that are popping up in vineyards along the Read More
Disease Control
May 6, 2016
Avocado Tree Killer On The Move In Flori…
Devastating laurel wilt disease now reported in 61 of 67 counties. Read More
Farm Management
May 6, 2016
$8.7 Million Available In Risk Managemen…
The funds are for programs to help underserved growers and ranchers to learn how to manage long-term risks and challenges. Read More
Food Safety
May 6, 2016
Recall Expands For Frozen Fruit And Vege…
CRF Frozen Foods of Pasco, WA, is expanding its voluntary recall of frozen organic and traditional fruits and vegetables due to potential Listeria contamination. Read More
Potatoes
May 6, 2016
National Potato Council Summer Meeting S…
The summer meeting is open to growers and industry leaders interested in advancing the potato industry’s most pressing policy debates. Read More
Vegetables
May 6, 2016
Head Of Harris Seeds To Retire; New CEO …
President Dick Chamberlin will be retiring after almost 29 years of continuous service to the company. Read More
Crop Protection
May 6, 2016
Late Blight Hotlines For Potato Growers …
Offering management information for five key potato-growing regions, the hotlines provide data on weather threats, disease pressures, and scouting. Read More
Production
May 6, 2016
A Close Look At The GMO Breeding Process
A plant breeding expert details the scientific procedures required to bring GMO crops to market. Read More
GenNext Growers
May 5, 2016
Washington Apple Education Foundation Aw…
Funded through donations, scholarship fund reaches major milestone. Read More
Crop Protection
May 5, 2016
Michigan Researcher Leads $2.3-Million G…
Michigan State’s Mary Hausbeck along with a multistate research team will work to enhance and refine early detection methods, allowing growers to identify the pathogen’s presence in the air -- before it infects fields. Read More
Berries
May 4, 2016
University of California Sued Over Straw…
This time it’s a private strawberry breeding company, which says the statewide breeding program has been decimated by UC-Davis. Read More
Citrus
May 4, 2016
$6 Million Available For Antimicrobial R…
Program to develop strategies to reduce public health risks in food chain. Read More
Citrus
May 3, 2016
New Program Targets Abandoned California…
Bayer and California Citrus Mutual have teamed up to help protect the state’s commercial citrus industry from deadly citrus greening. Read More
Fruits
May 3, 2016
How To Build A Better Beetle Trap
UF/IFAS study on luring, snaring the notorious redbay ambrosia beetle earns prestigious award. Read More
Cucurbits
May 3, 2016
New Cantaloupe, Pumpkin, And Squash Seed…
New varieties developed at the University of New Hampshire are the result of the longest continuous squash and pumpkin breeding program in North America. Read More
Crop Protection
May 3, 2016
Almond Board To Hold Three Weed Manageme…
The second in a series of free educational workshops aimed at the state’s almond growers will cover weed identification and Read More
Food Safety
May 2, 2016
Department Of Justice Investigates Dole …
Salad mix packaged at the company’s Ohio facility has been connected to four deaths. Read More
Citrus
May 2, 2016
As Hurricane Season Nears, South Florida…
All basins around the region experience rainfall deficits during April as totals fall far below average. Read More
[gravityform id="62" title="false" description="false"]