Apple Harvesters Use Vision To “See” Fruit

Editor’s Note: This past October, Tony Koselka, Jillian Cannons, and Ryan Carlon of Vision Robotics Corporation (VRC) returned to Washington for further field trials of the Newton Scout, a vision-based system for automatic crop load assessment. The following report highlights their results.

Developed in part through the USDA-funded project Comprehensive Automation for Specialty Crops (CASC), the Newton Scout will revolutionize orchard operations by providing growers with a map that accurately maps fruit placement, fruit size, and number throughout the growing season. Fruit maps will be used to determine crop load, which in turn will give growers the information to develop harvest and marketing strategies. And this is just the beginning. Developing reliable vision systems that are robust enough for use in orchards opens up all kinds of possibilities in the area of pest and disease detection, plant health assessment, and the use of robotics at harvest.

The Newton Scout integrates stereo cameras with a custom image processing algorithm. The algorithms use classifiers to identify the apples, where the simplest classifier uses color and shape. For red apples, color is the primary classifier. For detecting green fruit, the Scout detects using statistical classifiers generated by boosting algorithms. The most challenging aspects of detecting fruit are occlusions, shadows, shape and color coincidences, and ambient lighting conditions such as bright sunlight. Once an apple is identified, the stereo cameras enable the system to determine its location and size.

The Vision Robotics field trial was conducted at Allan Brothers Othello Orchard in Washington. Prior to VRC’s arrival, Washington State University (WSU) Extension Educator Karen Lewis, and interns with the Washington Tree Fruit Research Commission, hand tagged, mapped, and sized fruit on several rows of fruit. This data allows for the groundtruthing or validation component of the test. Koselka took time out to speak about the development of a similar unit that counts oranges with the interns.

On one of the field trials for that unit, Koselka asked the grower to estimate his crop. The farm manager responded with 1,200 bushels. For this test, Vision Robotics scouted and counted 1,900 — a sizable difference and one that would have a large financial impact.

As Koselka stated, “not only will it [Scout] be able to accurately count a crop and locate each piece of fruit, it will ultimately feed into a robot harvester that will pick all the apples.” And if that was not enough, Koselka added, “Oh, and a pruner is not far off either!”

Even More Potential

The Newton Scout utility will be enhanced in the future with the addition of other possible technologies, including near infrared (NIR). As Newton evolves, it will have multiple uses throughout the fruit growing cycle — for example, assessment of sugar and acid content and internal damage. CASC is also making progress in other automation fronts that bridge with Vision Robotics’ work.

Carnegie Mellon University (CMU) researchers, led by Sanjiv Singh, are developing an Autonomous Prime Mover (APM) capable of towing the Newton Scout safely through entire orchard blocks. In the future, they will also automate an agricultural harvesting platform so that workers no longer need to climb on ladders to pick fruit. The APM’s multifunctionality, autonomous drive, and wireless networking could move it into the center of precision management of specialty crops.

The CASC team, led by CMU, includes engineers, scientists, Extension educators, growers, equipment manufacturers, and industry representatives. This plurality of expertise has led to synergistic endeavors that push the envelope in both the engineering and the science components of the project. For example, Ben Grocholsky at CMU is developing a geospatial system to map and locate each tree in an orchard; in cooperation with the plant science group, he was able to augment this information with local temperature and NDVI data that has the future possibility of being incorporated to detect plant stress and disease.

In another example of the synergy of the CASC team, during a field trial of an automated caliper measurement device, growers told the CASC team that a simpler tree counter would also be very useful. Singh’s group returned to CMU and produced a prototype tree counter ready to be tested in weeks; they will return to nurseries and test the counter this fall. The total time from conception to development will be less than four months, and if successful, this device could go into production within a year.

It will only be a matter of time before specialty crop growers also start using vehicles, sensors, devices, and software that automate farm operations, increase farm efficiency, and meet the ultimate goal of delivering to the consumer a consistently high quality affordable eating experience while returning money to the farming operation.

For more information on Comprehensive Automation for Specialty Crops, please visit www.cascrop.com.

Leave a Reply

Featured Stories
spotted-lantern-flay-free-use-feature
Farm Marketing
September 30, 2016
Insect Issues Making The News In September
A number of stories have hit the news this month, from trying to control spotted lantern fly in Pennsylvania to calling researchers relying on mud daubers to go where they can't in a search for an invasive spider. Read More
Long lunch line at 2015 Florida Ag Expo
Florida Ag Expo
September 30, 2016
The Florida Ag Expo Serves Growers [Opinion]
This annual industry gathering is such an important event, Jack Payne, Senior VP of UF/IFAS, has invited his entire leadership team — including the directors of all 12 UF/IFAS research centers and 14 academic department chairs — to participate. Read More
plate-of-the-union-screen-cap-feature
Farm Marketing
September 29, 2016
Coalition To Presidential Candidates: Reform America’s Food System
The coalition Plate Of The Union would like local farms to play a more central roll in U.S. food policy. Read More
A thorough assessment of nematode populations by conducting soil sampling and laboratory examination is essential before deciding on soil fumigation.  (Photo credit:  Almond Board of California)
Nuts
September 29, 2016
Use Soil Sampling To Inform Your Almond Orchard Fumigation Decisions
Whether planting on new ground or replanting, sizing up the enemy — nematodes — is a must. Read More
The team at Sawmill Creek Vineyards which includes Tina Hazlitt, Jason Hazlitt, Ryan Yaskulski, Courtney Griffin, and Eric Hazlitt take great pride in their vineyard, knowing their customers have come to depend the vineyard’s attention to detail. (Photo credit: Christina Herrick)
Grapes
September 29, 2016
Vinifera Finds A Home In The Finger Lakes
In a region known for spectacular sights, varieties are making their mark, just ask the team at Sawmill Creek Vineyards. Read More
Accuweather Winter 2016-2017 forecast map
Citrus
September 29, 2016
Winter Is Coming, And It Could Get Wicked
Forecast sees heightened freeze threat for Florida citrus; frequent snow to blast Northeast. Read More
Spring Valley Farms in Umatilla, FL, from a drone
Citrus
September 29, 2016
Celebrate National Farmer’s Day Every Day
Florida agriculture has a compelling story to tell every day, but what better time to tell it than National Farmer’s Day? Read More
Photo credit: Christina Herrick
Stone Fruit
September 28, 2016
New Research Gets To The Bottom Of Cherry Cracking
German researchers hope to identify root causes of rain-induced problem. Read More
Part of a sustainable cropping system includes desiging fields to wrap around hilly terrain to limit runoff and soil erosion.  Photo credit: Rosemary Gordon
Farm Management
September 28, 2016
Grants Available For Sustainable Ag In The West
More than $1 million is marked for grants to study, demonstrate, and improve sustainable ag practices. Read More
The Latest
Farm Marketing
September 30, 2016
Insect Issues Making The News In Septemb…
A number of stories have hit the news this month, from trying to control spotted lantern fly in Pennsylvania to calling researchers relying on mud daubers to go where they can't in a search for an invasive spider. Read More
Florida Ag Expo
September 30, 2016
The Florida Ag Expo Serves Growers [Opin…
This annual industry gathering is such an important event, Jack Payne, Senior VP of UF/IFAS, has invited his entire leadership team — including the directors of all 12 UF/IFAS research centers and 14 academic department chairs — to participate. Read More
Farm Marketing
September 29, 2016
Kids Up For Adoption Get A Boost From He…
The Heartland Maze has gone the extra mile and created a maze design that promotes a cause. Together with a Read More
Farm Marketing
September 29, 2016
Coalition To Presidential Candidates: Re…
The coalition Plate Of The Union would like local farms to play a more central roll in U.S. food policy. Read More
Nuts
September 29, 2016
Use Soil Sampling To Inform Your Almond …
Whether planting on new ground or replanting, sizing up the enemy — nematodes — is a must. Read More
Grapes
September 29, 2016
Vinifera Finds A Home In The Finger Lake…
In a region known for spectacular sights, varieties are making their mark, just ask the team at Sawmill Creek Vineyards. Read More
Citrus
September 29, 2016
Winter Is Coming, And It Could Get Wicke…
Forecast sees heightened freeze threat for Florida citrus; frequent snow to blast Northeast. Read More
Citrus
September 29, 2016
Celebrate National Farmer’s Day Ev…
Florida agriculture has a compelling story to tell every day, but what better time to tell it than National Farmer’s Day? Read More
Stone Fruit
September 28, 2016
New Research Gets To The Bottom Of Cherr…
German researchers hope to identify root causes of rain-induced problem. Read More
Farm Management
September 28, 2016
Grants Available For Sustainable Ag In T…
More than $1 million is marked for grants to study, demonstrate, and improve sustainable ag practices. Read More
Equipment
September 28, 2016
Small Tractor Sales On The Rebound
Retail sales of tractors under 40 horsepower gained more than 21% last month, while sales of larger tractors continued their double-digit declines. Read More
Farm Management
September 28, 2016
The Time For Climate Change Denial Is Ov…
Extreme cold, droughts, flooding, record-setting temperatures have all been a part of this year’s growing season. Read More
Fruits
September 27, 2016
Focusing On Phenomics In Specialty Crops
It’s not just the nature of your crops — the genomics — it’s also how they are nurtured. Read More
Grower Achievement Award
September 27, 2016
Tasteful Selections Is The Recipient Of …
The award was presented to Bob Bender of Tasteful Selections Sept. 13 at United Fresh's Washington Conference. Read More
Fruits
September 27, 2016
Rodale Institute Launches An Organic Far…
A new national organization for organic growers is designed to provide advocacy, the latest research and education, and information-sharing among farmers. Read More
Farm Management
September 27, 2016
Politicians May Be Taking Ag For Granted…
Growers have become masters at ensuring a steady food supply, but warning, politicians: Ignore them at your peril. Read More
Citrus
September 26, 2016
Presidential Candidates Comment On Agric…
Hillary Clinton and Donald Trump weigh in on a number of topics of concern to growers. Read More
Apples & Pears
September 26, 2016
New Apple Variety Is One Stemilt Hopes C…
University of Minnesota apple variety will be grown and packed exclusively by Stemilt. Read More
[gravityform id="62" title="false" description="false"]