Apple Harvesters Use Vision To “See” Fruit

Editor’s Note: This past October, Tony Koselka, Jillian Cannons, and Ryan Carlon of Vision Robotics Corporation (VRC) returned to Washington for further field trials of the Newton Scout, a vision-based system for automatic crop load assessment. The following report highlights their results.

Developed in part through the USDA-funded project Comprehensive Automation for Specialty Crops (CASC), the Newton Scout will revolutionize orchard operations by providing growers with a map that accurately maps fruit placement, fruit size, and number throughout the growing season. Fruit maps will be used to determine crop load, which in turn will give growers the information to develop harvest and marketing strategies. And this is just the beginning. Developing reliable vision systems that are robust enough for use in orchards opens up all kinds of possibilities in the area of pest and disease detection, plant health assessment, and the use of robotics at harvest.

The Newton Scout integrates stereo cameras with a custom image processing algorithm. The algorithms use classifiers to identify the apples, where the simplest classifier uses color and shape. For red apples, color is the primary classifier. For detecting green fruit, the Scout detects using statistical classifiers generated by boosting algorithms. The most challenging aspects of detecting fruit are occlusions, shadows, shape and color coincidences, and ambient lighting conditions such as bright sunlight. Once an apple is identified, the stereo cameras enable the system to determine its location and size.

The Vision Robotics field trial was conducted at Allan Brothers Othello Orchard in Washington. Prior to VRC’s arrival, Washington State University (WSU) Extension Educator Karen Lewis, and interns with the Washington Tree Fruit Research Commission, hand tagged, mapped, and sized fruit on several rows of fruit. This data allows for the groundtruthing or validation component of the test. Koselka took time out to speak about the development of a similar unit that counts oranges with the interns.

On one of the field trials for that unit, Koselka asked the grower to estimate his crop. The farm manager responded with 1,200 bushels. For this test, Vision Robotics scouted and counted 1,900 — a sizable difference and one that would have a large financial impact.

As Koselka stated, “not only will it [Scout] be able to accurately count a crop and locate each piece of fruit, it will ultimately feed into a robot harvester that will pick all the apples.” And if that was not enough, Koselka added, “Oh, and a pruner is not far off either!”

Even More Potential

The Newton Scout utility will be enhanced in the future with the addition of other possible technologies, including near infrared (NIR). As Newton evolves, it will have multiple uses throughout the fruit growing cycle — for example, assessment of sugar and acid content and internal damage. CASC is also making progress in other automation fronts that bridge with Vision Robotics’ work.

Carnegie Mellon University (CMU) researchers, led by Sanjiv Singh, are developing an Autonomous Prime Mover (APM) capable of towing the Newton Scout safely through entire orchard blocks. In the future, they will also automate an agricultural harvesting platform so that workers no longer need to climb on ladders to pick fruit. The APM’s multifunctionality, autonomous drive, and wireless networking could move it into the center of precision management of specialty crops.

The CASC team, led by CMU, includes engineers, scientists, Extension educators, growers, equipment manufacturers, and industry representatives. This plurality of expertise has led to synergistic endeavors that push the envelope in both the engineering and the science components of the project. For example, Ben Grocholsky at CMU is developing a geospatial system to map and locate each tree in an orchard; in cooperation with the plant science group, he was able to augment this information with local temperature and NDVI data that has the future possibility of being incorporated to detect plant stress and disease.

In another example of the synergy of the CASC team, during a field trial of an automated caliper measurement device, growers told the CASC team that a simpler tree counter would also be very useful. Singh’s group returned to CMU and produced a prototype tree counter ready to be tested in weeks; they will return to nurseries and test the counter this fall. The total time from conception to development will be less than four months, and if successful, this device could go into production within a year.

It will only be a matter of time before specialty crop growers also start using vehicles, sensors, devices, and software that automate farm operations, increase farm efficiency, and meet the ultimate goal of delivering to the consumer a consistently high quality affordable eating experience while returning money to the farming operation.

For more information on Comprehensive Automation for Specialty Crops, please visit www.cascrop.com.

Leave a Reply

Featured Stories
Apples & PearsHard Cider Organization Offers Accredited Program
September 1, 2015
United States Association of Cider Makers (USACM) introduces Sicera, the first-ever hard Cider Certification Program. This accreditation program is designed Read More
Apples & PearsProducts To Help Prevent Preharvest Drop In Apple Orchards
September 1, 2015
In the latest Extension bulletin from Penn State University, James Schupp associate professor of pomology suggests several products to apply Read More
Apples & PearsMarvin Sundquist Dies At 89
September 1, 2015
Marvin Sundquist, the retired owner of Sundquist Fruit and Cold Storage, died on Aug. 29 at the age of 89, Read More
Apples & PearsSyngenta Receives EPA Registration For Solatenol  Fungicide
September 1, 2015
The fungicide is available in four product offerings, pending state registrations. Read More
Grower Achievement AwardPero Family Farms Is The 2015 Grower Achievement Award Recipient
September 1, 2015
The farm family that runs this Delray Beach-FL based operation develops, produces, packages, and markets its fresh-cut products while continuing to think outside the box. Read More
Dennis P. Broadaway
CitrusFlorida Citrus Packers Bestows Association’s Highest Honor To One Of Its Own
September 1, 2015
For his dedication to the Sunshine State’s fresh citrus sector, Dennis Broadaway receives John T. Lesley Award for Excellence. Read More
Citrus Achievement AwardSaving Citrus A Perpetual Task
September 1, 2015
Orie Lee, the 2015 Citrus Achievement Award winner, talks about how growing citrus has changed in his lifetime and how it hasn't. Read More
Florida Ag Expo10th Annual Florida Ag Expo Field Tours To Tackle Production Challenges
September 1, 2015
There will be plenty to see and learn during the milestone event's exclusive field trials. Read More
Apples & PearsPennsylvania Congressman Supports CIDER Act
August 31, 2015
Freshman Congressman Ryan Costello (PA-06) spent a recent afternoon at Frecon Farms in Boyertown, PA, to tout the CIDER Act, Read More
Oriental Fruit fly
CitrusExotic Fruit Flies Invade South Florida
August 31, 2015
Efforts are under way to eradicate pest that attacks more than 230 different fruits, vegetables, and nuts. Read More
The Latest
Apples & PearsHard Cider Organization Offers Accredited Program
September 1, 2015
United States Association of Cider Makers (USACM) introduces Sicera, the first-ever hard Cider Certification Program. This accreditation program is designed Read More
Apples & PearsProducts To Help Prevent Preharvest Drop In Apple Orcha…
September 1, 2015
In the latest Extension bulletin from Penn State University, James Schupp associate professor of pomology suggests several products to apply Read More
Apples & PearsMarvin Sundquist Dies At 89
September 1, 2015
Marvin Sundquist, the retired owner of Sundquist Fruit and Cold Storage, died on Aug. 29 at the age of 89, Read More
Apples & PearsSyngenta Receives EPA Registration For Solatenol  Fungi…
September 1, 2015
The fungicide is available in four product offerings, pending state registrations. Read More
Grower Achievement AwardPero Family Farms Is The 2015 Grower Achievement Award …
September 1, 2015
The farm family that runs this Delray Beach-FL based operation develops, produces, packages, and markets its fresh-cut products while continuing to think outside the box. Read More
Dennis P. Broadaway
CitrusFlorida Citrus Packers Bestows Association’s Highest Ho…
September 1, 2015
For his dedication to the Sunshine State’s fresh citrus sector, Dennis Broadaway receives John T. Lesley Award for Excellence. Read More
Citrus Achievement AwardSaving Citrus A Perpetual Task
September 1, 2015
Orie Lee, the 2015 Citrus Achievement Award winner, talks about how growing citrus has changed in his lifetime and how it hasn't. Read More
Florida Ag Expo10th Annual Florida Ag Expo Field Tours To Tackle Produ…
September 1, 2015
There will be plenty to see and learn during the milestone event's exclusive field trials. Read More
Apples & PearsPennsylvania Congressman Supports CIDER Act
August 31, 2015
Freshman Congressman Ryan Costello (PA-06) spent a recent afternoon at Frecon Farms in Boyertown, PA, to tout the CIDER Act, Read More
Oriental Fruit fly
CitrusExotic Fruit Flies Invade South Florida
August 31, 2015
Efforts are under way to eradicate pest that attacks more than 230 different fruits, vegetables, and nuts. Read More
Business PlanningBe Food Safety Recall-Ready [Opinion]
August 31, 2015
No one ever wants to make someone sick after consuming fresh produce, but in spite of the protocols in place, recallls happen and you must be prepared. Read More
Crop ProtectionSustainable Options For Controlling Brown Marmorated St…
August 31, 2015
A researcher from Rutgers University offers pointers to help you control this pest of fruits and vegetables. Read More
CitrusNominations Open For Next Class Of Florida Citrus Hall …
August 31, 2015
Inductees to be honored during the 54th annual Citrus Celebration Luncheon. Read More
CitrusWhole-Farm Revenue Protection Insurance To Be Available…
August 31, 2015
USDA offers greater flexibility for beginning, organic and fruit and vegetable growers. Read More
Cucurbits8 Must-Have Melons
August 31, 2015
This month’s variety specs feature focuses on the latest cantaloupe and melon varieties. Some of the features highlighted include disease Read More
drought management; irrigation; water management
CitrusClear Up Misconceptions About Water Use In Ag
August 31, 2015
Those in the industry need to to clear up misconceptions about water use in agriculture. Read More
FruitsNew Airless Tire For Center Pivots
August 31, 2015
Aperture design provides flexibility over ridges to help prevent overloading of the drive train. Read More
Stone FruitHow Canopy Structures Can Affect Sweet Cherry Productiv…
August 31, 2015
We have previously explored orchard cover technologies to modify climatic risks in sweet cherry production, such as rain-cracking and spring Read More