Making Video Games for Robots to Play

When the founders of advanced.farm first endeavored to build robotic harvesters for fruit picking, they knew it was going to be hard. Designing and building a robot that can pick delicate fruit is difficult enough without the added challenges that come from an unstructured farm environment that is constantly changing based on climate, plant biology, and a myriad of other factors. This unstructured environment is a far cry from the tightly controlled factory floors where traditional automation tools were developed. Additionally, due to the seasonal nature of farming, you might have to wait an entire year before encountering a specific challenge again. At advanced.farm we use our software emulator to take control back from the chaos of the field and keep progressing agricultural automation. 

What is the tool?

The emulator is a simulation environment that was developed to test code and measure the performance of the software controlling our harvesters. This simulation interacts with the harvester code in almost exactly the same way as the real environment, and it looks real too! 

Our data team monitors changes in the field throughout the year and collects data about volume of fruit, numbers of leaves,stems, and other metrics as the plants grow and change. From this data, we procedurally generate the rows of plants. The harvester travels down the simulated rows running all the same code as our physical machine, except instead of getting images from a camera, we generate the images using the Unity game engine, a tool traditionally used for video game development. This tool can be run on an engineer’s laptop for development purposes or run on our bank of servers during the continuous integration test suite. The emulator is often our first step in the development process, and the last in the line of tests before code is deployed to the harvesters. 

Why do we need it?

When advanced.farm started work on an apple harvester early this year, the fruits were just beginning to bud in the Yakima Valley. The apples would not mature until late summer, meaning developing computer vision systems and picking algorithms in March would be nearly impossible without the emulator. By using the simulation environment, we were able to catch many issues ahead of time, even making major hardware changes because of this testing. 

Strawberries have their own challenges: as the plant develops over the growing season it becomes bushier and it increases its fruit production. This means our harvester needs to adapt to the changing conditions of the field to pick effectively; also there is no guarantee that software changes that are successful in May will show the same improvement in February. By generating a profile of the plants over the season, we are able to forecast our performance at any point in the year and make concrete decisions about what changes are the best on the whole. For example we can develop our technology with an emulator scene that is specific for Santa Maria in May with a specific variety of strawberry. We can also provide insight to growers how changes in operations or performance will affect their bottom line, all without touching a piece of hardware. 

Where to go from here? 

The emulator continues to solve more of our problems every day. It will be an invaluable tool for moving into new markets and it will continue to be the backbone of our testing framework as we scale our business. It’s hard to see a future where simulation doesn’t drive development in agricultural robotics and I am excited to be a part of that.

What is the tool?

The emulator is a simulation environment that was developed to test code and measure the performance of the software controlling our harvesters. This simulation interacts with the harvester code in almost exactly the same way as the real environment, and it looks real too!

By: Rayne Milner

Rayne is a Software Engineering Manager at Advanced Farm. He completed a BA in Physics at Cornell University with a concentration in Aerospace Engineering before studying Agricultural Robotics during graduate work at UC Davis.