Harvesting happens outdoors, and the target sits in a different place and a different shape every time. Almost nobody has automated it with a humanoid, and there is little prior work to copy. We ran this PoC to find out how far the current stack actually gets under those conditions.
It got to the point where the robot finds an okra pod in an open field, reaches for it and grasps it. In an indoor test setup the grasp succeeded 16 times out of 20. Development took 70 days. We ran out of time while building the cutter, so the full harvest cycle is not closed yet.
What we put together
The robot is a Unitree G1. A ZED-M stereo camera is fixed to its chest to find the pods, the right hand carries a gripper and a mount for the cutter, and a basket for the harvested pods hangs off the left arm. A Jetson Orin and a battery ride on its back, so all inference runs on the machine itself.
The harvest is defined as a workflow on the robot OS, and one cycle is split into these eight steps.
| Step | What happens | How |
|---|---|---|
| Walk | Move sideways along the row, looking for pods | Velocity commands to the walking policy (LocoClient) |
| Detect | Find and segment okra pods in the camera image | YOLO fine-tuned on okra images |
| Locate | Turn the detection into a point cloud and take its centroid | Depth from the ZED-M stereo camera |
| Transform | Convert camera coordinates into right-arm coordinates | Geometry |
| Reach | Bring the hand close to the pod | Inverse kinematics (IK) |
| Align | Close the last centimeters until the cutter straddles the stem | Diffusion Policy |
| Cut and grasp | Change the hand aperture to cut the stem and hold the pod | Aperture control |
| Stow | Drop the pod in the basket on the left arm and move to the next plant | Fixed motion |
Decide what the model is allowed to do, first
Of those eight steps, a learned policy drives exactly one: the alignment. Walking, detection, coordinate math and the reach itself are all classical control and geometry.
YOLO, a stereo camera and inverse kinematics are enough to put the hand near the pod. That path is cheap to compute, fast to execute, and needs no training data at all. What needs a model is the last few centimeters, where the plant moves in the wind, the contrast shifts with the sun and a leaf hides the pod — the part you cannot write down in advance.
Every step you hand to a model adds uncertainty, and with it more data to collect and more failure modes to debug. Fix the sequence of motions first, then carve out only the part that genuinely requires a policy. That line is what sets your development time.
Keep the training data small enough to inspect
When a policy does not behave, the candidate causes are the data quality, the amount of data, or a change between collection and execution. With conventional control you read the code to understand a motion; with a learned policy the code tells you nothing. Checking what it was trained on is effectively the only debugging method you have.
That turns data reviewability into a design constraint. We cut episodes at two seconds. Five hundred episodes is then 1,000 seconds of footage, which a person can actually go through. Keep the same 500 episodes but make each one 60 seconds and you have thirty times the footage to inspect. The longer the task you hand over, the faster debugging cost grows.
The cut is excluded from the policy for the same reason. Training the cut would mean dropping more than 500 pods to collect it. Splitting the repeatable alignment from the deterministic cut gets far more usable data out of the same field.
Small decisions that mattered
- Record actions as relative motion. “Where to move from here” does not depend on absolute position, so data collected elsewhere stays usable
- Use a fisheye lens on the gripper camera. With a normal lens the target fills the frame and none of the surrounding context is captured
- Mix your collection methods. Moving the robot directly gives precise data but costs hours; filming a human hand trajectory is less precise and more than three times faster
- Before going to the robot, 3D-scan the field and the indoor rig, load them into simulation and run the whole flow there
What outdoors adds to the problem
Outdoors is not only harder to perceive; it constrains the machine. Bipedal walking spends power on staying upright and moving, and the G1 runs about two hours on a charge — less on a day with a lot of walking. On top of that, a lower body that sways while the hand closes moves the camera with it, and the grasp success rate drops.
For work like harvesting, which means covering ground outdoors for hours, a semi-humanoid on wheels fits the conditions better: roughly six hours of battery and a steadier upper body. Legs earn their keep somewhere else — crouching into the inside of a car body on a factory line, for instance.
What the PoC was really for
The goal was never okra alone. Detect the target, carry the arm most of the way with classical control, close the last centimeters with a learned policy: that structure is not specific to harvesting. It transfers to factory and construction tasks in the same shape. Building a software base that carries to the next site is what this PoC was actually for.
Orboh puts engineers on site, narrows the task, fixes the workflow and implements the robot. The okra harvest is that method tried in the least forgiving place we could find.
With
- Toyota Auto Body Research
- Kagoshima Prefectural Agricultural Development Support Center
- Interns and faculty from Kyushu Institute of Technology