End-to-End Coconut Harvesting Robot
Teguh Satrio Wibowo
∗, Indra Adji Sulistijono
†and Anhar Risnumawan
‡ Mechatronics Engineering Division†Graduate School of Engineering Technology
Politeknik Elektronika Negeri Surabaya (PENS)
Kampus PENS, Jalan Raya ITS Sukolilo, Surabaya, 60111 Indonesia Tel: +62-31-594 7280; Fax: +62-31-594 6114
Email: ∗teguhsatrio@me.student.pens.ac.id,{†indra, ‡anhar}@pens.ac.id
Abstract—World coconuts production increasingly demands a huge number of raw coconuts to be cultivated. This is starting as everyday needs, foods, traditional cuisines, to many industries which depend on raw coconuts. While coconut production in Indonesia has only contributed to 27%, yet Indonesia has the biggest coconut area in the world. This is because coconuts are often cultivated conventionally, climbing by men. The condition is worsen as the number of men climbing coconut trees has increasingly decreased and industries requires to reduce cost production. In this paper, we develop an end-to-end autonomous coconut-harvesting-robot for easy operation and portable to carry along in large plantations. First, a novel climbing and fastening mechanism to climb a tree with varying diameter is introduced. Second, position of a coconut is then detected using steady-state genetic algorithm followed by sending a command to harvest the detected coconut using an extendable arm equipped with a specifically designed saw. Experiment results show the ability of the robot to harvest coconuts.
I. INTRODUCTION
World coconuts production is 64,897 million nuts and Indonesia is one of the leading coconuts production after India and Philippines. Indonesia contributes to 27% of coconuts production and has the biggest coconuts area in the world. From 2000 to 2011 the area is around 3.69 to 3.91 million hectares. Although it is the biggest coconuts area in the world, Indonesia only produces coconuts around 3.04 to 3.25 million tons. As compared to India and Philippines, which have area of 2 and 3.8 million hectares respectively and able to contribute around 90% of coconut production, Indonesia is considered lower in coconut production which is around 0.8 tons per hectare. Farmers of marginal income hold roughly 90% of coconut holding in Indonesia. Moreover, farmers often utilize coconut trees as everyday needs for which each and every part of the tree is used by people. From coconuts and trees what can be used to produce copra, oil, milk, chips, milk powder, toddy, refined sugar, palm sugar, flower syrup, jam, vinegar, jelly, tender coconut water, fuel for traditional cooking, etc. Maximizing the abundant amount of coconuts production is therefore extremely important in Indonesia.
Coconut harvesting is traditionally carried out by men who belong to the economically lower class of society. As the number of literacy rate increases and awareness about the high paid job opportunities, the number of men climbing coconut trees for harvesting coconut has decreased increasingly. As these men usually do not have any insurance coverage, any accidents while climbing the trees would affect the entire family. In many such households, people suffer a lot because
Camera
Wireless Communication
Camera View
Autonomous
Coconut Harvesting Robot
Fig. 1. Proposed coconut harvesting robot. The robot is autonomously searching for a coconut reaching the top of the tree harvesting coconut using a specifically designed saw. User can view its action using mobile phone screen.
this would be the only source of income for their livelihood. In addition, the people who fall during climbing greater heights are bedridden for the rest of their life.
The condition is worsen by several industries, which de-pend on raw coconuts, for its processing and manufacturing of various value-added products. The inflorescence sap of coconut palms used to produce palm nectar, desiccated coconut and its powder, packed coconut milk, coconut cream, coconut milk powder, tender coconut water, vinegar, nata-de-coco, etc. are few of them. The growing market for all coconut products is increasing especially in Indonesia that often uses coconut in their cuisine. Hence, any impact in coconut cultivation sector will adversely affect these industries and their markets. Thousands of people get employment from these industries. One of the main threats of this industry is the increasing cost of production and the decrease in the labour resource.
system is highly important for successful harvesting in chal-lenging conditions. In this paper, an end-to-end autonomous coconut harvesting robot system starting from mechanical climbing and harvesting designs to vision-based coconut de-tection and harvesting is introduced, as shown in Fig. 1. The robot is portable to carry along in large plantations, easy to operate, and equipped with a novel climbing and auto-fastening mechanism to climb a tree with varying diameter. The robot has hexagonal main body, two degrees of freedom arm, grinstone and camera on top of the arm, three wheels and one spring in every wheel. The position of a coconut is then detected using steady-state genetic algorithm followed by sending a command to harvest the detected coconut using extendable arm equipped with a specifically designed saw.
The rest of this paper is organized as follows. Section II describes related work. In Section III explains the methodology and it consists of III-A the robot mechanism and control and III-B describes the coconut detection algorithm. IV and V describe experiments and conclusion, respectively.
II. RELATEDWORK
Conventional harvesting method on reaching the top of a coconut tree, to test its maturity the climber taps the nut in the lowermost bunch with its harvesting knife. If the coconut is mature, the bunch at the base of the stalk is cut. When it drops to the ground, the climber also cleans the crown and removes the dry leaves, sheaths and spathes. For small trees, cutting the coconut bunches with a knife, attached to a long bamboo pole will do. Climbing the coconut palm in order to stay beneath the leaf crown to harvest fruits and cut old leaves seems quite simple. But it proves to be much more challenging to climb up the coconut leaf crown in order to reach the young inflorescences for making controlled pollinations with bagging for research purposes, and for harvesting the toddy (the sweet sap from coconut inflorescences), which serves to produce sugar, vinegar, wine and spirits. This motivated us to propose robotic harvesting method, where it can peep and reach out to any remote nook and corner at the top of the tree.
There is a high risk associated with coconut tree climbers because accidents can lead to severe physical damage, some of which may even be fatal. On the other side there is significant increase in price of coconut and coconut products. To tackle all these problems, the process of coconut harvesting automaticallyd reduce the need of human labour and also accident involved in the process. The robotic arm plays a crucial role in making the robotic coconut tree climber a successful machine. A robust and proper robotic arm would also help in balancing the body of such a robotic climber.
Previous works on robotic crop harvesting, for segmenting and detecting crops, have been explored by several researchers [2], [6]–[11]. The grape detector of Nuske et al. [7], [8] was initially proposed and one of the earliest crop detection systems. A radial symmetry transform was used to detect grapes in an image and then used this information to perform accurate yield estimation. Human head tracking was proposed by [12], but the problem is different from coconut detection. A limitation of their system was that they could not detect partially occluded grapes (crops). However, as they were performing yield estimation and not accurate crop detection they were able to cope with this limitation.
The problem of apple detection has been explored by Wang et al. [10] so that they could perform yield prediction. The method was based on color and distinctive specular reflection pattern. Further information, such as the the average size of apples, was used to either remove erroneous detections or to split regions that could contain multiple apples. Another heuristic employed was to accept as detections only those regions which were mostly round.
Segmentation approach for sweet peppers has been pro-posed by Bac et al. [11] and Hung et al. [6] utilized conditional random fields (CRFs) for almond segmentation. Bac et al. aimed to develop a robotic harvesting system and introduced a class segmentation approach in order to build an accurate obstacle map. They used a six band multi-spectral camera (with bandwidths of between 40−60nm2) and used a range
of features including the raw multi-spectral data, normalized difference indeces, as well as entropy based texture features. Experiments in a highly controlled glasshouse environment showed that this approach produced reasonably accurate seg-mentation results, however, the authors noted that it was not accurate enough to build a reliable obstacle map.
On the other hand, Hung et al. developed an almond segmentation approach in order to perform yield estimation. They proposed a class segmentation approach which learnt features using a sparse auto-encoder. These features were then used within a CRF framework and was shown to outperform previous work. They achieved impressive segmentation perfor-mance, but did not perform object detection. Furthermore, they noted that occlusion presented a major challenge.
More recently, tomato detection has been proposed by Yamamoto et al. [9] by first performing class segmentation. Color and shape features were used to train a classifier and regression trees classifier. This produced a segmentation map and grouped connected pixels into regions. Each region was declared to be a detection and to reduce the number of false alarms they trained a non-fruit classifier using a random forest.
The aforementioned works only focus on a certain part mostly on fruits detection neglecting the fact that robot mech-anism is also important for successful harvesting in challenging conditions. Several approaches have addressed only the crop segmentation task [6], [11] and not detection. Most of the work that has examined crop detection has predominantly been developed for yield estimation [7], [10] and so accu-rate detection was not necessary. The limited work that has examined accurate crop detection has done so for crops in controlled glasshouse environments [9]. As such the issue of crop detection in highly challenging conditions remains unsolved.
III. METHODOLOGY
Fig. 2. CAD final assembly of coconut harvesting robot. Note that it has three auto-fastening sliders to adjust with varying tree diameter.
A. Hardware
The proposed coconut harvesting robot comprises of main body and robotic arm, as shown in Fig. 2. The body shape of a hexagon made of hollow aluminium material with a width of 2.5 x 2.5 cm and a length of 38 cm per side. For portability and ease of use, the robot is equipped with a clip lock on one side of the robot frame that serves as a manual lock and can be opened or closed to be inserted onto the tree trunk. To deal with varying tree trunk diameter, the robot has an auto-fastening mechanism using three wheels connected with springs on all three sides so that the robot can provide better grip wrapped around a tree trunk.
1) Robot main body: The wheel dimension is 8.5cm of diameter, 21cm wide, and made of durable solid sponge. Se-lection of this sponge as the wheel surface is highly important as it has the characteristic to fully attach on the tree surface. Each wheel is mounted on the robot base frame with 60 degree orientation facing to the other wheels to form an equilateral triangle when viewed from above.
1
2
3
Fig. 3. Auto-fastening mechanism to grab varying tree trunk diameter. (1) Spring, (2) Slider, and (3) Durable sponge-wrapped wheel.
Each wheel has an auto-fastening mechanism using a spring which serves as a suspension as well, as shown in Fig. 3. This suspension mechanism is important as to provide better stability of the viewing camera for later processing of coconut detection. The spring which is mounted on the rod parallel
to the pipe and slider, operated by pushing the slider to the inside so as to sufficiently grab the trunk and adjust to varying tree trunk diameter. The length of the spring for the maximum pressure is 5 cm, whereas under unpressed conditions is 15 cm, this is because of the limited expansion length of a slider.
Slider maintains the position of the wheels to be remain perpendicular to the supporting side of the robot body and also as a way of translational movement mechanism when the wheel adjust to the tree diameter. Slider is designed with stainless steel material and has expansion capabilities up to 10 cm so that when all of the sliders fully suppressed, tree trunks diameter can reach up to 40 cm. Slider is mounted on the side of the robot body with connections made of aluminum shaped elbows using bolts. Fig. 4 shows final assembly of the real robot.
Fig. 4. Final assembly realization of coconut harvesting robot.
2) Robot arm: The arm is made of aluminium with a length of 170 cm. The arm can move up and down using DC motor by a pulley. The DC motor rotation is converted into linear motion using a timing-belt. The arm linear movement is designed to be able to reach the coconuts. At the top arm, a specifically designed saw and wireless camera are attached. The saw is designed using brushless DC motors for high speed rotation in order to quickly cut the coconut stem. The blade is a sharp grinding for cutting.
Central Processing Accelerometer Sensor
IP Camera
Driver Motor
Coconut Harvesting Robot
Fig. 5. Electronic circuitry of the robot.
3) Electronics control: The main control of the robot as shown in Fig. 5 is using the Arduino Mega 2560. While for image processing side a sophisticated Raspberry Pi 2 is used for computation, portable, and OS-based platform containing many open libraries such as the well-known image processing OpenCV library.
as a receiver (receiver). Data will be read and recognized as a character ASCII (American Standard Code for Information In-terchange). Data received by the bluetooth will be re-processed to be converted into movement of the robot in combination with data from the balance sensor.
B. Software
1) Genetic Algorithm: The Genetic algorithms (GAs) [13] are stochastic global search method that imitates the evolution of natural biological process. GAs operate on a population of candidate solutions applying the fundamental of natural genetics and survival of the best, to converge to a better approximations of solution. At each generation, a new set of approximations is created by the process of selecting individuals according to their level of fitness in the problem domain and breeding them together using operators borrowed from natural genetics. Thus, These iterations leads to the evolution of populations of individuals that are better suited to their environment than the individuals that they were created from, just as in natural adaptation. Due to the GAs extensive emulation of nature, biological terms have been borrowed and deserve cursory explanation. Accordingly, they are contrasted by demonstrating their role both in nature, as well as the algorithm.
Each individual is called chromosomes. Each basic unit of a chromosome is called a gene. A chromosome can be thought of as a string or a vector where each of its components is a symbol from a possible set of values. The set of vectors collectively make up the population. The main idea lying behind the GA is the reproduction of a population. Because of the reproduction, a new population is formed. Although many variations are possible and available, the main parts of reproduction, analogous to biological reproduction, are selection, crossover, and mutation.
Selection is an essential part of the natural evolution process and hence is an essential part of GA. The most com-monly known selection element is roulette-wheel selection. In roulette-wheel method, the candidate individuals are given a slice of the wheel proportional to their assigned probabilities obtained by scaling. Then the wheel is rotatedN times where N is the number of individuals or parents needed for repro-duction. Thus, the best individuals have a higher probability of being chosen. However, undesired solutions also have a small chance of survival. More computation time effective way to select individuals from the wheel is to create one random number and from that point continue in increments of equal size (1/N). Another alternative for selection is tournament in which chromosomes are selected randomly and the best is chosen among the them. This procedure is again repeated N times. There can also be variations of this tournament where the best string does not always win but wins with a chance of certain threshold.
Crossover of a single point is one of the most basic one. The position of the crossover is determined randomly. Then the determined length of the strings is exchanged between the mating pair. Variations of this crossover exist. Scattered crossover where a vector the same size of the chromosome is created containing zeros and ones. The new child obtains the genes of the first parent whenever there is a one on the vector
and gets the genes of the second parent whenever there is a zero on the vector. Two-point-crossover is another one where two positions for crossover is selected.
Mutation operator is basically randomly changing the value of a gene. For real valued genes, this change could be done by selecting a number uniformly from the range of possible values. One other possible way is to use a distribution like Gaussian with mean zero and a given variance where the number generated by this distribution is added to the value of the gene. It may be adjusted such that the variance decreases as the algorithm iterates. In our analysis for Gaussian mutation, the variance is chosen to be 1 and shrinks linearly to zero with each generation.
At the beginning of the computation a number of individ-uals (the population) are randomly initialized. The objective function is evaluated for these individuals. The first / initial generation is produced. If the optimization criteria does not meet the creation of a new generation starts. Individuals are selected according to their fitness for the production of offspring. Parents are recombined to produce offspring. All offspring will be mutated with a certain probability. The fitness of the offspring is then computed. The offspring are inserted into the population to replace the parents, produce a new generation. This cycle is performed until the optimization criteria are reached.
2) Steady State Genetic Algorithm: The steady-state ge-netic algorithm [14] uses overlapping populations. In each generation, a portion of the population is replaced by the newly generated individuals. At one extreme, only one or two individuals may be replaced each generation (close to 100% overlap). At the other extreme, the steady-state algorithm becomes a simple genetic algorithm when the entire population is replaced (0% overlap).
Since the algorithm only replaces a portion of the pop-ulation of each generation, the best individuals are more likely to be selected and the population quickly converges to a single individual. As a result, the steady-state algorithm often converges prematurely to a suboptimal solution. Once again, the crossover and mutation operators are key to the algorithm performance; a crossover operator that generates children unlike their parents and/or a high mutation rate can delay the convergence.
Fitness function is computed using the information from circle Hough transform [15]. Unlike the linear Hough trans-form [16], the circle Hough transtrans-form relies on circle equa-tions. The circle equation is,
r2= (x−a)2+ (y−b)2 (1)
where a and b represent the center coordinates, and r is the circle radius. The parametric representation of this circle is,
x=a+r∗cos(θ)
y=b+r∗sin(θ) (2)
computation time and memory for storage, increasing the com-plexity of extracting information from our image. In practice, we perform Canny edge detector on HSV color domain and then we code the circle Hough transform by setting the radius to a predefined value. For each edge pixel, a circle is drawn with that point as origin and radiusr. An array 3-dimensional with the first two dimensions representing the coordinates of the circle is used and the last third specifying the radius. The values in the array are increased every time a circle is drawn with the desired radius over every edge pixel. The array, which count of how many circles pass through coordinates of each edge pixel, proceeds to a score to find the highest count. The coordinates of the center of the circles in the images are the coordinates with the highest count.
IV. EXPERIMENTS
Experiments are conducted in coconut plantation Blitar, East Java, Indonesia. We select this place as it is nearby Indian Ocean and it has highest coconut production in East Java. Setup of coconut harvesting robot is shown in Fig. 6.
Fig. 6. Setup of coconut harvesting robot in coconut plantation Blitar, East Java, Indonesia.
A bluetooth communication is selected as a wireless medium because it is still in range of the data sender and an average peak coconut trees, as shown in table I. Moreover it is easier for hardware setup.
TABLE I. MAXIMUM DISTANCE OF TRANSCEIVING DATA
Transmitter Receiver Maximum Distance
HC-05 HC-06 30 meter
Android HC-06 25-30 meter
Wireless Wireless Receiver 10 meter
The robot showed relatively well to move up and down on a tree, auto-fastening using the spring has shown quite well in reducing and adapting to varying coconut tree trunk diameter, so that the robot could climb coconut trees quickly. The average height of a coconut tree is approximately 20 meters, so that the robot maximum speed to climb-up above the palm tree is within 22 seconds, from table II.
TABLE II. ROBOT WHEEL ROTATION SPEED
Wheel diameter 80mm
PWM RPM RPS cm/s
128 102 1.7 42.7
160 130 2.17 54.43
192 159 2.65 66.57
224 188 3.13 78.71
255 215 3.58 90.01
Accelerometer sensor influences the rotation of the wheel motor in order to maintain the robot movement to be balanced and not skewed. The experimental result is conducted so as to roughly balance the robot due to high noise which could be attributed by limited space of sensors placement. Testing data on accelerometer can be seen in table III. Data sensor processing was performed using a common low pass filter in order to remove high frequency noise due to undesired vibration.
TABLE III. ACCELEROMETER(MMA7361)DATA AGAINT ROBOT POSITION
Position x y z
Normal 450 300 245
Wheel-1 330
Wheel-2 260 238
Wheel-3 270
Wheel-3-1 330 255
Wheel-1-2 230
Wheel-2-3 270
At trial cut coconuts robot body was hit by coconuts that weighs about 10kg, but the robot does not fall down when exposed to loads of coconuts, one of them due to the mechanical spring system that has worked well for pressing coconut trees. At the cutting blade mechanical systems, the selection of blades and motors used influence on whether or not cut stems coconuts and long- or cutting process. In experiments using fine grinding blades, the blades can cut stems coconuts, but the cutting process takes a long time.
Random initialization of steady state GA is shown in Fig. 8 and coconut detection result is shown in Fig. 9. From these observations, it is clear that using color alone in a straight forward manner is likely to provide poor results in many cases as the coconut and leaf color is almost similar. Thus we attempt to find visual signature that is quite robust using circle Hough transform, as it could be the case of human perception. Fig. 7 shows coconut harvesting robot has the ability to perform autonomous harvesting.
V. CONCLUSION
Fig. 7. Proposed autonomous coconut harvesting robot in action.
Fig. 8. Random initialization of GA.
Fig. 9. Coconut detection result.
novel climbing and fastening mechanism to climb a tree with varying diameter has been introduced. The position of a co-conut is detected using steady-state genetic algorithm followed by sending a command to harvest the detected coconut using an extendable arm equipped with a saw. The experiments show encouraging results which would be beneficial to increase coconut production and to support coconut industries.
REFERENCES
[1] R. K. Megalingam, R. Venumadhav, A. Pavan, A. Mahadevan, and T. Charly, “Kinect based wireless robotic coconut tree climber.” [2] K. Kapach, E. Barnea, R. Mairon, Y. Edan, and O. Ben-Shahar,
“Com-puter vision for fruit harvesting robots–state of the art and challenges
ahead,” International Journal of Computational Vision and Robotics, vol. 3, no. 1-2, pp. 4–34, 2012.
[3] A. Abraham, M. Girish, H. Vitala, and M. Praveen, “Design of harvest-ing mechanism for advanced remote-controlled coconut harvestharvest-ing robot (arch-1),”Indian Journal of Science and Technology, vol. 7, no. 10, p. 1465, 2014.
[4] R. K. Megalingam, T. Pathmakumar, T. Venugopal, G. Maruthiyodan, and A. Philip, “Dtmf based robotic arm design and control for robotic coconut tree climber,” inComputer, Communication and Control (IC4),
2015 International Conference on. IEEE, 2015, pp. 1–6.
[5] R. Fern´andez, C. Salinas, H. Montes, and J. Sarria, “Multisensory system for fruit harvesting robots. experimental testing in natural scenarios and with different kinds of crops,”Sensors, vol. 14, no. 12, pp. 23 885–23 904, 2014.
[6] C. Hung, J. Nieto, Z. Taylor, J. Underwood, and S. Sukkarieh, “Or-chard fruit segmentation using multi-spectral feature learning,” in2013
IEEE/RSJ International Conference on Intelligent Robots and Systems.
IEEE, 2013, pp. 5314–5320.
[7] S. Nuske, S. Achar, T. Bates, S. Narasimhan, and S. Singh, “Yield estimation in vineyards by visual grape detection,” in2011 IEEE/RSJ
International Conference on Intelligent Robots and Systems. IEEE,
2011, pp. 2352–2358.
[8] S. Nuske, K. Wilshusen, S. Achar, L. Yoder, S. Narasimhan, and S. Singh, “Automated visual yield estimation in vineyards,” Journal
of Field Robotics, vol. 31, no. 5, pp. 837–860, 2014.
[9] K. Yamamoto, W. Guo, Y. Yoshioka, and S. Ninomiya, “On plant detection of intact tomato fruits using image analysis and machine learning methods,”Sensors, vol. 14, no. 7, pp. 12 191–12 206, 2014. [10] Q. Wang, S. Nuske, M. Bergerman, and S. Singh, “Automated crop yield
estimation for apple orchards,” in Experimental Robotics. Springer, 2013, pp. 745–758.
[11] C. Bac, J. Hemming, and E. Van Henten, “Robust pixel-based classifi-cation of obstacles for robotic harvesting of sweet-pepper,”Computers
and electronics in agriculture, vol. 96, pp. 148–162, 2013.
[12] I. A. Sulistijono and N. Kubota, “Evolutionary robot vision and particle swarm optimization for multiple human heads tracking of a partner robot,” in2007 IEEE Congress On Evolutionary Computation. IEEE, 2007, pp. 1535–1541.
[13] J. H. Holland, “Adaptation in natural and artificial systems, univ. of mich. press,”Ann Arbor, 1975.
[14] D. Whitley and J. Kauth, GENITOR: A different genetic algorithm. Colorado State University, Department of Computer Science, 1988. [15] S. J. K. Pedersen, “Circular hough transform,” Aalborg University,
Vision, Graphics, and Interactive Systems, vol. 123, 2007.
