Recognising leafy
plants with in-air sonar
Phillip John McKerrow and
Neil Lindsay Harper
An ideal range sensing system provides three types of information: object location, object motion, and object classification. In-air sonar research and development has achieved con-siderable success with the first of these. Object location can be measured in 3D (Akbarally and Kleeman, 1995), indoor en-vironments can be mapped (McKerrow, 1993) and mobile robots navigated (Stevens
et al.,1995).
Using in-air sonar to recognise and classify objects is a new development. The energy in the echo of a sonar chirp from an object is a function of the surface area of the object. The distribution of that energy with time is a function of the relative ranges of the elemental areas that make up the complex surface. Using measurements of the distribution of the energy in an echo, a sonar system can calculate features proportional to the depth and solid area of complex objects. With these features it can recognise objects.
Leafy plants are naturally occurring objects with complex geometric structure. When blind people were trained to use a sonar mobility aid (Kay, 1974) they reported that they could tell the difference between differ-ent plants (Gisonni, 1966). When using a sensor to recognise plants one problem is the asymmetry of plants with rotation. The echo received by the sensor varies depending on the angle from which a plant is insonified. However, we discovered that there is infor-mation in the echo which characterises the structure of the plant and is relatively invar-iant with orientation.
A sensor that can recognise plants has many potential applications. One is the detection of landmarks for the outdoor navigation of mobile robots. Another is the detection of weeds in crop rows. Selective spraying of herbicides can considerably reduce the amount of poison introduced into the envir-onment. A third is the detection of fruit during harvesting.
The techniques for plant recognition can be applied to the sensing of any set of objects that can be separated using the measurement of their geometric structure. A sensor that can
The authors
Phillip John McKerrowandNeil Lindsay Harperare both at the School of Information Technology and Computer Science, University of Wollongong, Wollongong, New South Wales, Australia.
Keywords
Sensors, Sonar, Ultrasonic
Abstract
Continuous Transmission Frequency Modulated (CTFM) ultrasonic sensors can be used to recognise plants. The echo from a plant is modelled as an acoustic density profile. Classification based on features extracted from the echo is more robust than classification based on the echo. Potential applications for this sensing system include landmark navigation and plant sensing for selective spraying of agricultural chemicals.
Electronic access
The research register for this journal is available at http://www2.mcb.co.uk/mcbrr/sr.asp
The current issue and full text archive of this journal is available at
http://www.emerald-library.com
Received and Revised April 1999. This research was supported by Thomson Marconi Sonar, and the sensing system was purchased from Bay Advanced Technologies in New Zealand.
Sensor Review
recognise objects has many potential applica-tions in industry. One is the recognition of sub-assemblies in a manufacturing process. Another is the measurement of the size of gravel in crushing plants. A third is the recognition of faces at airports.
CTFM
Continuous Transmission Frequency Modu-lated (CTFM) Sonar was developed for in-air applications by Leslie Kay (Goughet al., 1984). A CTFM system (Figure 1) transmits a frequency swept sine wave using a wide bandwidth electrostatic transducer. The re-gion of insonification is determined by the geometry of the transducer. Ideally, the echo is a time delayed copy of the transmitted signal, but the shape of the envelope is changed by the filtering effects of reflection off objects and the frequency response of the receiver.
After reception, the echo is demodulated with the transmitted signal (Figure 2) to produce fixed frequency tones for stationary objects. The frequency of a tone is propor-tional to the range to the reflecting object. The amplitude of the tone is proportional to the energy reflected at that range. A Fast Fourier Transform (FFT) is used to convert the time domain tones into a frequency spectrum.
The signal processing for the generation of the sweep and the demodulation of the echo can be done in analog electronics or in software. Digital signal processing requires analog anti-aliasing filters to remove digitisa-tion noise from the transmitted signal. The gains of the filter stages are adjusted to give a fixed amplitude tone for an echo from a reference object that is located on the axis of the transmitted beam.
The sweep time (tsin Figure 2) is determined by the maximum range, the maximum de-modulation frequency and the length of the FFT sampling period. The audio tone for a target is continuous from the time at which the echo arrives until the end of the sweep. Thus, the sweep consists of two time periods: the time to the arrival of an echo from maximum range (td) and the time to capture the samples for the FFT (to). The range of the sensor is selected by choosing the sweep time. A typical set up is a 50kHz sweep (fsweep) from 100kHz down to 50kHz and a sweep time of ts = 102.4 msec. With these settings, and a maximum demodulation frequency of famax = 5kHz, the time to arrival of an echo from maximum range is td= 10.24msec. The velocity of sound in air is 331.6 + 0.6 *T (where T is the temperature in ëC). Thus, the maximum range at 20ëC is 1.75m. During the rest of the sweep, the demodulated frequency is constant enabling the signal to be sampled for frequency calculation. When sampling at 12KHz, a sample period of 86 msec is required to obtain the samples for a 1024 point FFT.
Echo data
In the time domain, the complexity of the audio signal is proportional to the geometric complexity of the target. In the frequency domain (Figure 3), a spectral line occurs for each range where energy is reflected. A
Figure 1 Block diagram of CTFM sensing system
1024 point FFT produces positive amplitude values for 512 * 9.77 Hz frequency bands, each equivalent to 3.43mm of range. The amplitude of the spectral line is proportional to the intensity of the echo and hence to the area of the reflecting surface normal to the receiver.
The foliage of the first plant in Figure 3 is sparse and oriented to form a shell around the plant. As a result, most of the energy is reflected from the front and the back of the plant. This is shown in the spectrum by the large peeks at the start and end of the echo. The second plant is more typical with a large peek in the spectrum toward the front of the plant, due to the denser foliage.
Features
Plants can be recognised by comparing the echo spectrum to a set of recorded spectra for the plants of interest. However, the spectrum varies with rotation. The variation depends on the symmetry of the plant with rotation (Figure 4).
Many plants are locally symmetric with their echo spectra giving good correlation over about 10ë of orientation. Some, like Crinum Pedunculatum, have regions with good local correlation separated by large steps due to the leaves being oriented parallel to a few planes, rather that being spread evenly around the plant. Consequently, most plants can be
recognised with the sensor aimed within 5ë of the orientation at which the reference spec-trum was recorded.
Features extracted from the echo spectrum vary less with rotation, and hence, are better for classification. Features can be selected in a number of ways. We chose to select features on the basis of their ability to discriminate between a set of plants and their correspon-dence to geometric features of the plant. When the features can be mapped to plant geometry we have a more robust sensing system because it is based on a theoretical understanding of the sensing process. The result of research into features is an acoustic density profile model.
Acoustic density profile
The aim of the acoustic density profile model is to map signal features into acoustic features and acoustic features into plant geometry. In most cases, we have achieved a one to one mapping from signal feature to acoustic feature. But the mapping from acoustic feature to plant geometry is often one to many. As a consequence, the acoustic density profile models the structure of the plant foliage rather than the details of the plant, such as leaf shape.
Each spectral line is a measure of the acoustic energy reflected at that range. The
Figure 3 Echo spectra of (a) Eucalyptus maculata and (b) Polyscias murrayi. Note: the plants were isolated from any background while being sensed
frequency is proportional to the range and the amplitude to the acoustic area of the reflecting surfaces at that range. A plant's acoustic area is determined by leaf area, leaf orientation, occlusion and multi-path reflec-tion. The complete spectrum for a plant is the acoustic density of the foliage as a function of depth. The length of the spectrum is the acoustic depth of the plant which is less than or equal to the physical depth.
The sum of the spectral lines (the area under the curve) is the total acoustic energy reflected by the plant and is a measure of the total acoustic area of the plant. The variation of the amplitudes of the spectral lines (envelope of the spectrum) represents the variation in acoustic density as the signal penetrates into the plant. Layering of the foliage results in repeated patterns in the spectrum. For example, the front and back surfaces of Eucalyptus maculata appear as large spikes at the start and end of the spectrum in Figure 3.
Threshold features are good for classifica-tion because they contain a lot of informaclassifica-tion about the structure of the foliage, but they have multiple mappings to geometry. Each threshold feature is the count of peeks above a threshold (one feature for each decade in the range 10.90 per cent) in the normalised spectrum. These features can be used to classify plants according to the denseness of their foliage. Blind users of Leslie Kay's mobility aid claim that they can sense the size of leaves. We are yet to discover a feature that represents leaf size.
Classification
After examining many features we arrived at a set of 19 that produce the best discrimination between plants. These can be further reduced to suit the group of plants in a specific sensing situations. When designing a classifier for an application, we can determine the features we need by measuring the error rate of the classifier when finding the linear separation of the plants. The error rate is the ratio of the number of classification errors to the number of cases.
In our research we measured 100 plants. A statistical classifier was run on all pairs of plants to determine which plants are the most similar. Pairwise classification also
calculates the average classification of a plant against all of the 99 other plants using the Sonar data. The result of pairwise classifica-tion for all 100 plants was an average classification of 90.5 per cent with a standard deviation of 9.6.
Most pairs produced good results and many pairs correctly classify 100 per cent of the test set. But, there are pairs of plants which are difficult to separate. This indicates either that their acoustic density profiles change a lot through rotation or that the plants are similar.
The results of pairwise classification can be used to determine which plants in a set will be easy to recognise and which plants will be difficult. As the number of plants increases the discrimination ability of a classifier re-duces. We use both statistical and neural network classifiers.
A statistical classifier has the advantage over a neural net classifier in that it provides a list of features that are ordered in terms of their discriminatory ability. This provides impor-tant information about the differences between different combinations of plants. A neural net classifier has the advantage that it is a non-linear classifier and generally achieves a better classification result.
Conclusion
Analysis of the acoustic density profiles of 100 different plants shows that they can be separated using pattern classification when a small sample is considered. As the number of plants increases the classification percentage generally decreases. The result of classifying a single return as one of 100 plants is low. When there is a large number of plants, there is a high probability of plants with similar acoustic density profiles.
Plants can be clustered either on the basis of botanical classification or on the basis of similar acoustic templates. The result is very different clusters. Clustering on the basis of acoustic templates gives similar clusters to clustering on the basis of physical character-istics, due to the fact that the acoustic features can be mapped to geometric features of the plant. Some plants show much more variation with rotation than others. Acoustic features show less variation with rotation than the acoustic density profiles.
outdoor mobile robot. Preliminary results indicate that we should be able to track the edges of paths and use plants as natural landmarks. Later this year, we will commence research into agricultural applications of this sensor. This research will focus on recognis-ing weeds among vegetables, such as
cabbages, and detecting fruit on vines, such as tomatoes.
References
Akbarally, H. and Kleeman, L. (1995), ``A sonar sensor for accurate 3D target localisation and classification'', Proceedings ICRA'95, IEEE, pp. 3003-8.
Gissoni, F. (1966), ``My cane is twenty feet long'',The New Outlook for the Blind, February.
Gough, P.T., de Roos, A. and Cusdin, M.J. (1984), ``Continuous transmission F.M. sonar with one octave bandwidth and no blind time'',IEE Pro-ceedings, Vol. 131, Part F No 3, pp. 270-4. Kay, L. (1974), ``A sonar aid to enhance spatial perception
of the blind: engineering design and evaluation'', The Radio and Electronic Engineer, Vol. 44 No. 11, pp. 605-27.
McKerrow, P.J. (1993), ``Echolocation ± from range to outline segments'',Robotics and Autonomous Systems, Elsevier, Vol. 11 No. 4, pp. 205-11. Stevens, M., Stevens, A., and Durrant-Whyte, H. (1995),
[Search] [Advanced Search] [Submit Research] [Edit Research] [Help]
Welcome to the Sensor Review
Internet Research Register
' This Internet-based Register seeks to be a comprehensive, international database for research work
related to sensor technology and development. It is validated and accessible to all those organisations
which subscribe to full-service MCB supporting journals; registrants and members of selected
professional associations and institutes.
The Register will help improve the process of scholarly communication by allowing workers to identify
the types of research already under way as well as identifying areas where research is needed. It will also
help contributors to "stake a claim" to a particular area of research while making sure that they are not
duplicating research that is in progress and hopefully it will help in identifying peers for collaborative
research projects.
Being Internet based gives such a Research Register many advantages over the comparatively static
printed page. I see the Register as a dynamic and changing entity that, because of the technology of the
Internet, can be constantly updated and therefore has the advantage over a printed Register of never
being out of date.
Because the Research Register is making use of this world-wide technology it will reach a truly
international audience. The Register has to incorporate large areas of active research effort to properly
reflect sensor research itself – for example, the design of new materials that will be used in sensors to
new methods of fabricating old materials and from new methods for analysing signals to new
applications for current sensors. I hope that this wide range of expertise and experience from the
contributors can be seen in the entries to the Research Register. I would also like to think that I am
bringing a wide range of expertise and experience to the post of Research Register Editor, having worked
in the area of instrumentation and measurement in both industry and academia for over twenty five years.
I can be contacted at [email protected] and welcome any comments or suggestions that you would like to
make regarding the Register.'
Dr. Ivan Basarab: Internet Research Register Editor
The database is available to all organisations which subscribe to the full information resource Sensor
Review, all registrants and members of selected professional associations/institutions. Please contact
The benefits of registering your research: This Internet Research Register provides the academic and
research community with pre-publication information and the potential for further networking, alongside
early information on new areas for application and development in the field. It allows registrants to:
* ensure that they are not duplicating research that is already underway;
* identify possible research methodologies;
* identify peers for collaborative research projects;
* identify possible sources of funding for research;
* identify types of research currently underway;
* identify areas where further research is required.
All entries are validated by
Dr Ivan Basarab, Sheffield Hallam University, UK.
No responsibility is accepted for the accuracy of information contained in the research presented within this Register. The
opinions expressed herein are not necessarily those of the Editors or the publisher.
© MCB University Press, UK and Reed Technology & Information Systems, UK.
Subscribers:
Your individual subscription details
Enter the library
Emerald e-mail alert
service registration
W
elcome to the Emerald intelligence and fulltext electronic library spanning subjects such as: marketing,
general management, human resources, quality, property, operations, production & economics, library &
information services, information management, training & education and engineering.
Please select one of the following links to access the specific table of contents for that journal or suite.
Alternatively select "Enter the Library" to access the Emerald Library
General Management (Suite)
(access expires 30/12/ 2007)
●
Human Resources (Suite)
(access expires 30/12/ 2007)
●
Information Management (Suite)
(access expires 30/12/ 2007)
●
Library & Information Services (Suite)
(access expires 30/12/ 2007)
●
Marketing (Suite)
(access expires 30/12/ 2007)
●
Property (Suite)
(access expires 30/12/ 2007)
●
Operations, Production & Economics (Suite)
(access expires 30/12/ 2007)
●
Training & Education (Suite)
(access expires 30/12/ 2007)
●
Quality (Suite)
(access expires 30/12/ 2007)
●
Engineering (Suite)
(access expires 30/12/ 2007)
●
Subscribers: register access to additional databases
If you subscribe to one or more MCB journals that are not listed above, please enter the requested information in
the boxes below and press the "Add database(s)" button. We will then register all of the subscriptions associated
with your customer number to this site.
Your registration will be confirmed by e-mail and access will be granted immediately.
Registration
1. Customer information
Your customer ID
Your name
Your e-mail address
Would you like to permit password protected document ordering (for some or all of your users) on-line using the
document delivery credits included with the subscription(s) you are now registering?
If yes please choose and enter twice the document delivery password you would like for your site.
Document delivery password
Document delivery password (again)
Are you missing out?
There are probably many more journals on this database which, at present, you do not have access to. Many of the
individual journals have a full archive - abstracts from 1989 to 1993 and full text from 1994 to the present day.
There may be other journals which you would be interested in viewing for a 30 day period. If you would like to
take advantage of this offer, please email Mark Hindwell quoting your customer ID and the journals you wish to
trial. Your "free trial" registration will be confirmed by e-mail.
Mark Hindwell
E-mail:
Telephone: (44) 1274 785278
Fax: (44) 1274 785204
Technical help desk
If you have any problems with registering additional subscription(s) to this site you should contact our technical
help desk for assistance. When doing so please quote your customer ID.
Technical Help Desk
E-mail:
Telephone: (44) 1274 785278
Fax: (44) 1274 785204
Please remember that the passwords that you select are case sensitive.
America Online users: your Internet e-mail address is your AOL user name followed by @aol.com. If your user name is xyz, your
e-mail address is [email protected]
CompuServe users: your Internet e-mail address is your CompuServe user id, with the middle comma replaced by a full stop,
followed by @compuserve.com. If your user id is 3000,100, your e-mail address is [email protected]
Yes
