Agilely Assigning
Sensing Assets to
Mission Tasks in a
Coalition Context
Alun Preece, Cardiff University
Tim Norman, University of Aberdeen
Geeth de Mel, IBM TJ Watson Research Center
Diego Pizzocaro, Cardiff University
Murat Sensoy, Ozyegin University
Tien Pham, US Army Research Laboratory
When managing
intelligence,
surveillance, and
reconnaissance
(ISR) operations in
a coalition context,
assigning available
sensing assets to
mission tasks can
be challenging. The
authors’ approach to
ISR asset assignment
uses ontologies,
allocation
algorithms, and a
service-oriented
architecture.
that end users can achieve using various types of visual sensing data (visible, radar, infrared, and multispectral).2 ISR analysts,
for instance, can’t be expected to have a specialist’s sensing knowledge; they must be able to state their information needs in terms of what they want (such as to track high-value targets in an area) rather than how the sensor data can satisfy those needs. Coalition partners must maintain control over how the assets they own are shared with other partners.1 Therefore, absent a
great deal of knowledge about sensing ca-pabilities and coalition asset availability, identifying suitable ISR assets is difficult.
Because ISR situations can evolve rapidly, the asset-provisioning infrastructure that supports ISR operations must be agile, re-sponsive to changing user needs as well as the availability of relevant assets.
Any solution to this problem requires a common representation of tasks and assets that users can extend to new task or asset types. It must express tasks at a high level in terms of what the user wants. To match tasks to available assets, efficient mecha-nisms are needed that consider all possible means of satisfying the task. To solve this problem, several works have proposed using a knowledge base or mapping that relates
I
n a coalition context, effectively using intelligence, surveillance, and
recon-naissance (ISR) assets is a challenge.
1Using sensor-provided data, there are
multiple ways to achieve an ISR task. For example, the National Image
C O A L I T I O N O P E R A T I O N S
sensor capabilities to task require-ments to help either automatically or semiautomatically identify suit-able assets for tasks.3–5 In our
previ-ous work,6 we developed an approach
to automatic asset-task assignment founded on the Military Missions and Means Framework (MMF).7 We
cre-ated ontologies for both task and asset types and an automatic procedure that matched one to the other based on re-quired and offered capabilities. Here, we describe our approach’s current status and its implementation using a service-oriented architecture (SOA) with mobile apps to serve users.
An Ontological Approach to Asset-Task Matching
To derive our asset-task matching on-tology, we formalized concepts and relationships from the MMF docu-mentation, tailored to the ISR do-main (see Figure 1). Missions consist of operations that are in turn made up of tasks. Tasks require capabilities, which assets provide. Assets include platforms and systems—including sensors—that are mounted on platforms.
The allocatedTo relationship shows that an asset is assigned to serve a particular task. We implement our ontology in the Web Ontology Lan-guage (specifically, OWL DL).
The current matching procedure using this ontology is based on the NIIRS framework. NIIRS associates various ISR tasks with ratings for the types of visual sensing that can col-lect sufficient data for achieving the tasks. We formalized a collection of statements derived from NIIRS and related literature in the form of a knowledge base, which in abstract form, contains a set of intelligence clause tuples with six elements:
• one of four intelligence capabilities—
localize, detect, distinguish, or identify (the latter three are capabilities in NIIRS);
• a set of detectable entities drawn from the NIIRS framework (for ex-ample, type of vehicle or building);
• a set of more specific features from the detectable entities (for example, a base’s roads or guard posts, or airport runways);
• a context that defines the precondi-tions that must hold for the intelli-gence clause to apply (for example, detecting a ship in the context of open water);
• the type of sensor-provided data, which includes the NIIRS types
visible, radar, infrared, and multi-spectral, plus other types such as
acoustic and seismic; and
• an NIIRS rating on a scale of zero to nine (for example, visible-6 or
radar-4).
Users express their tasks in terms of the required intelligence capabil-ity and detectable entities—for ex-ample, detect {tank} or distinguish {t ank, jee p}. Requ i red t asks fea-ture either a single detectable (for de-tect, identify, and localize tasks) or a pair of them (for distinguish tasks). Users also specify the area of inter-est and other parameters, which we describe later in the context of our mobile app.
To determine the capabilities a task requires, the system matches the task’s intelligence capability and detectable entity to the intelligence clauses’ knowledge base. This in-volves two kinds of inference: First, some task types imply others (for ex-ample, the ability to identify some-thing implies an ability to detect it). Second, a hierarchy of detectables means that, for example, any clause involving detect and a type of detect-able covers all the more specialized types of that detectable—for exam-ple, the task detect {car} covers spe-cialized car types, such as jeep, SUV, saloon, and so on.
The system allocates sensors to tasks in terms of bundles (a task might require more than one sensor— for example, a pair to localize an ob-ject in 2D), and a bundle type com-bines a platform type and a set of sensor types that can be mounted Figure 1. The Missions and Means Framework (MMF) intelligence, surveillance,
and reconnaissance (ISR) ontology. The ontology lets us relate sensing assets to mission tasks.
Task Capability
Operation
Mission
Asset
Platform System
Sensor comprises toAccomplish
comprises toAccomplish
toPerform
is-a
is-a is-a
mounts
attachedTo requires
provides allocatedTo
on that platform type. The ontol-ogy contains the additional rela-tionship interferesWith to cover cases in which sensor types are in-compatible. We derive bundle types from deployable configurations as discussed in the next section. These bundle types also take into account restrictions on the number of sen-sors that can be mounted simul-taneously on a platform. In terms of our MMF ontology, a bundle type unifies the capabilities that ei-t her ei-t he plaei-t form or sensor ei-t y pe provides.
A bundle type matches a task type if the capabilities the bundle type provides contain those capabilities the task type requires. We say that a bundle type minimally matches a task type when the system can’t re-move any sensor type from the bun-dle type such that the matches rela-tionship still holds.
Using these definitions, the match-ing procedure operates as follows.
First, a user creates a task from which the system derives the corre-sponding required intelligence capa-bility and detectables. Second, the system determines all bundle types that minimally match the task.
Third, the system determines all possible bundle instances that con-form to each bundle type in the re-quired area of interest. A bundle in-stance is a deployable, configured set of platform and sensor type assets de-fined by the bundle type. The system determines bundle instances by que-rying the coalition’s asset catalogue and governs them via sharing and deployment policies, as we describe later. For example, an unmanned aer-ial vehicle (UAV) wouldn’t be selected for an area under a no-fly zone.
Finally, the system uses an alloca-tion mechanism to choose the most
appropriate bundle instance to assign to the user’s task. Depending on the use context, the system can do this several ways; one method is to use a distributed allocation protocol8 that
attempts to maximize the overall util-ity of assigned assets in the face of multiple competing tasks. The pro-tocol aims at maximizing the util-ity provided to the most important tasks—that is, it lets the system re-allocate sensor assets to more impor-tant, newly created tasks (we refer to this as preemption). This requires an appropriate utility function to deter-mine the “goodness” of assigning a particular bundle instance to a given task. (See the “End-to-End Asset- Task Allocation and Information Delivery” sidebar for an example.)
We designed our knowledge-based approach to be extensible and main-tainable. The core sensor, platform, and task ontologies are based on
F
scribed in the main text. The process operatesclock-wise from the top left. The user specifies a task to localize sport utility vehicles (SUVs) in an area of interest. The matching procedure uses the Military Missions and Means Framework’s intelligence, surveillance, and recon-naissance (ISR) ontology and knowledge base clauses to determine that this task can be achieved by assets with a visible NIIRS rating of four (or higher) or an acoustic NIIRS rating of six (or higher). These ratings are provided by the following bundle types, respectively: an unmanned aerial vehicle (UAV) platform with a video camera sensor or an unattended ground system (UGS) platform with an acoustic array sensor. (This example is simplified; many more possi-bilities and more specific asset types exist for this task.)
Next,the system generates bundle instances and ranks them with a utility function for the localization task. A bun-dle instance can contain more than one instantiation of the bundle type when more than one set of deployed assets is needed to achieve the task. In our example, this results in three pairs of acoustic-sensing bundles, each containing two instances of the bundle type <UGS,AcousticArray>
(at least two assets are required for triangulation). For the bundle type <UAV,VideoCamera>, the bundle-generation process results in two visual-sensing bundle instances com-posed of a single UAV mounting a camera. In total, five can-didate bundles exist. The ranking process enables the sys-tem to choose one acoustic bundle and assign it to the task.
We envisage our approach being deployed in the context of a service-oriented architecture, in which sensors and pro-cessing services are “wrapped” as Web services, allowing an agile service configuration to deliver information to users— potentially to mobile devices if users are in the field—once assets are assigned.
Figure A. Overview of the intelligence, surveillance, and reconnaissance (ISR) asset-task assignment approach. The process operates clockwise from the top left.
UGS + acoustic array
UAV + video camera
Bundle types
A2 A1
A1 A3 A2 A3
SOA for information delivery
U2 U1
Localize SUV
Task
MMF ontology + NIIRS knowledge base
Sensor bundle generation
Utlity-based asset allocation
Sensor Web service
C O A L I T I O N O P E R A T I O N S
preexisting models that are relatively stable and accepted.6 The ontologies
are open to new asset, task, and ca-pability types—in fact, we added the NIIRS framework as an extension to the original ontologies. Some plat-form types aren’t yet incorporated into our system, such as satellites, but they’re covered by the ontologies on which our approach builds (for ex-ample, Ontosensor5), so adding them
would be straightforward. In general, we would expect new asset, task, and capability types to appear only rarely. To make its assets available to our system, a new coalition partner must create representations of its asset in-stances in a shared asset catalogue. The underlying middleware will de-termine how this occurs. Our later example approach uses a SOA archi-tecture. In future work, we plan to
extend the approach by including hu-mans as sensors.
Extensible Matching Framework
To be useful in a coalition context, an asset-task assignment system must be extensible and flexible in terms of not only new ontological and knowledge base elements (for example, to incor-porate new kinds of sensing assets and alternative formulations of ISR tasks); we must also be able to extend it to matching schemes (including al-ternative allocation procedures). In our system, we enable this extensibil-ity and flexibilextensibil-ity using Ontological Logic Programming (OLP),9 which
combines Prolog with DL-based rea-soning. An OLP program can dy-namically import various ontologies and use the terms (that is, classes,
properties, and individuals) directly within an OLP program. An OWL DL reasoner interprets the ontolog-ical terms during OLP program in-terpretation to support operations such as subsumption, satisfiability, consistency, class equivalence, and class instance checking.
Figure 2 shows a fragment of our OLP implementation for computing deployable configurations (for use as bundle types). The OLP program is a Prolog program in which concepts and properties from the underlying ontologies are referenced directly. T he M M F ISR ontolog y (ht tp: // hom epa ge s . ab d n . ac .u k /c .e m ele / pages/ita/index.php?page=resources) is imported on the first line. The get-Configurations predicate computes deployable configurations (bundle types) for a specific task. Each sen-sor must be carried by a deploy-able platform that provides all of the task’s operational requirements (for example, constant surveillance). If a sensor can’t be carried by a de-ployable platform, we can’t consider deployable configurations with that sensor type.
Using this knowledge, the system can employ a tailored and efficient matchmaker that first identifies the deployable platforms that meet the task’s requirements. Once this pro-cess narrows the many possibilities, the system determines the sensor types that provide a task’s required intelligence capabilities incrementally so that those sensors can be mounted on the deployable platforms. In the code, terms f rom t he M M F I SR ontolog y have t he pref i x istar:. Most of these are shown in Figure 1 (requireOperationalCapability can be considered a specialization of requiresCapability).
This method for computing deploy-able configurations is based on the idea that we can significantly reduce Figure 2. Ontological Logic Programming code to compute deployable
configurations. Terms from the Military Missions and Means Framework’s intelligence, surveillance, and reconnaissance (ISR) ontology have the prefix istar:.
%import http://…/ISTAR.owl deployablePlatform(T,P), extendSolution(T,P,[],S). istar:'Platform'(P),
not((istar:'requireOperationalCapability'(T,C), not(istar:'provideCapability'(P,C)))).
requireSensor(T,P,Prev,X), istar:'mounts'(P,X), A=[X|Prev],
extendSolution(T,P,A,Next).
not(requireCapability(T,P,S,_)).
requireCapability(T,P,S,C), istar:'provideCapability'(X,C).
edge. For this purpose, the system ex-ploits dependencies between sensors and platforms. Using this principle, during each iteration, the matchmaking algorithm rules out many combi-nations and significantly reduces the time required to compute deploy-able configurations. We compared the computational performance of the OLP program in Figure 2 with a set-covering algorithm that exhaustively searches for deployable configurations. As the size of deployable configura-tions increases, our experiments have shown that the OLP-based approach outperforms the exhaustive search ap-proach significantly; that is, the time required for an exhaustive search in-creases exponentially, whereas that of the proposed approach is linear.9
Asset-Task Matching in an SOA
To enable our approach to operate within an SOA for sensor information processing and delivery, we deployed the asset-task matching mechanism as a service within the International Technology Alliance’s ITA Fabric.10
The Fabric provides a distributed stream-oriented middleware layer that mitigates the complexity of man-aging message flows between coali-tion partners and across disparate networks while encapsulating net-work and security management for resource-constrained networks. The Fabric implements a two-way mes-saging bus and a set of middleware services that provide connectivity among all network resources, to each other, and to users. A typical Fabric node consists of a message broker, a Fabric manager, and a Fabric reg-istry. The Fabric manager manages all the communication channels be-tween nodes—that is, message rout-ing between nodes, sensrout-ing resources, and users. The Fabric registry records
information about all the nodes, routes, and ISR resources; this database is distributed across each of the Fab-ric nodes, recording resource types, physical locations, operational char-acteristics, task commitments, and current operational status and avail-ability. The registry implements the asset catalogue. Thus, one of the im-mediate benefits of embedding our approach in the ITA Fabric is that the matching process can use the Fab-ric registry to limit the generation of bundle types to only those deploy-able configurations in which asset in-stances are available for assignment.
Figure 3 illustrates how we inte-grated our approach in the ITA Fab-ric. R1 through R5 represent sensing resources, and N1 through N4 are Fabric nodes. Because Fabric nodes are intended to be lightweight and deployable on low-powered sensor platforms, running the computation-ally expensive DL-based reasoning operations is infeasible. Instead, we divided our approach’s functionalities into two separate components: we deploy the OLP implementation of the reasoning procedure on a server that’s accessible via an API defined by a set of RESTful Web services. This is separate from the user-facing applica-tion that lets users submit tasks and receive recommendations. We refer to
the reasoning API as reasoning-as-a-service (RaaS).
Asset-Task Matching via a Mobile Device
To conceptually illustrate our ap-proach, we created mobile apps for both smartphone and tablet plat-forms. The most recent version is im-plemented as an iPad app. Its main features let users
• create an ISR task in an area of in-terest via a convenient user inter-face and submit the task for asset assignment;
• view all tasks with assigned assets in an area of interest (subject to ac-cess policies); and
• sha re t asks a mong each ot her (again, subject to access policies).
The motivation for task sharing was to reduce competition for resources by making it easier for users to share an existing task than to create a new resource request. The app aims to give users an overview of how well-covered an area is in ISR terms—not only what tasks are currently being resourced but also these tasks’ likely permanence (by letting users view task details, such as ownership and priority). This differs significantly from simply showing the current Figure 3. Integrating our approach in the ITA Fabric. R1 through R5 represent sensing resources, and N1 through N4 are fabric nodes.
N2
N3
R3
R4
R5
ITA sensor
fabric N4
C O A L I T I O N O P E R A T I O N S
location of sensor assets deployed in the field; in fact, displaying such in-formation might be infeasible due to visibility policies. For example, a co-alition member might not be allowed to see other partners’ sensor locations or exact resource types, but it might have the right to access the data col-lected. Moreover, displaying sensor locations could be misleading for a commander, who might plan a mis-sion in a certain area because it’s “better covered” by sensors while those sensors might in fact be busy or preempted to serve other, more im-portant tasks.
Figure 4 shows two screens from the iPad app. The task list is on the left and task locations are shown as circular regions on the map. Us-ers can obtain the task list by que-rying the registry on a local Fabric node and contextualize it based on the area of interest. The latter is de-termined by the iPad’s location (via GPS) by default, but users can also set it manually. After defining the area of interest’s radius, the user can
move the task to the desired location on the map by dragging the center of the defined area of interest. Defining areas as circular is a limitation, but it would be straightforward to im-plement more complex shapes. The interface also lets users set the task type, task priority, and expected task duration. The user interface is de-signed for a logged-in user who be-longs to a single coalition partner and can see a list of tasks (and their details) belonging to other coalition partners according to predefined ac-cess policies. Acac-cess policies are rule-based,1 so they can consider factors
such as the user’s coalition partner membership, rank, and member-ship in a particular coalition group as well as the partner ownership, rank, and group associated with the task.
Users can search the task list us-ing the box on the top left, which lets them filter the displayed list via an intelligence capability or type of de-tectable. They can obtain more detail by selecting an individual task, which
results in a display such as that at the far right. Assuming the task has as-signed sensors, the display summa-rizes the task and the assigned asset bundle. With this amount of detail, users can understand how a task is currently being resourced. Including the NIIRS rating indicates the quality of the data the asset bundle can col-lect. We’re considering other ways to convey quality information because it might be unreasonable to assume users are familiar with the NIIRS scale. The task priority indicates how “stable” the task is—lower-priority tasks are more prone to preemption if assets are too scarce to cover all tasks. Information on the task owner and other users who have joined the task should have a “social” effect be-cause a logged-in user is likely to know other users in the region.
B a s e d o n t h i s d e t a i l e d t a s k- assignment information, users can touch the buttons on the bottom left to join or edit the existing task. This entails a more efficient use of net-work resources because avoiding the creation of a new task reduces com-petition among users when sensing resources are limited. In some cases, a user’s information requirements might be satisfiable with preexisting tasks that are already being served by allocated bundles; thus, there would be no need to preempt resources from other tasks.
We demonstrated our iPad app to ISR specialists from the US Depart-ment of Defense, the UK Ministry of Defense, and NATO communities in September and October 2011. Users appreciated how the app separates the information the user requires and how they can obtain this information via various sensing assets. The demos also elicited positive comments re-garding the mobile version’s usabil-ity and simplicusabil-ity in terms of how us-ers can request complex information Figure 4. Sensor assignment to Missions iPad app. The task list is on the left, and the
Moreover, the specialists appreciated that low-level details of both sensors and tasks (for example, particular sensor capabilities required to sat-isfy a task) can remain hidden to a user, making the system accessible to nonexperts.
G
oing forward, we plan to ex-periment with richer ways of interacting with users via natural language interfaces and using the ap-proach to assign precollected infor-mation sources and streams in addi-tion to sensing assets. Other related, ongoing work considers network con-straints (for example, bandwidth) when allocating asset bundles as well as trust and information obfuscation issues in relation to ISR asset sharing among members of heterogeneous co-alition teams.Acknowledgments
T his research was sponsored by the US A rmy Research Laboratory and the U K M inistry of Defence and was ac-compl i she d u nder ag re ement nu mb er W911NF-06-3-0001. The views and con-clusions contained in this document are those of the authors and should not be in-terpreted as representing the official poli-cies, either expressed or implied, of the US Army Research Laboratory, the US govern-ment, the UK Ministry of Defence, or the UK government. The US and UK govern-ments are authorized to reproduce and dis-tribute reprints for government purposes notwithstanding any copyright notation herein.
References
1. T. Pham et al., “Intelligence, Surveil-lance, and Reconnaisance Fusion for Coalition Operations,” Proc 11th Int’l Conf. Information Fusion, IEEE, 2008, pp. 1–8.
2. J.M. Irvine, “National Imagery In-terpretability Rating Scales (NIIRS),”
Encyclopedia of Optical Eng., 2003, pp. 1442–1456.
3. L. Lefort, C. Henson, and K. Taylor,
Semantic Sensor Network XG Final Report, W3C, 2011; www.w3.org/ 20 05/ I ncubator/ssn / XGR-ssn- 20110628/.
4. T. Mullen, V. Avasarala, and D.L. Hall, “Customer-Driven Sensor Manage-ment,” IEEE Intelligent Systems, vol. 21, no. 2, 2006, pp. 41–49. 5. D. Russomanno, C. Kothari, and
O. Thomas, “Building a Sensor Ontology: A Practical Approach Leveraging ISO and OGC Models,” Proc. 2005 Int’l Conf. Artificial Intelligence, CSREA Press, vol. 2, 2005, pp. 637–643. 6. M. Gomez et al., “An Ontology-Centric
Approach to Sensor-Mission Assign-ment,” Proc 16th Int’l Conf. Knowl-edge Eng. and KnowlKnowl-edge Manage-ment (EKAW 08), Springer, 2008, pp. 347–363.
7. J.H. Sheehan et al., “The Military Mis-sions and Means Framework,” Proc. Interservice/Industry Training and
Simulation and Education Conf., I/ITSEC, 2003, pp. 655–663. 8. D. Pizzocaro et al., “A Distributed
Architecture for Heterogeneous Multi-Sensor Task Allocation,” Proc. 7th IEEE Int’l Conf. Distributed Comput-ing in Sensor Systems (DCOSS 11), IEEE CS , 2011, pp. 1–8.
9. M. Sensoy et al., “Ontological Logic Programming,” Proc. Int’l Conf. Web Intelligence, Mining, and Semantics
(W IMS 11), ACM, 2011, article no. 44.
10. J. Wright et al., “A Dynamic Infra-structure for Interconnecting Disparate ISR / ISTAR Assets (the ITA Sensor Fabric),” Proc. 12th Int’l Conf. Information Fusion, IEEE , 2009, pp. 1393–1400.
vironments. Preece has a PhD in computer science from the University of Wales, Swansea. He serves on the editorial boards of IEEE Intelligent Systems and the Knowledge Engi-neering Review. Contact him at a.d.preece@cs.cardiff.ac.uk.
Tim Norman is a professor of computing science at Aberdeen University. His research interests include multiagent systems, with a focus on computational models of policies (or norms), trust, and argumentation. Norman has a PhD in computer science from the University of London (University College). Contact him at t.j.norman@abdn.ac.uk.
Geeth de Mel is a postdoctoral researcher at the IBM T.J. Watson Research Center and the US Army Research Laboratory. His research interests include artificial intelligence, especially Semantic Web technologies, and decision-support systems in the presence (or lack) of dynamicity, trust, and provenance. De Mel is a PhD candidate and an honorary research fellow at the University of Aberdeen, UK. Contact him at grdemel@us.ibm.com.
Diego Pizzocaro is a research associate at Cardiff University. His research interests are in resource allocation in sensor networks, mobile computing, and intelligent sys-tems. Pizzocaro has a PhD in computer science from Cardiff University. Contact him at d.pizzocaro@cs.cardiff.ac.uk.
Murat Sensoy is a lecturer in computer science at Ozyegin University and an honorary lecturer at Aberdeen University, UK. His research interests include the Semantic Web, multiagent systems, and trust and reputation. Sensoy has a PhD in computer engineering from Bogazici University. Contact him at murat.sensoy@ozyegin.edu.tr.
Tien Pham is a team leader within the Networked Sensing and Fusion Branch of the US Army Research Laboratory. He’s responsible for research in network science, sensor net-works, and acoustic and multimodal sensing. Pham has a PhD in electrical engineering from the University of Maryland at College Park. He’s a member of IEEE and the Acous-tical Society of America. Contact him at tien.pham1.civ@mail.mil.
