ContentslistsavailableatScienceDirect
Internet of Things
journalhomepage:www.elsevier.com/locate/iot
IoT-based human action prediction and support
Gabriel Machado Lunardi
a, Fadi Al Machot
b, Vladimir A. Shekhovtsov
c, Vinícius Maran
d, Guilherme Medeiros Machado
a, Alencar Machado
e, Heinrich C. Mayr
c,∗, José Palazzo M. de Oliveira
aaPrograma de Pós-Graduação em Computação (PPGC), Universidade Federal do Rio Grande do Sul, Porto Alegre, Brazil
bResearch Center Borstel–Leibniz Center for Medicine and Biosciences, Borstel, Germany
cInstitute of Applied Informatics. Alpen-Adria Universität, Klagenfurt, Austria
dLaboratory of Ubiquitous, Mobile and Applied Computing (LUMAC). Universidade Federal de Santa Maria, Cachoeira do sul, Brazil
eColégio Politécnico, Universidade Federal de Santa Maria, Santa Maria, Brazil
a rt i c l e i n f o
Article history:
Received 5 September 2018 Accepted 10 September 2018 Available online 13 September 2018 Keywords:
Probabilistic ontologies Uncertainty reasoning Ambient assistance Smart home Context-awareness
a b s t ra c t
It is an important topicin Active and Assisted Living (AAL)researchand development tosupportelderly peoplesuffering frommemoryimpairmentintheirdailyactivities. A promisingapproachtosuchsupportisprovidingmemoryaidsbasedonknowledgeofhow thepersontobesupportedusually(i.e.,inanunimpairedcondition)copeswithher/his dailyactivities.SuchknowledgemaybecapturedbyIoTsolutions,appropriatelystructured andstoredinaknowledgebase,andexploitedwhentheneedofsupportisdetected.De- terminingthebesthelpforagivensituationimpliesdecision-making,sincetheactions–
flow(behavior)ofanactivityusuallyinvolvesprobabilisticbranches:Anautomatedsys- temneedstodecide whichofthepossible nextactionsis bestsuitedfor theuser ina givensituation. Problemsofthisnature involveuncertaintylevels thathave tobedealt with.Manyapproachestothisproblemexploitstatisticaldataonly,thusignoringimpor- tant semanticdataas,for instance,areprovided byOntologies.However, ontologiesdo notsupportreasoningoveruncertaintynatively.Inthispaper,wepresentaprobabilistic semanticmodelthatrepresent informationfromIoTsources andenablesreasoningover uncertaintywithoutlosingsemanticinformation.Thismodelisimplementedasanexten- sionof theHuman BehaviorMonitoringand Support(HBMS) approachthat providesa conceptual”humancognitivemodel” forrepresentingtheuser–sbehavioranditscontext inher/hislivingenvironment.Theperformanceofthisapproachwasevaluatedusingreal datacollectedfromasmarthomeprototypeequippedwithinstallablesensorsandIoTde- vices.Theexperimentsprovidedpromisingresultswhichwewilldiscussregardinglimits andchallengestoovercome.
© 2018ElsevierB.V.Allrightsreserved.
∗ Corresponding Author.
E-mail addresses: [email protected] (G.M. Lunardi), [email protected] (F.A. Machot), [email protected] (V.A. Shekhovtsov), [email protected] (V. Maran), [email protected] (G.M. Machado), [email protected] (A. Machado), [email protected] (H.C. Mayr), [email protected] (J.P.M. de Oliveira).
https://doi.org/10.1016/j.iot.2018.09.007
2542-6605/© 2018 Elsevier B.V. All rights reserved.
1. Introduction
Thelikelihoodofdiseaseincreaseswithage.Some ofthesediseases,e.g.memoryimpairments,compromise theability tocopewiththeActivitiesofDailyLife(ADLs)[1]likedressing,cooking,usingelectronic devices,takingamedicationand soon.Thisinturnaffectstheabilityofthepersonconcernedto liveindependentlyandself-determinedinhis orher ac- customedhome environment.ActiveandAssistedLiving(AAL)[2–4]isthedisciplinethatseeksformaintainingindividual independence.Itaimsatprovidingappropriateassistancetailoredtotheparticularsituationbyvariouskindofserviceslike mobilityassistance,socialinclusion,communicationandADL facilitation.Thechallengehereistodeveloprobust,unobtru- sive,automated,easytouseandsafesystemsforhumanassistance.
TheHBMS(HumanBehaviorMonitoringandSupport)project[5–8]addressesthatchallenge.Itaimsatderivingsupport servicesfromintegratedmodelsofabilities,currentcontextandepisodicknowledgethatan individualhadorhas,buthas temporarilyforgotten. The coreof theHBMS systemis the HumanCognitiveModel (HCM) [6].It preservesthe episodic memory(roughlyspeaking: the flow andexecutionofactions in ADLs)of a personinthe formof conceptualmodels of behaviorlinkedtocontextinformationrelatedtotheseactivities.Accordingly,theHBMSsystemfollowstheModelCentered Architecture(MCA)paradigm[7].
Likeinotherapproaches, usersupportisadecision-makingprobleminHBMS, sincethereistypicallynotonlyonese- quenceofactionsleadingtoa(ADL)goal:oftentherearealternativeswherethesystemneedstodecidewhichofthepos- siblenextactionsisthemostappropriateonefortheuser.Thiscanbeseenasapredictionproblem,sincethenextactions correspondtotheimmediatefuture.Problemsofthisnaturegenerallyinvolveuncertaintylevelsthat mustberepresented andtreated[9].
Aninitial solutiontopredictinguserbehavior wasto reasonoveran OWL-DLontology [10]representingthe HCMse- manticknowledgewithin theMCAlanguage framework.Themainlimitationofthissolutionisthatdescriptionlogic(DL) doesnotgivenativesupportforreasoningaboutuncertainty;itisasubsetofFirstOrderLogic(FOL)whichdealswithsets ofaxioms, “thelogicalconsequencesofwhicharesentencesthataretrueinallinterpretations”[11].FOLthusislimitedly suitablewhenthereisuncertaintyintherelationshipsamongconceptsasinourcase.
Someresearcheffortstrytoovercometheselimitationsusingfuzzylogic,neural networksorBayesiannetworks.How- ever,thesestrategiesproducepurelystatisticalresultsasthepredictionismadewithoutexploitingsemanticinformationas couldbeprovidedbyanontologyrepresentinguncertainty.
Therefore,thispaperproposesprobabilisticextensionstoboththeHCMsemanticmodelanditsOWLrepresentation,i.e.
addressesthemonsemanticandlanguage representationlevelsoftheMCA language paradigm.Thiswillallow reasoning overuncertaintyaimedatprovidingsupporttotheuser.
The viabilityof the approachwas demonstratedusing a casestudy scenariowhere the usersupport is simulatedfor particularADLs.It proved that theprobabilistic semanticmodel supports theautomatic generation ofBayesian Networks inaspecificsituationdependingontheavailableevidencewithinthat model.Afterthat,we evaluatedtheperformanceof theapproachusingtheHBMSsmarthomeprototypewheredatawascollectedthroughsensors.Thisexperimentprovided promisingresultsandrevealedlimitsandchallengestoovercome.
Thepaperisorganizedasfollows:Section2presentsthenecessaryconceptualbackgroundontheareascoveredbythis study.Section3shortlysketchesHBMS,thesystemweusedforimplementingourapproachwhichispresentedinSection4. Section5outlinesthecasestudy.Evaluationanddiscussionisexposed inSection6.Comparisontorelatedworksisgiven inSection7followedbyanoutlooktofutureperspectivesinSection8.
2. Conceptualbackground
2.1. ActiveandassistedlivingbasedonIoT
AALisaresearchfieldfocusingoninnovative solutionsallowingpeopletoageindependently,safelyandcomfortablyin theirlivingenvironments[12].ItexpandsontheconceptofSmartHome(SH),i.e.ahome environmentassembledwitha seriesofsensors,actuatorsanddevicestomonitoraperson’sADLsandtotakecareofbasictaskslikeswitchingoff electrical devices,closingwindowsandsoon[13].
BeingabletoliveindependentlyroughlymeansbeingabletoperformthebasicandinstrumentalADLs[1,14,15].Conse- quentlyanAALsystemhastobeawareoftheseinordertoprovideassistanceattherighttime.ManyAALsolutions rely, forthatpurpose,ontheinformationprovidedbyIoTsmartdevices[16,17].Suchdevicesareabletoanswerrequestsrelated totheirsetofproperties,currentstate values,allowed operatinginstructions,andotherinformation.Amongthesolutions supportingsuchdeviceswecandistinguishtheOpenSemanticFrameworkbySiemensandtherelatedprojects[18,19],and theWeaveproject[20].
Other AAL approachestry to exploit the output ofindependent Activity Recognition Systems (ARS) [21]. An example isourown HBMSsystemprototype: iteven maybecoupled toseveralARSat atime. For,duringourinvestigations and experiments,itturned outthatcurrentlyavailableARShaveonlyrestrictedrecognitionrealms.As aconsequence,they do notreveal allaspects toberecognized whenaiming atsupporting ADLs.Incontrasttothat,integratingandreasoningon theoutputsofacombinationofARSdeliversacceptableresults[22].
2.2. Context-awarenessandmodeling
Activitiestakeplaceinanenvironment:thecontext.Contextualinformation,therefore,iscrucialforactivityrecognition aswellasforadvicedeductionandproviding assistance.Consequently,AALsystemshavetobecontext-aware [23].There areseveraltypesofcontextualinformationsuchaslocation,people,objects,time,things,etc.[24].Context-awareapplica- tionsexploitsuchinformationforansweringquestionssuchas“who”,“where”,“when”,“what” and“withwhat” inorderto determine“why” aparticularsituationisoccurring[25].
Thisiswheremodelingcomesin:Behaviorandcontextmodelsarethebasisfordata-drivenapproaches(supportdeduc- tionbyanalytics),knowledge-driven(deductionbyreasoning)oracombinationofthem.Forexample[24],whenaperson hesitatesinan ADLanddoesnot rememberthesteps (actions[26]) tocomplete,anAALsystemcouldgive advicesbased ontherespectivemodels.Among theexistingtechniquesforcontextmodeling are:ConceptualModeling,Key-ValuePairs, MarkupSchemas,Graphics,Object-oriented,Logic-basedandOntologyBased[27].
Mooreetal.[27]analyzedandcomparedsuchtechniquesonthebasis ofthefollowingsixfeaturescommonlyrequired byintelligentcontext-awaresystems:
• DistributedComposition:referstothefactthatcontext-awareapplicationsaregenerallyimplementedindistributedand dynamicsystems;
• Partialvalidation:abilitytopartiallyvalidatecontextualknowledgeatthestructurallevel,aswellasattheinstancelevel, giventhepotentialexistenceoferrorsinthedefinitionofcontextualrelationsbetweenentities;
• QualityofInformation:thequalityofinformationprovidedby sensors(e.g.withinan IoTdevice)overtime, aswell as thewealthofinformationcharacterizingentities;
• IncompletenessandAmbiguity:abilitytodealwithincompleteandambiguousinformation;
• LevelofFormality:descriptionoffacts(contextual)andinterrelationsinanaccurateandtraceableway;and
• Applicability/Adaptability:allowsuseinexistingdomains,systems,andinfrastructure.
The comparison revealed that modeling techniques based on ontologies achieve good results regarding these criteria [27,28].ThemostcommonlyusedlanguageforimplementingontologiesisOWL[29].
2.3. Handlinguncertainty
Uncertainty is omnipresent inwhat happens around us, consequently assertions are mostlyuncertain, like a weather forecast;adiagnosisofadiseaseortheperceptionoftheworld[30].Uncertaintymaybeunderstoodasthelackofadequate informationtomakeadecision.Forexample,contextualinformationderivedfromdatacollectedbyIoTdevicesensorsisnot always accurate,duetohardwarefailure,lackofpower,communicationproblems,etc.Thus,acontext-aware systemmust besensitivetosuchuncertaintyformakingthebestdecisions.Therepresentationandtreatmentofuncertaintyappearsas acrucialnecessityforasatisfactoryandlogicalreasoning[2].
Amongthemain approachestoreasoningaboutuncertaintywe highlight:Fuzzylogic,probabilisticlogic,Bayesiannet- works,HiddenMarkovmodelandDempster–Shafer’stheoryofevidence.Eachofthesehasadifferentsuitabilitydepending ontheintendeddomain.Ingeneral,probabilisticreasoningallowstheprocessingonuncertainty,becauseittreatshypothe- sestoidentifytheprobabilityofoccurrenceevenbeforethesetofinformationthatevidencesthemisknown.Withthis,it ispossibletomakepredictionsthroughuncertainreasoning[29].
BayesianNetworks (BN)arecommonlyused tocoverthelimitationofDLregardinguncertainty.ABNmodels asetof randomvariables X=X1,...,Xn, eachvariablebeinganode ofaDirectedAcyclicGraph(DAG).Thenodesareconnected throughdirectedarcsrepresentingdependencebetweenthem.Forexample,letXi→XjrepresentthatXiistheparentofXj. Eachnode ofthegraphcontainsa ConditionalProbability Table(CPT)P(Xi|parents(Xi))whichrepresentstheprobability of Xiconditionedtoitsparents.ThissuggeststocombineforuncertaintytreatmenttheexpressivepowerofFOLasisintrinsic toOWL-DLencodedontologies,andprobabilistic reasoningthroughBayesianNetworks.Thisideaisaddressedby thearea ofprobabilisticontologies.
2.4. Probabilisticontologies
Probabilisticontologiescanbe definedasbeingexplicit,formal,structured andshareablerepresentationsofknowledge aboutan applicationdomain by means of: entitytypes; propertiesof theseentities; relationships;processes andevents that occur withthese entities; statisticalregularities that characterize the domain; inconclusive, ambiguous,incomplete, unreliableknowledgerelatedtodomainentities;anduncertaintyoverall previousforms.Assuch,probabilistic ontologies expandtheexpressivepowerofstandardontologiesbyallowingforanadequaterepresentationofthestatisticalregularities ofanapplicationdomain[31].
One wayto generatesuch ontologies is touse a First-OrderProbabilistic Language(FOPL) which combinesaspects of probabilistic representation withFOL. The Multi-EntityBayesian Network(MEBN)can be regarded asthe state-of-the-art of such languages[32]. TocombineMEBN withOWL-DL, an upper ontology called ProbabilisticOntology Web Language (PR-OWL)wascreatedby P. Costa[31]theexpressivenessofwhichis powerfulenoughtorepresentevenhighlycomplex
domains.Currently itisinsecond version, andanopen-source visual tool calledUNBBayes(availableat:http://unbbayes.
sourceforge.net/)wasdevelopedtosupporttheconstructionofPR-OWL2ontologies[33].
PR-OWLis afirst orderBayesian logic basedon MEBNandassuch integratesFOLwithprobability theory.The Bayes’
Theoremprovidesabasisforinference[34].
2.5.Multi-entityBayesiannetworks
Inpractical terms, MEBNrepresents an Universe ofDiscourseas aset of interrelatedentities andtheir respectiveat- tributes.KnowledgeabouttheattributesofentitiesandtheirrelationshipsisrepresentedasacollectionofMEBNfragments (MFrags)organizedinMEBNtheories(MTheories).AMTheoryisdefinedasasetofMFragsthattogethersatisfyconstraints ofconsistency ensuring the existence ofa single joint probability distribution. An isolated MFrag can be compared to a standardBNinwhich itsrandom variablesarecallednodesrepresentingtheattributesandpropertiesofaset ofentities andits arcsare relations ofdirect dependencebetweennodes [11].It is alsoworth to saythat MEBN by itself doesnot specifyastandard forCPTs,buttheUnBBayestool providesaflexible wayofdeclaringthemtobecalledLocalProbability Distribution(LPD).Thisisaccomplishedthroughaspecialscriptinglanguage.Furtherdetailscanbefoundin[35].
AnMFragconsistsofthreetypesofnodes[35].
• residentnodes:randomvariablesformingthecenterofaMFrag.TheLPDsaredefinedhereexclusivelyandexplicitly.If they cannotbeexplicitlydefined, auniformdistributionisassumed. Thepossiblevaluesofa residentnodecanbean existingentityoralistofmutuallyexclusivevalues;
• inputnodes:influencethedistributionoftheresidentnodes;theirdistributionsaresetintheirownMFrags;
• contextnodes:representconditionsthatmustbesatisfiedfortheinfluencesandlocaldistributionsofanMFrag;
The MEBN reasoning andinference process is carriedout, first,by the interposition of a questionwhich implies the generationofaSpecificSituationBayesianNetwork(SSBN). ThisisacommonBNthataims todeterminetheprobabilities ofasituationinthedomain.Instancesandevidences(knownfactsofthedomain)haveafundamentalroleinthisstageas theyaretheonesthatcomposethequestionstobesubmittedtothereasoningalgorithm[36].
2.6.Modelcenteredarchitecture
OurcontributionisinspiredbytheModelCenteredArchitecture(MCA)paradigm[7,37].Accordingtothisparadigm,an informationsystemistreatedasacompoundofvariousmodelsand‘’modelhandlers” thatproduceand/orconsume/exploit models. The models, in particular, describe not only the application domain and it’s functionality but also the system’s interfaces(Fig.1).EachofthesemodelsisformedwiththemeansofaDomainSpecificModelingLanguage(DSML)providing thebackboneforsemanticsandstructure, andforcomputer-aided processing,represented bymeans ofarelatedDomain SpecificRepresentationLanguage(DSRL).
MCAcomprisesametamodelhierarchy,arepresentationlanguageframework,andasetofarchitecturalpatterns. Metamodelhierarchy
The metamodel hierarchyforms the semantic coreof a MCA system:MOF [38] level M2 defines the metamodels of theparticularDSMLs,M1specifies allsystemanddatacomponentsintheformofextensions(instances)oftherespective M2 metamodels; M0 represents extensions (instances) of the M1 models, i.e.models of concrete objects, functions and processes.Fig.1showsthedifferentareasofconceptualizationoftheMCAparadigminmoredetail:
1.theapplicationdomain(s)togetherwithapplicationdataexchangeanduserinterfaces;foreveryapplication(sub)domain aswellasforeveryinterface,anappropriateM2-levelDSMLhastobedefined;
2.themodelexchangeandmodelmanagementinterfaces.Themanagementinterfacesallowforhandlingmetamodelsand models asartifacts; the exchange interfaces allow for the export and import of(meta)models on all levels and thus support theintegrationof artifactsfromexternalsources. Aconcrete applicationsystemresultsfromtheinstantiation of the M2 metamodelson the M1 level.The running applicationitself results fromcreating extensionsof M1 model elementsontheM0level,forexamplebyMDA[39]ormodels@runtime[40]mechanisms.
Languagerepresentationframework(Fig.2)
Thisframeworkprovidesthesyntacticmeansforrepresentingthecoresemanticsby useofdifferentappropriate repre- sentationlanguages:fromgrammardefinitionmeansviatheconcretegrammarsforthelanguagesspecifyingthemeansof representingsemanticconcepts,anddowntoprovidingconcretelanguage representationsofcoresemanticconcepts from differentlevels.AsdepictedonFig.2,wedistinguishthefollowingelementsofthelanguagedefinitionframework:
1.Grammardefinitionelements.Inourresearch,we usea specificversionofEBNF,compatiblewiththeANTLRgrammar definitionlanguage[41].
2.Language definition elements: define grammars for the various representation languages (RL) related to the defined DSMLs:meta-meta-modelRLs(notshownonFig.2forbrevity),meta-modelRLs,modelRLsandinstance/dataRLs.
3.Representationelements:representationsofthemodelsofalllevels.
Fig. 1. MCA Architecture (adapted from [7,37] ).
Fig. 2. MCA Language architecture (extended from [37] ).
Fig. 3. MCA pattern architecture (from [37] ).
Fig. 4. Simplified HBMS-System’s components and its stakeholders [6] .
Architecturalpatterns
Aset ofarchitecturalpatterns (Fig. 3) includesthe modeling tool pattern(components usedfor creatingmodels), the modeltransferinterfaceandthemodeladapterpatterns(forthecomponentsprovidingthemodelstotheconsumersystem), themodelmanagementpattern, thestoragepattern,andthemodelhandlerpatternwhichdescribesthecomponents,that use,createandmanipulatethemodelstoprovidethefunctionalityofthesystem.
3. Humanbehaviormonitoringandsupport
TheHBMS(HumanBehaviorMonitoringandSupport)systemistheresultofanAALprojectofthesamename.It’saim istomaintain,aslongaspossible,thepersonalautonomyofanindividualwithmildtomoderatecognitiveimpairmentsin performinghis/herdailyactivities.TheHBMSsystemestablishestwocloselyinterwovenprocesses:
1.In the learningmode (i) the behavior ofa target person isobserved usingavailable activityrecognition infrastructure andsystems,and(ii)theseobservationsaretransformed,integratedandpreservedinaknowledgebase,theconceptual HumanCognitiveModel(HCM).
2.Inthesupport mode,theperson’sbehaviorisobservedaswell,howeverprimarilyforrecognizingsituationsofneedof assistance;ifsuchsituationsoccur,theHCMandadomainontologyareexploitedfordeterminingthemostappropriate actiontohelpthepersonreachingher/hiscurrentgoal[6].
Theoverallsystem’sarchitectureimplementsMCAarchitecturalpatterns[7].Thisarchitectureaswellasthestakeholder’s rolesaredepictedinFig.4:Thetargetuserlivesinasmarthome.ThemodelerusestheHCM-Ltoolto(1)specifythefloor planofthesmarthome,(2)adaptandextend,ifnecessary,theHCMestablished byobservation,or(3)establishtheHCM manuallyifno automaticrecognition isavailable. Themodelersupports theHuman CognitiveModeling Language(HCM-L), aDSMLfordescribing aperson’sepisodicknowledge(autobiographicaleventsandcontextualinformation) anditscontext
intheformofconceptualmodels[42].TheAdministratormanagesthesysteminstallationandconfiguration,andmonitors thesystem’sstate.The knowledgesupplierprovidesandintegratesdomainknowledgeintotheHBMSdata;moreover,the knowledge clientmaybe used by doctorsandcaregiversfordiagnosis purposes.Clearly, anykindofaccessto theuser’s personaldataisbasedonpreviousaccessauthorization.
The HBMSObservationInterface(HBMS-OI)acquires observedbehavioraldata fromheterogeneousactivityrecognition systems(environmentalmonitoringmiddlewareinFig.4).TheinterfacetosuchsystemsisagaindefinedusingaDSML,the ActivityRecognitionEnvironmentModelingLanguage(AREM-L)[22].TheremainingHBMSkernelcomponentsarethe“model handlers” accordingtotheMCAparadigm:TheobservationenginelistenstothedataarrivingfromHBMS-OI,analyzes and processesthisdata,andtransfers recognizedbehavior situationstotheHBMSBehaviorEngine.TheBehavior Enginetrans- forms the received informationinto HCM-L modelfragments (recognizedcontext andbehavior). Inlearning-mode,these fragmentsarecollectedtoformsequencesofactionswhichthenareintegratedintothe(growing)HCM.Astheintegration isbasedonheuristicrules,interactionviatheintegrationclientmightimprovetheHCMcreationifthetargetuserisable andwilling to cooperate. Insupport mode,the Behavior Enginecontinuously matches the observedbehavior against the HCM,retrieves appropriateknowledge fromHCMwhilelookingfordiscrepancies, determinesthenecessary knowledgeto correctthesediscrepancies(e.g.inaformofsetsofcorrectingactions)andtransferssuchknowledgetotheHBMSSupport Engine.Thelatterisresponsibleforthecontextsensitivemulti-modalassistanceofthetargetuserbyusinganappropriate supportclient[6].
TheHBMSdatalayeriscomposedofatleastfourdatasources:(i)TheHCMKnowledgeBasestoresthecompletecognitive modelof atarget user.(ii) TheSituationCache containscurrentlyrelevantandreferencedHCM fragments,andstate data collectedfromobservations; i.e., it is thesystem’s operationalknowledge base.(iii) TheDomain Ontologystoresdomain- specific knowledge that is referencedfrom both HCM and Situation Cache. (iv)The CaseBase stores all observed action sequences.Makinguseofthisinformationfordeterminingthemostlikelynextsteptobeadvisedisthechallengeaddressed inthispaper.
ThedevelopmentofHCM-Lwasguidedby theconceptualizationofdailyactivities ofaperson’sprivate life(ADLs,see Section 2). Therefore,HCM-Lis basedon activitytheoryasproposed by Leontyev[43].AllHCM-L concepts andtheir se- manticrelationshipsareexpressedusingthemeta-modelpublishedin[6].RegardingtheMCAlanguageframework,HCM-L formsapartofthesemanticcore.TheHBMSknowledgebase,i.e.theHCM,isrepresentedusingOWLasa representation language.Whereneeded,OWLexpressionsaretranslatedintoconstructsofanotherrepresentationlanguage.
Themodelingtool,HCM-LModeler,hasbeendevelopedusingthemetamodelingplatformADOxx®[5].Thetool(avail- ableat:http://www.open-models.at/web/hcm-l/download)providesthemeansforvisualizing,analyzing,verifyingandvali- datingHCM-Lmodels,inparticulartheHCM.
4. Approachforpredictinghumanactions
Initsfirstversion,theHBMSBehavioralEngineutilizednon-probabilisticreasoningforsolvingitstasks.Forthepurpose ofhandlinguncertainty,wehadto
1. injectprobabilityextensionsintothesemanticcoreforHCM-L(themetamodelinghierarchy);
2. converttheresultingextendedcoretogetaPR-OWLrepresentationthat canbe usedbyontologyreasonerssupporting OWL.
The measures forextending the semanticconcepts andrepresenting them by meansof OWL2to be usedfor reason- ing, are shown on Fig. 5: We started fromthe two separate languages, HCM-L andPR-OWL (1). Then we extractedthe probabilistic conceptsofPR-OWLfromthePR-OWL metamodelandinjectedthemintotheHCM-Lmetamodeltoformthe resultingextendedPR-HCM-Lmetamodel(2).ToprovidearepresentationforPR-HCM-L,we developedaconverter[44]to transformtheUML-basedHCM-LmetamodelrepresentationintoOWL2suchthat thePR-HCM-Lmetamodelrepresentation isconformanttothe(common)OWL2grammar(3).
Thedetaileddescriptionofthisextensionprocessispresentedinthenextsection.
4.1. Extensionofthesemanticcoremodelbyprobabilisticinformation
According to the MCA LanguageFramework, we distinguishthe core concepts (semantics) ofthe metamodel andthe constructs of the representation language. In the case of HCM-L, the metamodel representation language may be UML- based,butmayalsohaveadifferentsyntax.Inthissection,weshowhowtheprobabilisticextensionsareinjectedintothe coreconcepts.
Coreconcepts
Fig. 6showsthe fourcoreconcepts of HCM-L andtheir relationships:(1) BehavioralUnitencapsulatesthe alternative sequences of(2)operations(actions)that apersonperformsinordertoachieve the(3)Goalofadailyactivity.Operations areconnectedby(4)Flowsthusformingsequences.Alternativeactionsequencesforreachingagivengoalresultinbranches andjointsintheBehavioralUnit.
Contextmodelingisaddressedby theconceptThinganditsrelationships. Itdescribesconcreteorabstractobjects,fea- tures andpeople, which havea purpose.Person represents a personby various information aboutthe psychological and
Fig. 5. Extending the Core Semantic Model by PR-OWL probabilistic concepts.
Fig. 6. HCM-L core meta-model and probabilistic extensions.
physicalstate,modeledasattributesand/orpartsofit.Locationisforthedescriptionofroomsandotherspaces,facilitating theacquisitionofdata(temperature,humidity,sound,etc.).EachOperationrelatestosomeThingthroughthreeassociations:
(i)Calling,Thingthat startoperations;(ii)Participating,Thingsthat contributeorare manipulatedbyoperations;and(iii) Executing,Thingsthatexecuteoperations.
Probabilisticextensions
The probabilistic extensions to the semantic core model were determined by separating the conceptual elements of a probabilistic ontology (implementedin PR-OWL)fromtheir OWLrepresentation,andinjecting these elementsinto the HCM-Lcore.TheresultingsetofextensionsisdepictedonFig.6inredcolor.Itconsistsofthetwoattributes,hasHistand hasSimiliarityGain;andthetworelations,isAtOperationandmayBeNextOperationOf.
TheseextensionsexploitedtheMTheorypresented inFig.7foranswering thefollowing question(query):Whatis the probabilityoftheuserperformingagivenoperation?Theanswerisprovidedbytheresidentnode mayNextOperationOf(p,op)
Fig. 7. MTheory for reasoning over uncertainty of operations.
whichmeans:“maybethenextoperationof”.Thisnamewaschosentomentiontheuncertaintylevelassociatedwiththe relationshipbetweenPersonandOperation.
AnMTheoryisarepeatablestructure(template)fromwhichSSBNsaregeneratedaccordingtothebehavioroftheuserin agivenADL.Thus,atruntime,whenthereisaneedtogenerateanSSBNtopredictanextoperation,i.e.,whenasituation ofcognitivedeclineisdetected,thenetworkstructureisgenerateddynamicallyaccordingtoMTheory.Thisiscomposedby thefourfragmentsdescribedbelowwhoserandomvariablesrelatedtothecontextnodesare: “p” forPerson,“currentOp”
and“op” (futureoperation)forOperation.Residentnodeswillbedisplayedas: <residentnodename> =states.
• Person_MFrag:representstheprobability oftheuserbeingstoppedatagiven“currentOp” withintheactivity.Thisfact isrepresentedbytheresidentnodeisAtOperation(p,currentOp)={True,False};
• SimilarityGain_MFrag:representsthelikelihoodof“op” being performedinasimilarmannerbydifferentusers.Inother words,itisaprobabilisticsimilaritymeasurebasedontheGaussiandistributionwhoseparametersarecalculatedusing thenumberoftimesa specificoperation ischosen byall theuserswho performedthesameactivity. Theresponsible residentnodeisasfollows:hasSimilarityGain(op)={True,False};
• hasHist_MFrag:representstheprobabilityof“op” beingexecutedbasedontheuser’shistory.Itcanbeunderstoodasthe historicalimportance,representedbytheresidentnodehasHist(op)={performed,notPerformed};
• NextOperation_MFrag: represents the probability of a next “op” being recommendedto the user through the resident nodemayBeNextOperationOf(p,op)={True,False}.Theprobabilitydistributionofthisnodeisdirectlyinfluencedbythe twopreviousMFragsthatworkasprioritymeasures.
Inordertogenerate,accordingtotheMTheory,thestructureoftheSSBNsdynamically, eachresidentnodeofanMFrag must alsoimplementits own localprobability distribution, whosevalues canbe reported bya domain expertorcan be generatedthroughmachinelearningalgorithms.
Injectingprobabilisticextensionsintothecoresemantics
Forinjectingtheprobabilisticextensionsintothecoresemanticswefollowedtheaspect-orientedparadigm[45],because ofitssimilar motivation[46]:theconceptsets tobecombinedbelongto differentconcerns ofinterestfortheproblemat hand(separation ofconcerns [47]). In ourcasethe concernsare human cognitivemodeling andprobability modeling. For thispurposewe
1. mapped the meta-metamodel of the injected PR-OWL metamodel into the target: the HCM-L meta-metamodel. This mappingwas straightforward asoriginally both meta-metamodelscontained a commonset ofelements whichare of interestforourproblem,inparticular,includingmetaclassesAttributeandRelation;
2. determined the set of extension points within the HCM-L metamodel, i.e., the metamodel elementswhere the meta- attributesandmeta-relationscanbeinjected.Inourcase,thissetconsistsofOperationandPerson;
3. specifiedtheinjections:
(a) hasHistandhasSimiliarityGainattributesareinjectedtobelongtoOperation;
(b)isAtOperationandmayBeNextOperationOfrelationsareinjectedtoconnectOperationandPerson. Asaresult,weobtainedthePR-HCM-Lmetamodel,graphicallyrepresentedinFig.6.
4.2. OWL2representation
ToobtainarepresentationofthePR-HCM-Lmetamodelsuitable forinference,we developeda UMLtoOWL2converter [44]and used it for converting the concepts originating from the HCM-Lmetamodel and originally represented by means of UML.Inparticular:
Fig. 8. Probabilistic HBMS behavior engine.
1.theconceptualclasseswererepresentedbyOWL2classeswiththeexceptionoftheFlowclassdescribedbelow;
2.the relations betweenThing (contextelements) and Operation (action) were represented as three direct-inversepairs ofobjectpropertiesinOWL2:(IsParticipating,IsPacticipatedBy);(isCalling,isCalledBy);and(isExecuting,isExecutedBy),the relationbetweenBehavioralUnitandPersonresultedinthepair(belongsTo,belongedTo),therelationbetweenBehavioral UnitandGoal-inthepair(has,isHadBy).
3.the compositional relation between BehavioralUnit and Operation were represented by a pair of object relations (is- ComposedOf, isComponentOf), together withthe axiomsmaking ita composition. Thisrelationship allowsformodeling activitieshierachically,i.e.,whenanoperationisnotasinglestepbutasubactivitywithinanotherupperactivity.
4.theBehavioralUnit’sattributeshasPossibleBeginning(indicatespossiblestartoperations)andhasSuccessEnding (indicates successfulendoperations)wererepresentedbytwoeponymousobjectproperties.
5.theFlowclasswasrepresentedbyanobjectpropertyhasFlowwithdomainandrangebeingOperation.
6.theConnectionclassanditsPart-OfspecializationwererepresentedbytheobjectpropertiesisIn,isOn,isUnder, isNextTo, isInFrontOf,isBehind,isWithandthetransitivepropoertyisPartOf,respectively.
Theconcepts andpropertiesoriginatedfromthePR-OWLmetamodel(hasHist,hasSimiliarityGain,isAtOperationandmay- BeNextOperationOf) did not requiretheabove conversionasthey were originally representedby means ofOWL2. Sothey justwereconnectedtotheconvertedHCM-Lelementsinquestion.
Asa result,we obtainedan OWL2representation ofthecompletePR-HCM-Lmetamodel,readyto serveforprediction andinference.
4.3.Supportarchitecture
Forexploitingtheextended HCMmetamodel,theHBMSBehaviorEngine(see Section3)hadtobeadaptedasoutlined inFig.8.NotethatthisFiguredoesnotshowthecompleteBehaviorEnginefunctionality,butonlythefunctionsaffectedby ourextensions.
TheimportantparthereisthePredictiveOperationModelthat,convertedintoasyntacticrepresentation,supportstool- basedreasoningandinference.OurcontributionistomakethismodelaninstanceofthePR-HCM-Lmetamodelrepresented bymeansofOWL2.
WedefinedthePredictiveOperationsModelusingtheUNBBayesmodelingtool.WithintheMCApatternframework,this toolimplementstheModelingToolpattern.
This modelis fed by LPDs, whose probability valuescan be derived by machine learning algorithms, existing in the BehaviorEngine.ThesealgorithmsaresupportedbythedatafromtheCaseBase,whichcontainsallthehistoricalsequences ofactionsperformedbytheuser.Otherwise,LPDsmayalsobeprovidedbyaspecialistifnoautomatedlearninghasbeen performed.
Then,instancesofthecurrentbehaviormodelareretrievedfromtheHCMknowledgebasebytheBehaviorEngineand alignedwiththedomainontology.Theseinstancesreplacethemarkersoftheprobabilisticsemanticmodel,thusgenerating theSSBNs,towhichanusual Bayesiannetworkinferencealgorithmisapplied.SSBNsaregeneratedforeachnext possible operationrelativetotheonetheuserstoppedat.Therefore,anintegrationmodulemergestheSSBNs,selectingtheoperation thatismostlikelytoberecommendedtotheuser.
ThoughwedevelopedthisapproachasapartoftheHBMSsystem,weclaimthatitissufficientlygenerictobeapplied toanysystembasedonMCAarchitecturalpatterns.For,thekeymodules(predictiveoperationsmodel+SSBNs) arebased onUnBBayesAPIandcanbeaccessedfromtheMCAmodelhandlercomponents.
5. Casestudy
Inthissectionwepresentacasestudydemonstratingtheapplicationoftheproposedapproach.
Fig. 9. John’s behavior and context models.
5.1. Scenario
The case studystarts froma very simple scenario,focusing on a elder person John who suffers from mild cognitive impairments,moderatehearinglossandvisualimpairment:
Johnlives in asmarthome (equippedwith theHBMSsystem)whichconsists ofa kitchen, a living room,a dining room, a bedroomandabathroom.Inthisenvironment,intheafternoon,JohnusuallywatchesamovieusingtheDVDplayer.Todothis, he entersthelivingroomanddecides onwhichsofahe willsit,on sofaAor sofaB.However,Johndoesnot alwaysremember whichsofaprovidesbetterdistanceandangleforsightandhearingrespectively,sofaA,orsofaB.Upondetectingthissituation,the HBMSsystemcomesintoactionrecommendingJohnthebestoptionbasedonhisownknowledge(whilehavingbeencognitively well).AfterfindingthemostsuitablesofaJohnactivatestheDVDandfinallytheTV.TheactivityendswhentheTVisturnedoff.
(Forsimplicityreasonsturningoff theDVDisnotmodeled.)
5.2. Applyingtheapproach
Fig.9showsagraphicalrepresentationofJohn’sformerbehaviorwhenperformingtheactivity“WatchaDVD” learnedby HBMSviaobservation.Assuch,the“BehavioralUnitModelWatchaDVD” isaMOFM1instanceofthemetamodelconcept
“BehavioralUnit” presentedinFig.6.ThisBehavioralUnitModelisdirectlyrelatedtothe“StructuralContext(Model)” which againis an instance ofthe particular metamodelconcepts (we useddifferent colors forbetter readability). The relations betweeneachOperationandthecontextelementsaswellasthedeficienciesofJohnweresuppressedfromthediagramfor readabilitypurposes,buttheyexist.
TheusersupportprocessisinitiatedwhenadeviationinJohn’sbehaviorisdetectedbytheHBMSBehaviorEnginesuch that Johnmightnot reachhis initiallyrecognized goaltowatch aDVDina bestway(i.e.,whenJohn hesitatesorchoses a suboptimal Sofa). Forthis purpose the HBMSBehavior Engine compares the time window spent when the operation is performed normally with the current time spent. In the current scenario, John stops at the “Enter the living Room”
operationforsometime,whichsuggeststhathehasacognitiveproblemindecidinguponwhattodonext.Thus,theLPDs areinformedtotheprobabilisticsemanticmodelthatgeneratestheSSBNsforeachpossiblenextoperation,“SittingonSofa A” and“SittingonSofaB”,asshowninFig.10,toestimatethechanceofrecommendingoneoranothertoJohn.Thevalues ofeachLPDswereobtainedfromtheHBMSdataset(seethenextsection).
ThevaluesinFig.10areexplainedusingthe“SitonSofaB”operation;thesamelogicappliestothe“SitonSofaA”.Thus, theLPDfortheresidentnodehasHist(sitOnSofaB)meansthattheoperationwasperformed67%ofthetime(performed)and notperformed33%ofthetime(notPerformed).Ontheotherhand,theLPDfortheresidentnodehasSimilarityGain(sitOnSofaB) meansthattheoperationwasperformedinasimilarway(True)in80.8%ofthecasesandperformedinanon-similarway (False)in19.2%ofthecases(assumingthatJohnliveswithapersonwhosharestheexecutionofsomeactivities).Finally,the LPDfortheresidentnodemayBeNextOpOf(John,sitOnSofaB)describeshowthehasHistandhasSimilarityGainvaluesinfluence theinferenceofprobabilitiesforthe“SitonSofaB”.Thus,if
• theoperationisperformedwellandinasimilarway,itwillhavea73.9%chanceofbeingrecommendedand26.1% of notbeingrecommended(line3).Ifitisperformedwell andinanon-similarway,thenthechanceofrecommendation fallsto43%andthatofnon-recommendationrisesto56.9%(line5);
Fig. 10. LPDs for each possible next operation.
Fig. 11. The resulting SSBNs for each possible next operation.
• theoperationisperformedpoorlyandinasimilarway,itwillhavea43.1%chanceofbeingrecommendedand56.9%of notbeingrecommended(line8).Ifitisperformedpoorlyandinanon-similarway,thenthechanceofrecommendation dropsto26.1%andthatofnon-recommendationrisesto73.9%(line10);
Withthis, and fromthe evidence isAtOperation= true that determines the operation ofthe activity inwhich John is stopped,twoqueries,one foreachpossiblenext operation,areperformed:(A)What istheprobabilitythatJohnwillsiton thesofaA? (B)WhatistheprobabilitythatJohnwillsitonthesofaB?Theresultofthesequeriesisexpressedbythemay- BeNextOpOfnode,shownonFig.11intheSSBNsforthepredictionandsubsequentrecommendationofthemostappropriate operation.TheupperSSBNreferstothequery(A),whoseprobability ofrecommendationis40.74%.Thelower SSBNrefers tothequery(B),whoselikelihoodofrecommendationis58.7%.Therefore,therecommendedoperationforJohnwouldbe
“SitonSofaB”,deliveredtothemostappropriatedevicee.g.asawarningsoundwhenthewrongsofaischosen.
6. Evaluation
Inthis section we describe how theevaluation wasconductedand discussthe results.The test aims to measure the performance of the proposed approach, checkinghow well John would be supported. In other words, we verified if the recommendedoperation“sitonSofaB”wouldbeperformedbyhimifhewasnotaffectedbycognitivedecline.Todoso,we firstlydetailourtestingenvironmentandtheperformancemeasuresadopted.Finally,wepresentanddiscussourresults.
6.1. Testingenvironmentanddataset
Wemadeour test ina smarthome prototypeofthe HBMSprojectfaithfully representedin Fig.12.Within thisenvi- ronment,“John’sactivity” wasperformedbyavolunteer10timesrepresentingJohnaswhenhewascognitivelywell.Thus, realdatawerecollectedthroughsensorsinstalledonthefollowingobjects[objectname(sensortype)]:(i)SofaA(sensing);
(ii)SofaB(sensing);(iii)livingroom(sensingcarpet);WealsousedthefollowingIoTdevices:(iv)TV(IoTswitch);and(v) DVD(IoTswitch).Thesesensorsanddevicesbasicallytracetheactionsoftheactivitycountinghowmanyactivationswere done.ThisresultedintheHBMSdataset.
Thesensorsincludedinthesettingwerechosenbyusingdifferentfactorswhichhavetobeconsideredwhenimplement- ingtheminrealenvironments. Forexample,small,cheap,non-intrusive,easytobe installed,compatiblewithIoTdevices.
Fig. 12. HBMS Smart Home prototype.
Suchrequirementsofferthepossibilitytousetheproposed approachinreallife scenarios.Additionally,duetoprivacyis- sues,wedidnotinvolveanyvisualoraudiosensors;asidefromthat,privacyissueswereoutofscopeforthisresearchbut willbeaddressedinfuturewhenintegratingvisualsensors.
Duringthestudy,theperformedactivities wereannotatedby usinganannotation-app onatablet.Foreachactivityof interest,aseparatedatachannelwascreatedinthedatabase.Everytimeapredefinedactivitywasstarted,thecorresponding channel wasset to1,andafterfinishing theactivity,thechannel wassetbackto0.Thisguaranteed apreciseannotation oftheperformedtaskswiththeir startandendtime,thatwasrequiredforfurtherdataanalysisandactivityclassification purposes.
Thedesiredbasisformat wasdefinedasafilewhereeach rowrepresentsthesensorstatesandannotationvaluesata specificpointoftime.Usingthisbase-file,dynamicwindowsforeachactivityhavebeencreated.Thisapproachallows for adynamicdefinitionofthesizeoftheneededdatasettoenableitsusageinvariousfieldsofapplication.
Although there are well known international datasets forAAL, such as CASAS (available at: http://ailab.eecs.wsu.edu/
casas/datasets.html), theydonotfitthesystemrequirementsproposed heredueto:(i)lackofdatathatofferchoices(be- tween operations) totrain the probabilistic semanticmodel; and(ii) toomuch concernin providingdata totest activity recognition algorithms. Therefore, we had to create anduse an appropriate (HBMS) dataset which now can be used by otherapproachesforcomparisonpurposes.
6.2. Resultsanddiscussion
Initially, we split the 10 samples of the HBMS dataset into two partitions. The first one corresponds to the training subset andthesecond tothetest subset.Thispartitioningprocess isknownasholdout andconsistsofdividingadataset into70%fortrainingand30%fortestingi.e.,inourcase,7instancesfortrainingand3instancesfortesting. Weusedeach trainingpartitiontoestimatetheprobabilityvaluesthatfeedtheprobabilisticsemanticmodelmakingitabletopredictand recommendthenext possibleoperation.Thus,weusedthetest partitiontoverifythenumberoftimesinwhichtheuser choseornotthecorrectrecommendedoperation,“sofaB”.
However,wehaveidentifiedaproblemwiththeuseofholdout:itisnotpossibletoevaluatehowmuchtheperformance ofanapproachvarieswiththedifferentcombinationsofinstancesinitstraining.Thus,tomaketheresultslessdependent onthedefinedpartition,10random partitionswereestablishedto obtainameanretentionperformance,a methodcalled randomsampling.Wedefinedthenumberofpartitionsbasedontests,since,becauseitwasrandomandbasedonfewdata, thesamplingcouldcreatethesameorverysimilarpartitions.Basedonthesetests,weverifiedthatthisproblemstartedto appearwhilecreatingmorethan10partitions,thusjustifyingtheupperlimitonthenumberofpartitions.Incidentally,this wouldimplyanerroneousincreaseordecreaseinthemeansofperformanceobtained.Moreover,itisworthtosaythatthe randompartitionswereobtainedthroughtheWekasoftwarepreprocessingmodulewiththerandomsamplingfilter.
In other words,we can say that 10“new” HBMS datasets havebeen createdand, asa consequence,we repeatedthe process described inthe first paragraph 10 times.From each test partition of each “new” dataset we got one confusion matrixusedtocalculatetheadoptedperformancemeasures:precision,recall,andF-measure.Wechosethemgiventhefact thattheyarecommonlyusedintestswhichinvolveprediction.Belowwepresentthenotionofeachoneofthem:
• Precision:Anumberoftimesthattheuserfollowedtherecommendedoperation,dividedbythetotalnumberofsamples inthetestset.
Fig. 13. Experiment results.
• Recall:Anumberoftimestheuserfollowedtherecommendedoperation,dividedbythetotalnumberofsampleswhere theuserfollowedtherecommendedoperation.
• F-measure:representstheweightedharmonicmeanofprecisionandrecall.Iftheweightequalsto1,thenprecisionand recallhavethesamedegreeofimportance;thisisthesettingweuseinthiswork.
Fig.13showstheresultsobtainedformetricsmentionedpreviously foreachexperiment.Standarddeviationvaluesare 18.106;0;and16.2forprecision,recallandf-measurerespectively.
Therefore,itwaspossibletoconcludethat,despitethelowamountofdataavailable,theobtainedresultswerepromising, withanaverageprecisionofapproximately70%.Webelievethatbetterresultscouldbeobtainedifmoredatawereavailable.
Therefore,anewdatacollectionisplannedinvolvingmoreusersintheHBMSsmarthomeprototypethat willresultina larger dataset. Then, new experiments will be possible. Moreover, we could check that the approach is able to provide supportto auser, speciallyifno datais available toformthe behavior modelandto estimate probability values.Inthis case,followingtheMCAparadigm,suchvaluescouldbeformedbyaspecialistinthedomain,usingtheuser-friendlyHCM- LModelertool,minimizingthecold-startproblemfacedbydata-driven-onlyapproaches.
7. Relatedwork
Mostof therelated research that offers some kindof supportto a user doit purely through data-driven techniques.
As identifiedthrough a broadstudy done by Ranasinghe etal.[21] aboutactivityrecognition methods, thesetechniques (BayesianModels,MarkovModels,DecisionTrees,Artificial NeuralNetworks,SupportVectorMachinesandothers)mostly rely on low-level or raw data. In contrast to data-driven techniques, knowledge-driven techniques (ontology based ap- proaches,logic,orevidencetheory)useknowledgeprovidedbyan expert.Thepoint isthat, intheexisting bodyofwork, knowledge-driventechniques aremostlyusedto supportdata-driven ones.So, thisisthemain differenceofour workas weprovidesupporttoausertotallythroughknowledge-basedmeans.Belowisanoverviewofthestate-of-theartresearch whichisclosestinscopetoourproposal.
Rashidietal.[48,49]presentCASAS,anintegratedsetofcomponentsthataimtowards applyingmachine learningand datamining techniques to asmart home environment to detect activitypatterns, generateautomationpolicies for those patterns,and adapt to the changes in those patterns. Therefore,we cite theseworks asexamples ofpurely data-driven approaches.Inaddition,theyonlyaddressactivityrecognitiontasksanddefinenoclearwayforprovidingtheusersupport likeweproposeinthiswork.
Serraletal.[50,51]presenta knowledge-drivenapproach thatuses modelsatruntimeto solvethe cold-startproblem ofuserbehavior automationin aSmart Home,i.e., requiring alarge representative datasetto supportmodel training. In particular,they proposeacontext-adaptivetaskmodelandacontextmodelthataidsinspecifyingbehaviorpatterns.They alsodesignandimplementasoftwareinfrastructuretosupporttheautomationofthesebehaviorpatternsinthedeployed system.Ontheother hand,ourwork isdifferentbecausewe donotautomateactivities butratherassisttheuserineach actionwhennecessary.Also,theirreasoningisbasedonpredefinedrulesandthereisnouser-friendlytoolforcreatingthe models.
Chenetal.[52]introduceanontology-based hybridapproachby incorporatingdata-drivenlearningcapabilitiesintoan approachformodelingADLsandminimizingthefollowingproblemsofdata-driven-onlytechniques:(i)cold-startproblem;
(ii) low model applicability and reusability; and (iii) incompleteness of activitymodels. However, the authors only con- sidertheactivityreconditiontopicwithoutanypredictionaspectrelatedtoactions.Additionally,althoughtheontology is used,thereisnoreasoningoverit.Hence,thisworkisanexamplewhich,unlike ourmethod,onlyusesknowledge-driven approachassupportingtechniqueandnotasitscore.
Ceballosetal.[53] useBPMN tomodel theworkflow ofdaily activities andBayesiannetworksto predictactions and assistusers.Althoughthisapproachlookssimilartoourwork,theauthorsuseageneralpurposemodelinglanguage(BPMN) thatimpliesitsknowledgebythemodeler,thismaynotbetrueforacaregiver,adoctor,orarelative.Inourcase,theHCM-
Table 1
A short summary of human support approaches in the frame of Active and Assisted Living (AAL)
Authors Modeling Reasoning Application
[55] Probabilistic models Hierarchical Bayesian models Alzheimer’s patients [56] Ontology Web Language (OWL) A Middleware to define reasoning approaches Taking medications
[54] Ontology Web Language (OWL) Multi-Entity Bayesian Network Pervasive Application for Medicines Management [57] Activity Theory (AT) framework Causal Diagrams (CD) Medical consultation
[58] Answer Set Programming Answer Set Programming and Dempster-Shafer Cognitive support
LisaDSMLspeciallydesignedtobeusedby suchpeople.Inaddition,thepredictionisbasedonthedirectconversionof a BPMNmodeltoa Bayesiannetwork withouttheuseofa probabilistic model,asituationthat canbring inconsistencies intheinferenceprocess,asthemodelsemanticsislostinsuchconversion.Inourworkwe aimatpreservingthemodel’s semanticmeaninginthereasoningphase.
Machadoetal.[54]developanAALsystemthatmonitorsaperson’sactivitiesthroughsensorsandcomparesthesensor data to a context modelin order to figureout the situation,i.e., the currentstate ofthe environment. The systemuses a probabilistic semantic model for predictingunwanted situations andacts pro-actively to prevent such situations from arising.Theseproactiveactionsareexecutedinformofinteractionswiththeuser(e.g.visualandaudiowarnings,noticesto thecaregiver)thataretriggeredwhenthepossibilityofanundesiredsituationisdetected.Therefore,thisworkisdirectly relatedtoours, onceits predictionisbased onaprobabilistic semanticmodel,too.However, we focusoncovering every ADLofauserwhileMachadoetal.focusonunwantedsituationslikeincorrectormissedmedication,improperhandlingof gas,fire,electric,water,amongothers.Table1showsashortsummaryofhumansupportapproachesintheframeofActive andAssistedLiving(AAL).
8. Conclusion
In thisworkwe presented aset ofprobabilistic extensionsto the HBMSapproachthat aims, fora while,togo along a person’slifeinher/hislivingenvironmentwhile thepersoniscognitivelywell, inordertolearnandkeep his/herADLs knowledge.Theseextensionsformaprobabilistic semanticmodelwhichissubsequentlyrepresentedinontologicalformat (bymeansofPR-OWLconversion)toprovidefortool-supportedreasoningandinference.
When a mild signal of memory loss startsto happen then the system starts retrieving pieces of learned knowledge oftheperson’sbehavior. Theontologicalrepresentationoftheprobabilisticsemanticmodelsupports thisprocess through reasoningover uncertaintythatallow topredict themostlikelynext operationtosupport theusersi.e.help themtore- memberwhattheywere usedtodo.OurpropositionwastestedintheHBMSsmarthome prototypewhererealdatawas collectedtrough installablesensors andIoT devicesfromvolunteers who performedcommonADLssimulating aperson’s healthycognitivestate.Afterthat,we usedthisdatatotrainourprobabilisticsemanticmodelandthentotestthepredic- tionperformance throughtheholdout process,randomsamplingandtheperformance measures.Ourresultsshowedthat usingtheprobabilistic semanticmodeltomake predictionsispromising,speciallywhennodatais available.Inthiscase, adomainexpertcould formprobability valuesenablingthesystemtoworkevenwithoutdata.Thisisworth whena user who wasnot monitoredbythe systemalonghislife issubmitted tothesystem’sguidance.Ourcontributionsare: (i)the probabilisticsemanticmodelwhichcontainsadditionalattributesandrelationshipsallowingtoderiveassistancetotheuser withoutlosing semanticmeaning;and(ii)thearchitecturalframework allowing forconvertingthe semanticmodelintoa PR-OWLrepresentationtoprovidefortool-supportedprobabilisticreasoninginHBMSsystem.
Infuture,weintendtorefinetheprobabilisticsemanticmodelwithnewprioritymeasurestoincreasetheadaptability ofpredictions,andperformnewtestsusingrealcasescenarios.
Acknowledgments
The authors wouldlike to thank CAPES, CNPqand FAPERGS Brazilian research agencies as well as the Klaus Tschira Stiftung,Heidelberg,forpartiallysupportingthiswork.
References
[1] D. Foti , J.S. Koketsu , Activities of daily living, Pedretti’s Occup. Therapy: Pract. Skills Phys. Dysfunct. 7 (2013) 157–232 .
[2] H. Aloulou , M. Mokhtari , T. Tiberghien , R. Endelin , J. Biswas , Uncertainty handling in semantic reasoning for accurate context understanding, Knowl.-Based Syst. 77 (2015) 16–28 .
[3] C. Siegel , T.E. Dorner , Information technologies for active and assisted living—influences to the quality of life of an ageing society, Int. J. Med. Inform.
100 (2017) 32–45 .
[4] A .A . Chaaraoui , F. Florez-Revuelta , M. Harbach , A. De Luca , S. Egelman , Technologies and applications for active and assisted living. current situation, Pragmatics 27 (3) (2017) 447–474 .
[5] J. Michael , H.C. Mayr , Creating a domain specific modelling method for ambient assistance, in: Proceedings of the Fifteenth International Conference on Advances in ICT for Emerging Regions (ICTer), 2015, 2015, pp. 119–124 .
[6] H.C. Mayr , F. Al Machot , J. Michael , G. Morak , S. Ranasinghe , V.A. Shekhovtsov , C. Steinberger , HCM-L: domain-specific modeling for active and assisted living, in: Domain-Specific Conceptual Modeling, Springer, 2016, pp. 527–552 .
[7] H.C. Mayr , J. Michael , S. Ranasinghe , V.A. Shekhovtsov , C. Steinberger , Model centered architecture, in: Conceptual Modeling Perspectives, Springer, 2017, pp. 85–104 .
[8] J. Michael , C. Steinberger , V.A. Shekhovtsov , F. Al Machot , S. Ranasinghe , G. Morak , The HBMS Story, Enterp. Model. Inf. Syst. Archit. 13 (2018) 345–370 . [9] G.M. Lunardi, G.M. Machado, F.A. Machot, V. Maran, A. Machado, H.C. Mayr, V.A. Shekhovtsov, J.P.M. de Oliveira, Probabilistic ontology reasoning in ambient assistance: predicting human actions, in: Proceedings of the 2018 IEEE 32nd International Conference on Advanced Information Networking and Applications (AINA), IEEE, 2018, pp. 593–600, doi: 10.1109/AINA.2018.0 0 092 .
[10] D.L. McGuinness, F. Van Harmelen, et al., OWL web ontology language overview, W3C Recomm. (10) (2004) https://www.w3.org/TR/2004/
REC- owl- features- 20040210/ .
[11] K.B. Laskey , MEBN: a language for first-order Bayesian knowledge bases, Artif. Intell. 172 (2–3) (2008) 140–178 .
[12] R. Li , B. Lu , K.D. McDonald-Maier , Cognitive assisted living ambient system: a survey, Digit. Commun. Netw. 1 (4) (2015) 229–252 . [13] D. Monekosso , F. Florez-Revuelta , P. Remagnino , Ambient assisted living, IEEE Intell. Syst. 30 (4) (2015) 2–6 .
[14] S. Katz , Assessing self-maintenance: activities of daily living, mobility, and instrumental activities of daily living, J. Am. Geriatr. Soc. 31 (12) (1983) 721–727 .
[15] M.P. Lawton , E.M. Brody , Assessment of older people: self-maintaining and instrumental activities of daily living, Gerontologist 9 (3_Part_1) (1969) 179–186 .
[16] D. Burmeister , B. Altakrouri , A. Schrader , Ambient reflection: towards self-explaining devices, in: Proceedings of the LMIS@ EICS, 2015, pp. 16–20 . [17] A. Al-Fuqaha , M. Guizani , M. Mohammadi , M. Aledhari , M. Ayyash , Internet of things: a survey on enabling technologies, protocols, and applications,
IEEE Commun. Surv. Tutor. 17 (4) (2015) 2347–2376 .
[18] S. Mayer , A. Tschofen , A.K. Dey , F. Mattern , User interfaces for smart things–a generative approach with semantic interaction descriptions, ACM Trans.
Comput.-Hum. Interact. (TOCHI) 21 (2) (2014) 12 .
[19] S. Mayer , J. Hodges , D. Yu , M. Kritzler , F. Michahelles , An open semantic framework for the industrial Internet of Things, IEEE Intell. Syst. 32 (1) (2017) 96–101 .
[20] P.-Y.P. Chi , Y. Li , Weave: scripting cross-device wearable interaction, in: Proceedings of the 33rd Annual ACM Conference on Human Factors in Com- puting Systems, 2015, pp. 3923–3932 .
[21] S. Ranasinghe , F. Al Machot , H.C. Mayr , A review on applications of activity recognition systems with regard to performance and evaluation, Int. J.
Distrib. Sens. Netw. 12 (8) (2016) .
[22] V.A. Shekhovtsov , H.C. Mayr , S. Ranasinghe , J. Michael , Domain specific models as system links, in: Proceedings of the ER 2018 Workshops, Springer, 2018 . Accepted for publication
[23] C. Perera , A. Zaslavsky , P. Christen , D. Georgakopoulos , Context aware computing for the Internet of Things: a survey, IEEE Commun. Surv. Tutor. 16 (1) (2014) 414–454 .
[24] Q. Ni , A.B. García Hernando , I.P. de la Cruz , The elderly’s independent living in smart homes: a characterization of activities and sensing infrastructure survey to facilitate services development, Sensors 15 (5) (2015) 11312–11362 .
[25] G. Abowd , A. Dey , P. Brown , N. Davies , M. Smith , P. Steggles , Towards a better understanding of context and context-awareness, in: Proceedings of the Handheld and Ubiquitous Computing, 1999, pp. 304–307 .
[26] N.D. Rodríguez , M.P. Cuéllar , J. Lilius , M.D. Calvo-Flores , A survey on ontologies for human behavior recognition, ACM Comput. Surv. (CSUR) 46 (4) (2014) 43 .
[27] P. Moore , B. Hu , X. Zhu , W. Campbell , M. Ratcliffe , A survey of context modeling for pervasive cooperative learning, in: Proceedings of the 2007 First IEEE International Symposium on Information Technologies and Applications in Education, 2007 .
[28] O. Yurur , C.H. Liu , Z. Sheng , V.C.M. Leung , W. Moreno , K.K. Leung , Context-awareness for mobile sensing: a survey and future directions, IEEE Commun.
Surv. Tutor. (2016) 68–93 .
[29] C. Bettini , O. Brdiczka , K. Henricksen , J. Indulska , D. Nicklas , A. Ranganathan , D. Riboni , A survey of context modelling and reasoning techniques, Pervas. Mob. Comput. 6 (2) (2010) 161–180 .
[30] Y. Li , J. Chen , L. Feng , Dealing with uncertainty: a survey of theories and practices, IEEE Trans. Knowl. Data Eng. 25 (11) (2013) 2463–2482 . [31] P.C.G. Costa , Bayesian Semantics for the Semantic Web, George Mason University, 2005 .
[32] C. Howard , M. Stumptner , A survey of directed entity-relation–based first-order probabilistic languages, ACM Comput. Surv. (CSUR) 47 (1) (2014) 4 . [33] R.N. Carvalho , K.B. Laskey , P.C. Costa , PR-OWL 2.0–bridging the gap to OWL semantics, in: Uncertainty Reasoning for the Semantic Web II, Springer,
2013, pp. 1–18 .
[34] P.C. Costa , K.B. Laskey , PR-OWL: A framework for probabilistic ontologies, Front. Artif. Intell. Appl. 150 (2006) 237–249 .
[35] S. Matsumoto, R.N. Carvalho, P.C. Costa, K.B. Laskey, L.L. Dos Santos, M. Ladeira, There’s No More Need to be a Night OWL: on the PR-OWL for a MEBN Tool Before Nightfall, 2011.
[36] R.N. Carvalho , S. Matsumoto , K.B. Laskey , P.C. Costa , M. Ladeira , L.L. Santos , Probabilistic ontology and knowledge fusion for procurement fraud detec- tion in Brazil, in: Uncertainty Reasoning for the Semantic Web II, Springer, 2013, pp. 19–40 .
[37] H.C. Mayr , J. Michael , V.A. Shekhovtsov , S. Ranasinghe , C. Steinberger , A model centered perspective on software-intensive systems, in: Proceedings of the EMISA 2018, CEUR-WS.org, 2018, pp. 58–64 .
[38] Object Management Group, Meta Object Facility (MOF) Core, 2016. http://www.omg.org/spec/MOF/ , (Accessed August 31.08.2018).
[39] A.G. Kleppe , J. Warmer , J.B. Warmer , W. Bast , MDA Explained: The Model Driven Architecture: Practice and Promise, Addison-Wesley Professional, 2003 .
[40] N. Bencomo , R.B. France , B.H. Cheng , U. Aßmann , [email protected]: foundations, applications, and roadmaps, Lecture Notes in Computer Science, 8378, Springer, 2014 .
[41] T. Parr , The Definitive ANTLR 4 Reference, Pragmatic Bookshelf, 2013 .
[42] J. Michael , Using cognitive models for behavioral assistance of humans, it-Inf. Technol. 58 (1) (2016) 44–48 . [43] A.N. Leontyev , Activity, Consciousness, and Personality, Prentice-Hall, 1978 .
[44] A. Süssl, ADOxx based HCM-L Model transformation to OWL 2.0 (for Reasoning), Master Thesis, Universität Klagenfurt, 2018.
[45] R. Filman , T. Elrad , S. Clarke , M. Ak ¸s it , Aspect-Oriented Software Development, Addison-Wesley Professional, 2004 .
[46] J. Kienzle , W. Al Abed , F. Fleurey , J.-M. Jézéquel , J. Klein , Aspect-oriented design with reusable aspect models, in: Transactions on Aspect-Oriented Software Development VII, Springer, 2010, pp. 272–320 .
[47] P. Tarr , H. Ossher , W. Harrison , S.M. Sutton , N degrees of separation: multi-dimensional separation of concerns, in: Proceedings of the 1999 Interna- tional Conference on Software Engineering, IEEE, 1999, pp. 107–119 .
[48] P. Rashidi , D.J. Cook , Keeping the resident in the loop: adapting the smart home to the user, IEEE Trans. Syst. Man Cybern.-Part A: Syst. Hum. 39 (5) (2009) 949–959 .
[49] P. Rashidi , D.J. Cook , L.B. Holder , M. Schmitter-Edgecombe , Discovering activities to recognize and track in a smart environment, IEEE Trans. Knowl.
Data Eng. 23 (4) (2011) 527–539 .
[50] E. Serral , P. Valderas , V. Pelechano , Improving the cold-start problem in user task automation by using models at runtime, in: Information Systems Development, Springer, 2011, pp. 671–683 .
[51] E. Serral , P. Valderas , V. Pelechano , Addressing the evolution of automated user behaviour patterns by runtime model interpretation, Softw. Syst.
Model. 14 (4) (2015) 1387–1420 .
[52] L. Chen , C. Nugent , G. Okeyo , An ontology-based hybrid approach to activity modeling for smart homes, IEEE Trans. Hum.-Mach. Syst. 44 (1) (2014) 92–105 .
