E X P E R T O P I N I O N
Editor: Daniel Zeng, University of Arizona and Chinese Academy of Sciences, zengdaniel@gmail.com
Brain
Informatics-Based Big Data and the
Wisdom Web of Things
Ning Zhong, Maebashi Institute of Technology and Beijing University of Technology
Stephen S. Yau, Arizona State University
Jianhua Ma, Hosei University
Shinsuke Shimojo, California Institute of Technology
Marcel Just, Carnegie Mellon University
Bin Hu, Lanzhou University
Guoyin Wang, Chongqing University of Posts and Telecommunications
Kazuhiro Oiwa, National Institute of Information and Communication Technology
Yuichiro Anzai, Japan Society for the Promotion of Science
integrates brain-related big data and human behav-ior-related big data in a social-cyber-physical space to realize a harmonious symbiosis. Brain informat-ics provides the key technique for implementing such an attempt by offering informatics-enabled brain studies and applications in a social-cyber-physical space, thereby forming a brain big data cycle.2
This cycle is implemented by processing, in-terpreting, and integrating multiple forms of the “brain big data” obtained from molecular and neuronal circuitry levels via advanced neuroimag-ing technologies, such as functional magnetic res-onance imaging (fMRI), magnetoencephalography (MEG), electroencephalography (EEG), functional near-infrared spectroscopy (fNIRS), positron emission tomography (PET), and wearable, por-table micro and nano devices. Brain big data not only help scientists improve their understand-ing of human thinkunderstand-ing, learnunderstand-ing, decision mak-ing, emotion, memory, and social behavior, but such data also help cure diseases, assist in men-tal healthcare and well-being, and encourage fur-ther development of brain-inspired intelligent technologies.
Developing a Big Data Sharing Mind on the W2T
Various Internet of Things/Web of Things (IoT/ WoT) and cloud computing-based applications are accelerating the amalgamation of a social-cyber-physical space. As Figure 1 shows, the W2T is being developed as an extension of the wisdom Web with big data.1 Here, “wisdom” means that each thing in
the IoT/WoT model is aware of both itself and oth-ers to provide the right service for the right object at the right time and context. Researchers have pro-posed WaaS (Wisdom as a Service),3 an open and
in-teroperable intelligence service architecture for data, information, knowledge, and wisdom (DIKW).
We currently live in a huge network of numerous computing devices, measuring devices, and u-things, where real physical objects are attached, embedded, or blended with computers, networks, and other de-vices such as sensors. Adapting and utilizing this new kind of human-machine relationship and developing human-level collective intelligence have become tan-gible goals in DIKW-related research. But realizing these goals will depend on holistic intelligence re-search: the human brain must be investigated as an information-processing system, with big data help-ing us understand its capacities and limitations.4
Brain big data collected in the social-cyber-physical space and integrated with human behavior big data and worldwide knowledge bases could help us real-ize human-level collective intelligence as a big data
T
he Wisdom Web of Things (W2T) provides asharing mind—in other words, a har-monized collectivity of consciousness.5
If we could apply this concept to the Shannon-Weaver communication model in a social-cyber-physical space, the information in the sender’s mind would be converted into data, with the sender taking what’s necessary and dis-carding what isn’t. The sender’s data would then be transferred through channels to the receiver, who would decode the data, interpret their mean-ing, and generate new information in his or her mind. Thus, there’s no direct relation between the sender’s informa-tion and the receiver’s. If we assume that the productivity of society, com-munities, and people is proportional to the amount of information transferred, we can introduce an efficiency factor as a percentage of what ideally could be expected in communication. In machine-to-machine communication, for example, we expect efficiency to be 100 percent, but in human-to-human communication, efficiency varies from minus infinity to plus infinity. As infor-mation communication becomes faster, larger, and more ubiquitous, while still requiring dependability, it’s crucial to address qualitative problems in human communication to convey real, mean-ingful, and understandable messages without restriction.
Let’s consider some scenarios. We start with an ambiguous figure, where some 3D information deteriorates into 2D—even if you and I see the same figure, I can’t tell what feelings your consciousness brings up or how you actually see the image.6 Here’s another
one: if you draw a caricature of a poli-tician, its data size can be compressed into something much smaller than a high-density (HD) photo, but the ef-fect it elicits is much larger than an HD photo. Finally, when we observe a hid-den figure in a visual image degraded by monochromatic binarization, we see only meaningless patterns until
after some seconds pass and we sud-denly perceive a meaningful object.6
All these experiences raise a question: What’s the essence of understanding? Even with a small amount of infor-mation, efficiency can compensate for productivity. The study of human-to-human communication must consider the process by which inner thought, emotion, belief, and self-awareness are coded into language prior to being sent. Especially in human-to-human communication, inspiration and cre-ativity are quite effective. These obser-vations lead to the complex research issues of communication in the social-cyber-physical space.
Heart-to-heart science (HHS) in-volves research and development that assist people in understanding the meaning of words and recognizing the context of information by scientifi-cally analyzing the higher-level brain functions related to the understanding, recognition, and effect that lie at the core of communication. When ambig-uous or incomplete information is pre-sented, a computer can’t understand
the meaning—but if humans see the same thing, they can guess the meaning through inspiration. The “inspiration of awareness” is key to improving the efficiency of information transfer. For information communication technol-ogy (ICT), especially for communica-tion between humans in a social-cyber-physical space, we must improve effi-ciency. Brain informatics provide op-portunities for improvement by help-ing us understand and apply how the brain identifies the “heart” of a piece of information, as well as helping us develop the brain-inspired W2T tech-nology required for communicating only true information—namely, send-ing only what we need to send and re-ceiving only what we need to receive.
Network-Based Big Data in a Social-Cyber-Physical Space
The relation between neuroscience and big data has several interactions. Brain- function measurements such as fMRI, PET, MEG, and EEG generate big data for information networks, and sensor
Figure 1. The Wisdom Web of Things (W2T) in the social-cyber-physical space. Here, “wisdom” means that each thing in the Internet of Things/Web of Things (IoT/WoT) model is aware of both itself and others to provide the right service for the right object at the right time and context.
Social world
Internet
Internet of Things Web/ Web of Things
Physical world
Big data
Wisdom as a Service (WaaS)
Knowledge as a Service (KaaS)
Information as a Service (InaaS)
Data as a Service (DaaS)
Individuals Companies/Societies
Wisdom Web of Things (W2T)
Cyberworld
Cloud
computing architect
networks generate human daily life and behavior data (another form of big data). Wearable sensors attached to a person continuously send health infor-mation, such as heart rate, blood pres-sure, blood glucose levels, and so on, to hospitals or doctors. These technol-ogies accurately record daily lives and social behavior as digital data logs that can be utilized for various analyses and studies. Both types of data offer a stim-ulus set for brain researchers. Network science provides analysis for network-based big data in a social-cyber-phys-ical space; its methods are also appli-cable in brain network analysis.7,8
Because big data implicitly includes ensemble behavioral data from the people inside a social-cyber-physical space, human behavior rules or struc-tures can be extracted from them and could reflect human brain functions. The behavior of complex dynamic systems has been extensively studied in mathematics, biology, and complex system sciences, and combining these studies with big data has provided important insights into ensemble be-havior. For neuro and cognitive sci-ences in particular, the utilization of big data stimulus sets has provided new ways to better understand human brain functions and mechanisms.9,10
For example, neuroeconomical studies
on decision making in a social con-text have revealed the network of brain regions responsible for social decision making. Many of these stud-ies have been carried out using be-havioral economics games.11 By using
these games, our research group aims to construct a computational model of human decision making that lets us predict future behaviors.12 We’re
par-ticularly interested in decision mak-ing in social settmak-ings, individual dif-ferences in decision making, and the learning mechanism of decision mak-ing. Our work should provide new in-sights for big data analysis, especially for social media.
MRI measurements provide a pow-erful noninvasive approach for in-vestigating brain networks, enabling inspections on the large amount of nodes and connections within the hu-man brain from both functional and structural views. Network analysis on the relationship between nodes helps build a model that autonomously re-duces the degrees of freedom.13 The
brain is an ensemble of transfer tions: we want to know these func-tions, the way information is repre-sented in the brain, and the transition between unconscious and conscious. For studies on brain functions, various stimuli (visual or auditory stimuli are
useful as stimulus sets) are applied to subjects, and the induced brain activ-ity can be detected by functional MRI measures. Moreover, human thought is the product of multiple collaborat-ing brain centers linked by white mat-ter tracts—thinking is a network func-tion, and white matter is the unsung hero that serves as the structural net-work to underlie human thought.14,15
This neural infrastructure can be cap-tured by another type of MRI mea-sures—diffusion tensor imaging.
To analyze the relationship between neural activity inputs and patterns, we can estimate the transfer function by using machine learning techniques. Shinji Nishimoto and colleagues col-lected various types of movie clips from the Internet and used them as stimulus sets.16 The fMRI data from when
sub-jects watched the movies were analyzed as evoked patterns in the visual cortex via machine learning, ultimately lead-ing to a reconstruction of visual experi-ences from brain activity.
How should we collect neural ac-tivity? To combine noninvasive mea-surement systems and ordinary human activity, we need to connect the daily-life environment in these measurement systems and bring brain activity mea-surements into our social and daily lives. Although fMRI and MEG are versatile and precise imaging methods that noninvasively measure and image human brain activities, they require an electromagnetic shield and vibra-tion isolavibra-tion system for accuracy at reasonable temporal resolutions, but these requirements make any result-ing systems far from portable. In addi-tion, fMRI and MEG are too large to be moved, highly constraining the sub-jects under study and removing them from their daily life. We need a simple and mobile measuring system of neu-ral activity that can be combined with measurements and records of other physical activities (see Figure 2). Simple and mobile
neuro- and daily-life activities measuring system
Link between lab and daily life
Collection of activity in daily life Wearable interfaces
Activity data Daily activities Economic activities Human relations Unbiased/biased
stimuli for experiments
EEG/NIRS unconstraint conditions
Non invasive measuring systems
MEG / fMRI high-constraint conditions
The other option is to establish daily lives in fMRI. Because an MRI ma-chine is very noisy, subjects inside it re-quire ear plugs. To address this prob-lem, two fMRIs with a tube equipped by microphones can facilitate natural dialog inside the MRI machine. For vi-sual stimulation, 3D images and movies can be projected in the machine. Ma-sahiko Haruno and Christopher Frith used dictator games to classify subjects as prosocial or selfish, measuring their brain activity via fMRI as the subjects rated the desirability of different re-ward pairs for themselves and others on a scale from one to four.12 Such studies
pave the way for balancing precise lab-oratory measurements with activities of daily life.
Brain Big Data-Based Wisdom Services
To demonstrate brain big data in W2T applications, Figure 3 gives an out-line of a smart hospital service system for brain and mental disorders. Based on a previous prototype of a portable brain and mental-health monitoring system,3,19 the development of such a
smart hospital service system must also consider various system- and content-level demands. For example, it has to effectively integrate multilevel brain and mental health big data and pro-vide multi-level and content-oriented services for different types of users through an open and extendable in-terface. The system is based on a WaaS architecture with a variety of data ac-quisition devices, a brain datacenter, and the LarKC semantic cloud plat-form.3, 17, 18, 20, 21 Three types of data
must be collected from patients:
• physiological data, based on wear-able systems;
• behavior data, based on wearable and monitoring systems; and
• data from traditional diagnosis and physiological measurements (including
fMRI, PET, MEG, EEG, and eye-tracking), based on scale ratings. Various wearable health devices, such as the EEG belt, voice, heart rate, and tremor monitors, and a sleep-monitoring system, are used as new physical examination devices to obtain macro- and mesolevel brain data. These wearable health data are analyzed and integrated with clini-cal physiclini-cal examination data, as well as various medical information and knowledge sources, such as medical re-cords, experimental reports, and LOD (linked open data) medical datasets. These integrated sources are also combined with personalized models of patients to DaaS (Data as a Ser-vice), IaaS (Information as a SerSer-vice), KaaS (Knowledge as a Service), and WaaS (Wisdom as a Service) to vari-ous types of users.3
A powerful brain datacenter is the global platform that supports the whole systematic brain informatics re-search process and its real-world ap-plications.2,22 As the core of the brain
big data cycle system, the Data-Brain represents a radically new way of storing and sharing data, information, and knowledge; it also enables high-speed, distributed, large-scale, multi-granular, and multimodal analysis and computation on the W2T.17,18,22
Emotional robotic individual as a novel u-thing views emotion and cog-nition as a starting point for the devel-opment of robotic information pro-cessing and personalized human-robot interaction on the social-cyber-phys-ical space of the W2T with brain big data.1,23 A cyberindividual model is
created by collecting brain data and so-cial behavior data from a specific user, along with basic robotic emotional and cognitive capabilities, such as per-ception processing, attention alloca-tion, anticipaalloca-tion, planning, complex motor coordination, reasoning about other agents, and perhaps even reason-ing about its own mental state.21,23 The
emotional robotic individual embod-ies the user’s behavior in the physical world (or the cyberworld, in the case Physical world
(Thing)
Data acquisition WaaS architecture
Wisdom as a Service (WaaS)
Knowledge as a Service (Kaas)
Information as a Service (Inaas)
Data as a Service (Daas)
LarKC cloud service paltform
Brain datacenter
Service objects
Depressives Families
Clinicians
Psychiatrists
Researchers
Rehabilitation physicians Maintenance
personnel Nurses
Monitoring system
Wearable system
Patient information Congnitive
games trackingEye fMRI EEG
Traditional diagnosis
Cyber world (Computer)
Social world (Human)
of simulated cognitive robotics). Ulti-mately, the robot must be able to act in the real world and interact with a specific user (such as a patient with depression) at a hospital or at home, to help the person with psychological treatment and rehabilitation.
Five Guiding Principles
To prepare for the massive progress to come in brain big data in the social-cyber-physical space, technological in-novation is, of course, necessary, but it isn’t sufficient on its own—we need a deeper understanding of how to col-lect brain and behavioral data and the nature of human perception and be-havior. To this end, we propose five guiding principles.
First, the “dynamic link across brain-body-world” is crucial. Think of how the “gaze cascade” effect illustrates how gaze shift uses perceptual process-ing and facilitation to form a dynamic positive-feedback loop toward a con-scious decision of visual preference.24
Second, the implicit cognitive pro-cess (“tacit knowing”) must be under-stood. The brain/mind processes that we’re consciously aware of are just the tip of the iceberg—the vast ma-jority of neural processing remains implicit, including the critical mecha-nisms underlying decision making.
Third, perception and action are in-teractive and ubiquitous. For example, in the “snake illusion,” an entirely static image gives a strong impression of motion (rotations, in this case). While the critical factors and underlying mechanisms are still under debate,25 it’s
obvious that the observer has a vigor-ous reaction.
Fourth, predicting the past is easy, but predicting the future is hard: human be-havior and decisions are strongly situa-tion-dependent. Many studies indicate that the data obtained from the same brain with the same stimulus materi-als and tasks are necessary to perform
reliable decoding, including our own EEG study of decoding facial prefer-ence.26 Expect a reasonable and feasible
ethical border here, to deal with related issues in the near future.
Finally, we want to emphasize that creativity is waiting out there. Con-sider the following tale. Two towns faced each other across a very deep valley, and people traveled hundreds of miles of mountain roads to trade with each other. Constructing a bridge seems like an obvious solution, but until one rich man came up with the idea and implemented it, nobody ever thought of it—you might consider this a creative insight. You could also ar-gue that the environment (the land-scape) was structured such that this creative idea was “implicitly waiting.”
T
o conclude, technical attempts and discussion around brain big data should be based on the keen realization of the vigorous interaction and interde-pendence of brain-body-environment. A brain information-based wisdom service cloud platform must be developed to realize human-level col-lective intelligence as a big data shar-ing mind—a harmonized collectiv-ity of consciousness on the W2T that uses brain-inspired intelligent technol-ogies to provide wisdom services.Acknowledgments
This work was supported by grants from the National Basic Research Program of China (2014CB744600), the International Science & Technology Cooperation Program of China (2013DFA32180), the National Natural Science Foundation of China (61420106005 and 61272345), the Beijing Natural Science Foundation (4132023), and the JSPS Grants-in-Aid for Scientific Research of Japan (26350994).
References
1. N. Zhong et al., “Research Challenges and Perspectives on Wisdom Web of
Things (W2T),” J. Supercomputing, vol. 64, no. 3, 2013, pp. 862–882.
2. N. Zhong et al., “Brain Informatics,” IEEE Intelligent Systems, vol. 26, no. 5, 2011, pp. 16–21.
3. J. Chen et al., “WaaS: Wisdom as a Service,” IEEE Intelligent Systems, vol. 29, no. 6, 2014, pp. 40–47.
4. D. Douglas, “The Limits of Intelligence,” Scientific Am., July 2011, pp. 37–43. 5. F. Heylighen, “The Global
Superorganism: An Evolutionary-Cybernetic Model of the Emerging Network Society,” Social Evolution & History, vol. 6, no. 1, 2007, pp. 58–119. 6. T. Murata et al., “Stochastic Process
Underlying Emergent Recognition of Visual Objects Hidden in Degraded Images,” PLoS One, vol. 9, 2014; doi: 10.1371/journal.pone.0115658. 7. O. Sporns, “Making Sense of Brain
Network Data,” Nature Methods, vol. 10, no. 6, 2013, pp. 491–493.
8. H.-J. Park and K. Friston, “Structural and Functional Brain Networks: From Con-nections to Cognition,” Science, vol. 342, 2013; doi: 10.1126/science.1238411. 9. T. Horikawa et al., “Neural Decoding of
Visual Imagery during Sleep,” Science, vol. 340, 2013, pp. 639–642.
10. T. Cukur et al., “Attention during Natural Vision Warps Semantic Representation across the Human Brain,” Nature Neuroscience, vol. 16, 2013, pp. 763–770. 11. V.K. Lee and L.T. Harris, “How Social
Cognition Can Inform Social Decision Making,” Front Neuroscience, 2013; doi: 10.3389/fnins.
12. M. Haruno and C. Frith, “Activity in the Amygdala Elicited by Unfair Divisions Predicts Social Value Orientation,” Nature Neuroscience, vol. 13, 2010, pp. 160–161.
13. N. Turk-Browne, “Functional Interactions as Big Data in the Human Brain,” Science, vol. 342, 2013, pp. 580–584.
15. T.A. Keller and M.A. Just, “Altering Cortical Connectivity: Remediation-Induced Changes in the White Matter of Poor Readers,” Neuron, vol. 64, no. 5, 2009, pp. 624–631.
16. S. Nishimoto et al., “Reconstructing Visual Experiences from Brain Activity Evoked by Natural Movies,” Current Biology, vol. 21, no. 19, 2011, pp. 1641–1646.
17. G.Y. Wang and J. Xu, “Granular Computing with Multiple Granular Layers for Brain Big Data Processing,” Brain Informatics, 2014; doi: 10.1007/ s40708-014-0001-z.
18. G.E. Hinton and R.R. Salakhutdinov, “Reducing the Dimensionality of Data with Neural Networks,” Science, vol. 313, 2006, pp. 504–507.
19. B. Hu et al., “EEG-Based Cognitive Interfaces for Ubiquitous Applications: Developments and Challenges,” IEEE Intelligent Systems, vol. 26, no. 5, 2011, pp. 46–53.
20. D. Fensel et al., “Towards LarKC: A Platform for Web-Scale Reasoning,” Proc. Int’l Conf. Semantic Computing, 2008, pp. 524–529.
21. J.H. Ma et al., “Cyber-Individual Meets Brain Informatics,” IEEE Intelligent Systems, vol. 26, no. 5, 2011, pp. 30–37. 22. N. Zhong and J. Chen, “Constructing a
New-Style Conceptual Model of Brain Data for Systematic Brain Informatics,” IEEE Trans. Knowledge and Data Eng., vol. 24, no. 12, 2012, pp. 2127–2142. 23. Y. Anzai, “Human-Robot Interaction
by Information Sharing,” Proc. Human-Robot Interaction, 2013, pp. 65–66. 24. S. Shimojo et al., “Gaze Bias Both Reflects
and Influences Preference,” Nature Neuroscience, vol. 6, 2003, pp. 13–17. 25. I. Murakami, A. Kitaoka, and H.
Ashida, “A Positive Correlation between Fixation Instability and the Strength of Illusory Motion in a Static Display,” Vision Research, vol. 46, no. 15, 2006, pp. 2421–2431.
26. J.P. Lindsen et al., “Neural Components Underlying Subjective Preferential
Decision Making,” NeuroImage, vol. 50, 2010, pp. 1626–1632.
Ning Zhong (corresponding author) is the head of the Knowledge Information Systems Laboratory and a professor in the Depart-ment of Life Science and Informatics, Mae-bashi Institute of Technology, Japan. He’s also the director and an adjunct professor in the International Web Intelligence Consor-tium (WIC) Institute at the Beijing University of Technology. His research interests include Web intelligence, brain informatics, data mining, granular computing, and intelligent information systems. Zhong has a PhD in the interdisciplinary course on advanced science and technology from the University of Tokyo. Contact him at zhong@maebashi-it.ac.jp
Stephen S. Yau is a professor of computer science and engineering and director of the Information Assurance Center at Arizona State University. His research interests in-clude cyber trust, cloud computing, soft-ware engineering, service-based systems, and parallel and distributed computing sys-tems. Contact him at yau@asu.edu
Jianhua Ma is a professor in the Faculty of Computer and Information Sciences, Hosei University, Japan. His research interests in-clude smart worlds and ubiquitous/perva-sive intelligence. Ma has a PhD in informa-tion engineering from Xidian University. Contact him at jianhua@hosei.ac.jp
Shinsuke Shimojo is a Gertrude Baltimore Professor of Experimental Psychology at the California Institute of Technology. His re-search interests include visual perception and its neural correlates, multisensory inter-actions and plasticity, emotion and decision making, and noninvasive modulation of the alert human brain. Shimojo has a PhD from MIT. Contact him at sshimojo@caltech.edu.
Marcel Just is a D.O. Hebb University Profes-sor in Cognitive Neuroscience, director of the Center for Cognitive Brain Imaging, and direc-tor of the Scientific Imaging and Brain Research
Center at Carnegie Mellon University. His research uses fMRI and other technologies to uncover the structure of human thought. Contact him at just@cmu.edu.
Bin Hu (corresponding author) is a profes-sor in and dean of the School of Information Science and Engineering, Lanzhou Univer-sity, China. His research interests include pervasive computing, computational psycho-physiology, cooperative work, and the se-mantic Web. Contact him at bh@lzu.edu.cn.
Guoyin Wang is a professor in and execu-tive dean of the College of Computer Science and Technology, the Chongqing University of Posts and Telecommunications, and the direc-tor of the Institute of Electronic Information Technology, Chongqing Institute of Green and Intelligent Technology, CAS, China. His research interests include rough sets, granular computing, knowledge technology, data min-ing, machine learnmin-ing, neural networks, soft computing, and cognitive computing. Wang has a PhD in computer organization and ar-chitecture from Xi’an Jiaotong University. Contact him at wanggy@cqupt.edu.cn.
Kazuhiro Oiwa is a distinguished re-searcher at the Advanced ICT Research In-stitute, National Institute of Information and Communications Technology, Japan. His research targets span biomolecule to cellular network levels by measuring, ana-lyzing, and controlling the wide range of bi-ological materials to understand and recon-struct biological functions. Contact him at oiwa@nict.go.jp.
Yuichiro Anzai is president of the Japan Society for the Promotion of Science. His re-search focuses on cognitive and information sciences. Contact him at anzai@jsps.go.jp.
