IV User Study Design
In this section, I outline the experiment in which participants created the interactive visual thumbnails for data-driven journalism with VTComp(see section 3). Before conducting the final experiment, we conducted a couple of times pilot study with lab members of designing and data-journalism background at the preliminary stage to find out bugs and flaws in the tool. The motive behind conducting the experiment is to evaluate the system’s performance, effectiveness, and usability.
Figure 16: Experiment process for comparing performance between VTComp group and Sketch group.
upon three requirements of news articles: they should be 1) easy for a layperson to read and understand; 2) not too lengthy for use in a brief study; and 3) include at least one visualization that is easy to understand and conveys article’s main point. After careful examination of candi- date articles from our survey, we chose an article from Pew Research Center "public trust [6]".
Further for observing the user experience we divided participants into two groups, VTComp group and Sketch group. We equally divided participants into both groups. The process of performing experiment include: first, VTComp group participants first create the data-visuals by using VTComp which was followed by sketch app data-visuals creation.Similarly, in sketch group, participants first create data-visuals using sketch app then VTComp as shown in Figure 16. Further, we provided them iPad(a1670) and touch pencil to perform the tasks. There was no restriction to participants, they were free to use any kind of HRO, and other elements inside the VTComp as per their perception to express the article insights visually.
During the experiment, we asked participants to freely ask if they have any queries regarding the creation of visual thumbnails. The experiment for creating informative visuals took around 30 minutes for every single user excluded interview session and introduction session. Next, the Interview session on average took 8 minutes for every single participant. Although there was no time restriction, participants were free to create their effective, expressive and informative visual thumbnails from the article URL is given to them. After the experiment, we conducted a short exit survey in which the participants were asked to answer four questions using a 7-point Likert measurement scale and to write the reasons for their answers. The questions were: 1) The system interface was easy to learn? 2) The system interface was easy to use? 3) Did I enjoy generating visualizations using VTComp? and 4) I will recommend this system to others.
The answer to the above questions was very satisfactory, which I will describe in the result section in more detail. Finally, we conducted the semi-structured interview session where we get there valuable feedback(discussed in the results section). We are very thankful to participants
to devote their time for successful completion of the study.
Participants and Study Environment
We considered that acquiring a sufficient number of professionals’ thumbnails for our study seemed infeasible. We, therefore, recruited 20 participants(18 men, 2 women) at a university, expecting that they can provide us with fresh, new design ideas despite the limitations in their professional design skills. Their majors were (electrical and computer/human factors) engineer- ing, life science, and industrial design. We recruited participants by a personal approach to them and by using social media. We describe them our goal before recruited them, once people were willing to perform then we selected them. Their mean age was 27.65, and the SD was 4.39 (M: Mean = 27.83, SD = 4.58,F: Mean = 26, SD = 1). Time taken by experiment for every single participant was less than 1 hour. Each participant received 15$ in exchange for their par- ticipation in our experiment. The participants gave us their responses regarding the strengths and weaknesses of presently accessible visual creation tools. During we introduced VTComp to these participants and asked them to use it, the reaction we got was quite convinced, further, they remarked that VTComp was easy to operate tool which can be helpful for less technical and beginner in designing domain.
We used two iPad Pro (12.9 inches, resolution: 2388x1688px, 2nd Generation) and touch the pencil. For observing the participants’ behavior while experiment we use screen capturing tools inbuilt in iPad, with the help of this tool we analyze the design strategies and importance of various features that frequently used by participants. The results of this evaluation are shown in Figure22. We have used the third party plugin called mouseflow [95] for recording the user logs graphically and the result of Figure22 is also obtained from it. The semi-structured interview also recorded in written format to evaluate the strengths and weaknesses of the systems.
V Experiment Results, Analysis and Implications
Evaluation of the VTComp consists of multiple approaches. 1) We benchmark its correctness in terms of extraction of textual, tabular and graphical data. 2) We also compare its user experience to the structured visual creation approach. 3) We present examples where VTComp performs the tentative browsing to rule out documents that contain little or no interesting temporal information, we also evaluate the compatibility of VTComp by processing numerous data-driven news articles from various top news organizations including"The New York Times, The Economist, Pew Research center and etc." and generated visuals, VTComp walkthrough for designing interactive visuals process. Finally, 4) after the pilot study, we performed the final experiment study, where the feedback from participants was very positive.
Figure 17: The results of the benchmark tests, which compare the gold manual standard manual creation of an event set with automated temporal event extraction, which further used for creating visuals. In this figure, I have used five different data-driven news articles from several news agencies like the New York Times, Pew Research center and fiviThirtyEight.