THE USED OF IT BALANCED SCORECARD TO BUILD THE
PERFORMANCE MEASUREMENT MODEL OF ACADEMIC
INFORMATION SYSTEMS (CASE STUDY ACADEMIC
INFORMATION SYSTEM OF SATYA WACANA)
Paskah Ika Nugroho,
Faculty of Economics and Business
Satya Wacana Christian University, Indonesia.
Prihanto Ngesti Basuki,
Faculty of Information Technology
Satya Wacana Christian University
Indonesia.
Evi Maria,
Faculty of Information Technology
Satya Wacana Christian University
Indonesia.
ABSTRACT
The aim of this research is to make a model of performance measurement of academic information system to facilitate the auditors in conducting a periodically performance measurement of Satya Wacana Academic Information System using IT Balanced Scorecard. SI performance measurement model was developed through systematic measures in the form of the action process, reflection, evaluation, and innovation by applying the method of survey research, development, experiments , and evaluation. Performance measurement modeling of Academic Information Systems (SIASAT) in SWCU has been done by making a framework model which was developed by considering the following parameters: (a) the duties and functions of the university, (b) the aspects of university management, (c) the duties and functions of the IT organization in university, (d) the need of information system for academic activities, and (e) the methodology of IT basic framework used, which is the IT Balanced Scorecard (IT-BSC).
Introduction:
The use of Information Technology (IT) in Higher Education institutions especially for the use of information systems and the Internet can not be separated due to the demands of the stakeholders (Indrajit, 2006). IT Management in Higher education institution is a Critical Success Factor (CSF) for leaders and partners of Higher education institutions (Henderi, 2010). However, the complexity of IT implementation makes the leaders of the various levels in the Higher education institutions and stakeholders have difficulty in managing the IT. The complexity of IT implementation in higher education institutions in Indonesia happens because the higher education institution does not have a specific framework model when establishing the information system (Mutyarini and Sembiring, 2006). As a result, the benefits of using IT is not comparable to the investments value which has already been incurred.
Satya Wacana Christian University is one of the universities, which has already used IT as an infrastructure and facility to provide services for students, lecturers and all the staff, and also assists the running of the activities around the work units. In carrying out its main activity, that is to provide educational services, SWCU has supported by IT of Satya Wacana Academic Information Systems (SIASAT). IT management has been applied in SWCU, but it has not been applied using a well-structured method and approach. On the other hand, IT implementation must be controlled because the control provides reasonable assurance to management that the implementation process has been done in accordance with the plans and goals of the organization (Maria, 2011).
Each IT process requires a controlled IT measurement to indicate the performance of IT in achieving the control objectives and facilitate the management to make improvements to the performance of IT. IT performance measurement can be performed by using IT Balanced Scorecard IT where the IT performance is measured from 4 perspectives: corporate contribution, user orientation, operational excellence, and future orientation (Van Grembergen, 2000). IT Balanced Scorecard is an effective method of managing IT organizations as well as evaluating the success and development of the system/application, the development of computer and network investment, quality of products and IT services, as well as improving the quality of human resources, even though, most universities in Indonesia have not been using this method (Prabowo, 2007).
In addition, monitoring and evaluating towards SIASAT performance has not been done periodically, but only if there are complaints from the working units about the SIASAT service (Maria and Haryani, 2011). This condition is not consistent with the results of Maniah and Surendro’s research (2005) which stated that SI performance measurement must be done periodically to ensure the sustainability of IT operations used by the organization or company as well as to assess the sustainability between the planning and implementation of the system. Since the importance of SI performance measurement should be done periodically, this research will try to make a model of performance measurement of academic information system to facilitate the auditors in conducting a periodically performance measurement of SIASAT using IT Balanced Scorecard.
Literature Review: IT Balanced Scorecard:
The balanced scorecard can be applied to the IT function and its processes as Gold (1994) and Willcocks (1995) have conceptually described and has been further developed by Van Grembergen and Van Bruggen (1997) and Van Grembergen and Timmerman (1998). IT-BSC has four perspectives: (1) Corporate Contribution, contains a measure which indicates how the management (the manager) evaluates/views the IT organization; (2) User orientation contains a measure which indicates how users evaluates/sees the results of the IT organization, (3) Operational Excellence contains a measure of the effectiveness and efficiency of the IT process, and (4) Future orientation contains a measure which describes how IT position within the next challenge.
Performance Measurement:
Mulyadi (2001) defines performance measurement as a process of assessment on the company operational activities in a particular period, whether it has been done based on the defined goals or not. The main purpose of the performance measurement is that the leader of the company has an objective basis in giving the compensation in accordance with the achievement which has been done by each department as a whole. It is expected that all of these will give motivation and stimulation in each section to work more effectively and efficiently.
Previous Researches:
IT model development and performance measurement can adopt IT standards such as ITIL, ISO/IEC 17799, COSO dan
in the business competition (Sa'adi and Suhardi, 2006).
IT Balanced Scorecard has not been widely used to measure the performance of information systems in universities in Indonesia, but actually this method is very effective for managing the IT organization and evaluating its success (Prabowo, 2007). This is because the Universities in Indonesia did not have a specific model of the framework when they build their academic information system (SI), so Mutyarini and Sembiring (2006) created an academic architecture Information system model by adapting the architecture of Monash University which used TOGAF in order to achieve the mission of Tri Dharma higher education.
The previous research on IT in SWCU as the research object, including the study of Maria and Haryani (2011) who found that the supervision and the assessment towards the IT performance in SWCU has not been carried out periodically, just only if there are complaints from the users (the working units) about the IT service. This research produced a model of information audit system
which is developed using the COBIT framework especially for delivery and support (DS) domain. Maria‘s research (2011) also
found that so far the IT management in SWCU has been done, but it has not been done using the structured method and approach. This research also choose to use COBIT framework in doing the comparison among the academic information systems because COBIT can notice the link between the business goals without neglecting the IT process as the focus. Maria’s research, et al (2012) found that IT in SWCU has been well managed where IT processes to support business goals has been standardized, documented and communicated well. There should be a continuous monitoring and evaluation of the IT in SWCU, so the quality of IT services in SWCU can be improved day by day in accordance with what is expected.
Research Methodology:
This type of research is a combination of descriptive studies that describes the phenomenon that actually occurs in an event or population and exploratory research that found a model of the SI Academic performance measurement done by doing an approach on the "Research and Development", that was a research program which was followed up by doing some development programs. SI performance measurement model was developed through systematic measures in the form of the action process, reflection, evaluation, and innovation by applying the method of survey research, development, experiments , and evaluation.
The location of this study, Satya Wacana Christian University Salatiga Indonesia, was chosen on purpose. Primary data of this study was the results of guided interviews and observation. While secondary data such as documents, reports and policies are taken based on the SIASAT
The steps of this study are as follow:
a. Preliminary studies
In the initial study, there were prelimanary research on previous studies, literature and standards that support the research topic, guided questionnaire drafting, and SIASAT understanding.
b. Data collection
At this stage, the data was obtained by interview, observation, and questionnaires given to the relevant units and users of SIASAT. The secondary data is also collected from related units of SIASAT.
c. Development of performance measurement model of IS
At this stage, development of performance measurement of IS was managed by interviews, observation and related documents to state parameters and Critical Success Factor (CSF), which will be used as constraints to determine criteria of performance measurement of SIASAT based on IT BSC perspective. Then we mapped the steps to measure the performance of acedemic information systems.
d. Conclusions
In the final stage of this research, a conclusion from all research processes was stated.
Result and Discussion:
Brief description of the Satya Wacana Academic Information Systems:
The internet-based of SWCU Academic Information System (SI), known as SIASAT , is an application used to record data from each student's academic administration from the entry (admission) to the exit (graduation). This application can be accessed easily via the SWCU homepage, http://www.uksw.edu address, then go to the SIASAT menu in the ACADEMIC group, or directly go to http://siasat.uksw.edu. These applications provide an online and a real time of academic information. All students who are listed as SWCU students, have the right to access the application via the homepage institution. It is important for SWCU students to know and master this application and its operations in order to see the financial obligations that they have to pay, course registration, and see the results of their study for each semester.
SIASAT Performance Measurement Model by Using IT Balanced Scorecard:
which was developed by considering the following parameters: (a) the duties and functions of the university, (b) the aspects of university management, (c) the duties and functions of the IT organization in university, (d) the need of information system for academic activities, and (e) the methodology of IT basic framework used, which is the IT Balanced Scorecard (IT-BSC). These parameters are expected to be the factors that determines the performance of academic information systems which were observed, and how these parameters can be controlled and regulated, in order to obtain a desired performance system. The relationship between the 5 parameters in creating the measurement model of academic information system performance by using IT-BSC is presented in Figure 1.
Figure 1. The relationship between the parameters in creating the performance measurement model
The main parameters of duties and functions of university in this research is the implementation Tridharma High Education, such as lectures, working in laboratory, practical work, the implementation of the final project, research and training, and the implementation of community service. Those are the things that encourage the chief of SWCU to formulate its business goals by using four perspectives of Balance Scorecard (BSC). The business goals of SWCU are presented in Table 1. To achieve those business goals, IT infrastructure is provided in the form of the use of computers, information systems implementation, and the use of internet technology. The information system was built in accordance with the internal business processes of SWCU starting from prospective new students since they enroll, be accepted, join the lectures, until graduate.
Table 1. Business goals SWCU
BSC Perspective SWCU Business Goals
Financial
1 Provide a good return on investment of IT-enabled business investments
2 Manage IT-related business risk
3 Improve corporate governance and transparency.
Customer
1 Improve customer orientation and service
2 Offer competitive products and services
3 Establish service continuity and availability
4 Create agility in responding to changing business requirements
5 Achieve cost optimisation of service delivery
Internal
1 Improve and maintain business process functionality
2 Lower process costs
3 Provide compliance with external laws, regulations and contracts
4 Provide compliance with internal policies
5 Improve corporate governance and transparency
6 Manage business change
7 Improve and maintain operational and staff productivity
Learning &Growth Manage product and business innovation
manifested in the form of organization structure and management systems in universities as presented in Table 2.
The parameters of IT management organization in SWCU are handled by the Bureau of Information Systems Technology (BTSI). IT developments are constantly increasing, so that SWCU must make arrangement in the IT management organization. BTSI plays an important role for the success of the implementation process. This is because BTSI not only manage the technical aspects of IT but also play a role in determining the organizational culture in SWCU in using IT. BTSI consists of 2 parts: (1) IT section which is in charge of audio-visual section, parts of the communication network and the Internet, and parts of computer, (2) information system section which is in charge of software parts, parts of information systems management, parts of flexible learning & web, and parts of documentation and training. The functions which are managed by BTSI are: (1) the function of technology development and the application of information systems, (2) the function of maintenance of information systems applications, database, digital documentation of information system, the content of learning resources, networks and computers, (3) the functions of settings and monitoring of IT implementations in the form of change management and user relationship, release system and audit.
Table 2. The key aspects of Universitites Management
Organizational Culture, Values and Work Ethics
The key process is to run Tri Dharma University, which consists of: Supporting processes, which include the
processes of: academic administration, students and alumni, financial administration, cooperation and external relations, and promotion
Information Systems Knowledge Management both tacit
knowledge and explicit knowledge.
Organizational Structure and Management Systems in Universities
On the other hand, the parameters of the IT framework methodology used in this study use IT Balanced Scorecard (IT BSC). Based on those IT organization functions in SWCU, it is necessary to establish IT Balanced Scorecard for each of those functions which is essentially a derivative of the IT Balanced Scorecard for universities. The process of identifying Critical Success Factor (CSF) is needed to determine the limits of performance measurement criteria in each IT process. The CSF of Academic Information Systems is presented in Table 3.
Table 3. The Critical Success Factors of Academic Information System based on IT-BSC Perspective
IT-BSC Perspective CSF Academic Information System
Business Contribution Control costs, increase revenue and improve service coverage
User Orientation Customer value proposition that includes the rates, quality, service provided, service
and partnerships
Operational Excellence Improvement of internal processes by implementing the operations management,
customer management and innovation.
Future Orientation Enhanced capabilities and skills through the strengthening of human capital,
strengthening of information capital, and strengthening of organization capital
For the parameters of the IS (Information System) requirements related to academic activities of universities which consist of information about admissions, student registration, course registration, grades, and graduation. Admission information consists of information regarding enrollment of prospective new students which covers registration process, the data inputting, photo-taking, selection, selection, announcement of selection result, printing of Rector Decree about new admission and information regarding re-signing up includes taking an acceptance letter, registration payment, informing a bank payment receipt, filling out a registration form,
and obtaining student’s number. The importance of IS related to student registration such as providing information about types of registration, information of procedures, requirements, and student registration deadline. Student registration is an activity of registration or recording of active-status as the University's student and must be done by students each semester.
Course registration is a subject registration process as a participant of a course in the current semester. Subject registration process includes academic supervision, financial (dispensation), internet, course registration schedule of study program/department. IS ought to provide information related to grade of subject for each student and each semester which will be presented either in a study result card or an academic transcript. As for graduation activities, IS should provide information includes registration, graduation ceremony, and diploma delivery.
system performance measurement based on IT BSC Perspective are presented in Table 4.
Table 4. The Methods of Academic Information System Performance measurement based on IT BSC Perspective.
IT-BSC Perspective Academic Information System Measurement
Business Contribution
1. Performing control towards IS, for example by comparing with the actual budget, analysing the use of budgets, calculating the cost of IS per number of staff.
2. Calculating the financial benefits derived from selling products and services.
3. Doing business assessment of the projected new IS when the university will create and develop IS, for example by evaluating business based on economical information and performing financial evaluation based on Return on Investment (ROI), Net Present Value (NPV), Internal Rate of Return (IRR), Payback Time (PB).
4. Doing business assessment of IS functions such as calculating the percentage of capacity related to IT strategy projects, analyzing the relationship between development/new infrastructure and the investment/investment displacement.
User Orientation
1. Assessment towards BTSI associated with applications that have been addressed, the percentage of applications that have been completed, etc..
2. Cooperation with the users of information systems when it will carry out the functions of IT organization, for example by calculating the number of users involved in the manufacturing process and the development of IS application.
3. Analyzing the IS’s user satisfaction by measuring the level of user friendliness on the application, calculating the index of user satisfaction, counting the number of applications and system availability.
Operational Excellence
1. Conducting an analysis towards the efficiency of software development, for example in terms of the average increase of unexpected budget, maintenance activities, the average number of delayed response of the application.
2. Conducting an analysis towards the efficiency of such operations by computing network availability, response time per category per person, the percentage of work which is done ontime, the ratio of operating costs of the system used.
3. Conducting an analysis towards the acquisition and application of personal computers if there is any upgrade.
4. Conducting an analysis towards problem sloving if the system is on trouble, for example by calculating the average charge time, troubleshooting time, the percentage of problems answered in a timely manner.
5. Conducting an analysis towards the training of the IT users.
Future Orientation
1. Conducting an analysis towards the training and expertise of IT staff both in terms of the budget which is owned by institutions and trained individuals based on the age, expertise, etc.
2. Conducting an analysis towards the age of the applications and opportunities for investment in new technologies.
SWCU Academic Information System Measurement Model by using the IT-BSC is illustrated in Figure 2:
Universities’ Business Goals by
Conclusion:
Based on those IT organization functions in SWCU, it is necessary to establish IT Balanced Scorecard for each of those functions which is essentially a derivative of the IT Balanced Scorecard for universities. The process of identifying Critical Success Factor (CSF) is needed to determine the limits of performance measurement criteria in each IT process. After determination of the CSF and know the needs of IS for the academic activities, it should begin mapping the steps what needs to be done in order to measure the performance of academic information system.
References:
[1] Dorian Pyle. (2003). Business Modeling and Data Mining. Morgan Kaufmann Publishers, ISBN:155860653X. [2] Gold, C. (1994). US measures — a balancing act. Boston: Ernst & Young Center for Business Innovation.
[3] Henderi. (2010). Good IT Governance: Framework and Prototype for Higher Eduation. Creative Communication and Inovative Technology Journal vol 3, no.2 ISSN: 1978-8282.
[4] Indrajit, Eko. (2006). Mengukur Tingkat Kematangan Pemanfaatan Teknologi Informasi untuk Institusi Pendidikan (Suatu Pendekatan Kesiapan Pemegang Kepentingan/Stakeholder). Conference proceeding of ICT for Indonesia, Bandung: 3-4 May 2006, 116-120.
[5] ISACA. (2004). COBIT Student Book. IT Governance Institute.
[6] Maniah and Surendro. (2005). Usulan Model Sistem Informasi (Studi Kasus: Sistem Informasi Perawatan Pesawat Terbang). National Seminar of Information Technology Application, Yogyakarta: 18th June 2005.
[7] Maria. (2011). Perbandingan Sistem Informasi Akademik Universitas Kristen Satya Wacana Menggunakan COBIT Framework. Journal of Eonomic Foccus, Vol X, Issue-2, 140-149.
[8] Maria, dan Haryani. (2011). Audit Model Development of Academic Information System: Case Studi on Academic Information System of Satya Wacana. Journal of Art, Science & Commerce, Researchers World, Vol II, Issue-2, 12-24. [9] Maria, et al. (2012). The Measurement of Information Technology Performance In Indonesian Higher Education Institutions in
The Context of Achieving Institutional Business Goals Using COBIT Framework Version 4.1: Case Studi Satya Wacana Christian University Salatiga. Journal of Art, Science&Commerce, Researchers World, Vol III, Issue-3(3), 9-19.
[10] Mulyadi. (2001). Balanced Scorecard. Jakarta: Salemba Press: p 416-420.
[11] Mutyarini and Sembiring. (2006). Arsitektur Sistem Informasi Untuk Institusi Perguruan Tinggi Di Indonesia. Conference proceeding of ICT for Indonesia, Bandung: 3-4 May 2006, 102-107.
[12] O’donnell, E. (2004). Discussion of Director Responsibility for IT Governance: A Perspective on Strategy. International Journal of Accounting Information Systems 5: p 101-04.
[13] Prabowo, Harjanto. (2007). Implementasi IT Balance Scorecard di Perguruan Tinggi. National Seminar of Information Technology Application, Yogyakarta: 16st June 2007.
[14] Sa’adi and Suhardi. (2006). Pengukuran Kinerja Penerapan Sistem Enterprise Resource Planning (ERP) di Universitas dengan Metode IT-Balaced Scorecard (IT-BSC). Conference proceeding of ICT for Indonesia, Bandung: 3-4 May 2006. [15] Van Grembergen, W. and Van Bruggen, R. (1997). Measuring and improving corporate information technology through
the balanced scorecard technique. Proceedings of the Fourth European Conference on the Evaluation of Information Technology, Delft: October 1997, 163-171.
[16] Van Grembergen, W. and Timmerman, D. (1998). Monitoring the IT process through the balanced scorecard. Proceedings of the 9th Information Resources Management (IRMA) International Conference, Boston: May 1998, 105-116.
[17] Van Grembergen, W. (2000). The Balanced Scorecard Technique and IT Governance. Accessed in http://www.isaca.org/Certification/CGEIT-Certified-in-the-Governance-of-Enterprise-IT/Prepare-for-the-Exam/Study-Materials/Documents/The-Balanced-Scorecard-and-IT-Governance.pdf
[18] Willcocks, L. (1995). Information Management. The Evaluation of Information Systems Investments. London: Chapman & Hall.
