Quality Factor Analysis and Technical Model for the OUC Test Bank
OUC
Liu Yuxia, Hu Xing
Abstract: The OUC test bank is the foundation for appointment-based examinations. Its quality is vital to the overall quality of appointment-based examinations, as well as to the success or failure of teaching reform, and the quality of learner formation. Based on an analysis of the quality factors of the Open University of China’s (OUC) test bank, this paper suggests that the factors include clearly aligned goals, a suitable structure, the efficacy of the strategy, and the timeliness of the content. Furthermore, the technical model should be developed based on practical requirements. Finally, this paper focuses on four key aspects of technical design including model proposition, equivalent storage, dynamic links and exposure warnings.
Keywords: The Open University of China, test bank quality, technical model
Test bank construction reflects the integration of modern information technology and effective testing. With the rapid development of testing across various industries, the technology and theory of test bank construction is becoming more and more mature. “The OUC was established on July 31, 2012 on the basis of the China Central Radio and TV University (CCRTVU) supported by modern information technology, it attaches a great deal of importance to degree education as well as non-degree education, and provides open and distance education to all members of society. It is a new type of university under the direct administration of the Ministry of Education (MOE).” The construction of the OUC requires the integration of information technology and distance education. As part of its construction, the OUC must learn from the achievements of previous research on test banks, in order to build a test bank that is suitable for distance learning, can be booked at any time, and is distributed nationally or internationally. Ensuring the quality of test bank construction is an inevitable problem.
I. Current development
1. Literature review
The construction of test banks both in China and abroad can be divided into two types according to the nature of the exams. The first is test banks built by social agencies, such as Cambridge Assessments in the UK or the American College Test (ACT). The second type is test banks built by education institutions (including ministries or universities), such as the MOE National Education Examination Authority and some test centres of universities in China. The test banks can also be divided into two types according to function. The first is the single function test bank, which is only used for a single purpose, such as the driving license test, the computer professional capacity level test, the CET band 4 and band 6 tests, or tests for a specific course or major. The other is the multifunctional test bank, which are comprehensive test banks and can be used for two or more purposes, such as the OUC test bank. In order to understand the current state of research in China and abroad, we searched for the term “test bank” on CNKI for the period 2000 to 2012. The search returned 849 articles. When we searched for “test bank construction”, we obtained 99 articles, of which 95 were effective results. 50 articles were about the meaning and functions of test bank construction, 18 articles were about test bank design, 16 articles were about test bank technologies and six articles provided an empirical analysis of test bank construction. Therefore, we can see that test bank construction has entered a period of rapid development.
There are two major theories of test bank construction: classic measurement theory and project reaction theory. Most test banks rely on classic measurement theory. Due to the complicated nature of its mathematics model, project reaction theory creates numerous difficulties during test bank construction. In terms of test bank design, a consensus has been reached on basic construction procedures including the two-way design breakdown, proposition scheme and proposition table. The only difference lies in design standards. Factors worthy of comprehensive consideration include assessment knowledge points, teaching requirements, assessment targets, question types, quantity, scores, time taken to answer question, difficulty, and differentiation. Strategies for composing test papers include random composition, genetic method composition, as well as composition based on the Moodle platform, the IRT model, OLE technology or NET technology. Test banks can be divided into question banks and paper banks. Most test banks are paper banks, which are not very intelligent.
An analysis of the existing research on test banks has identified the following problems in the construction of a test bank. Firstly, test bank design standards are not uniform. There are no uniform references and standards for test type, quantity, design factors, question properties, and systemic indicators. Secondly, test composition strategies have yet to achieve intelligent composition in the true sense. Limited by the constraints of test bank design and test bank technologies, the strategy is often to compose examination papers by computer in advance, making it difficult to achieve real-time intelligent test composition. Thirdly, it is difficult for composed exam papers to achieve predetermined design objectives. Limited by imperfect composition, the exam papers often have problems such as low coverage of prescribed content, repetitive content, and overall arrangement that is not in line with the law of thought. Many questions within the test bank are not used for long periods of time. Lastly, the exposure rate of test items is relatively high during use. The construction of a test bank requires a great deal of time, effort and financial resources. Test banks are general used for long periods of time, so if there are not enough stored test questions, frequency of use and exposure will be high.
2. Test bank construction at the OUC
The OUC is “constructed on the basis of the CCRTVU. It emphasizes the gathering, integration, and sharing of quality educational resources, draws support from modern information technology, and explores the integration of modern information technology and education”. The construction of test banks at the OUC has gone through three phases: First was a period of exploration. From 1999 to 2003, research on computer-based summative testing was conducted, and a test bank for summative testing was established. Secondly came the pilot project phase. From 2004 to 2006, the “Web-based course reform pilot scheme” was initiated. Research on the formative assessment system was completed and a pilot scheme was carried out in 2005 and 2006. On March 10, 2007, the OUC formative assessment system passed scientific technology achievement appraisal (Appraisal (education SH2007) No.005). The third period centred on promotion. From 2007 to the present, the number of web-based assessment reform pilot courses increased from six in 2005 to 161 in autumn 2012, among which 93 courses have established a test bank for summative testing.