Implementation of the Research
To ensure the successful implementation of the follow-up survey, more than 1,000 faculty members from 44 RTVUs at the provincial level participated in the administrative process. In addition, more than one million RMB was invested in the research programme to guarantee the smooth operation of the survey. The implementation will be discussed in terms of two major stages, i.e., the design and modification of the measurement instrument for the follow-up survey and the administrative process of the follow-up survey.
The design and modification of the measurement instrument for the follow-up survey
The measurement instrument was designed to cover four major aspects related to CCRTVU distance education programmes, i.e., curriculum design, learning resources, course delivery and learner support. Specifically, it was used to gain quantitative information on whether the course structure was rational and suitable, whether the course content was scientific and catering to the needs of the society, whether the learning resources were effectively used by distance learners, whether the means of course delivery was helpful for the learners and whether the learner support provided by RTVU satisfied the needs of the learners. Feedback on these four aspects would give those concerned some insight into the quality of CCRTVU open learning programmes. In addition, the measurement instrument was also designed to elicit information from the graduates’ employers on four aspects concerning the overall quality of RTVU graduates, i.e., morality and professional ethics, knowledge, ability and work performance.
The measurement instrument for the follow-up survey was designed in accordance with the theory of educational assessment and the theory of psychological measurement. It includes indicators for measurement, assessment model, rating scale and questionnaire, etc. Before it was put into use, a pilot survey was conducted between December 2003 and March 2004 at six PRTVUs in Beijing, Tianjin, Shaanxi, Yunnan, Fujian and Hebei. Based on the data collected from 370 questionnaires from the graduates and 246 questionnaires from their employers, modification was made to each component of the measurement instrument. For example, similar items were combined. "Courseware" in "learning resources" was combined with "learning resources on the Internet", resulting in "learning resources on the Internet (courseware included)"; "live classroom" in "course delivery" was combined with "teaching on the Internet", resulting in "Teaching on the Internet (live classroom included)", etc. In addition, some of the indicators were redefined for clarity. For example, the term "project or thesis on graduation" in fieldwork was rewritten to be clearer. Moreover, some unsuitable items were deleted, such as "computer skills" in "learning competence", as it was unsuitable for graduates in remote and less developed regions.
As a result, the data collected using the modified measurement instrument showed that the internal consistence reliability of the questionnaire was above 0.9, and the validity analysis showed that the loading of 63 indicators ranged from 0.411 to 0.821. The CFI (Comparative Fit Index) and TLI (Tucker-Lewis Index) of the Default model and Independence model were above 0.9, and the RMSEA (Root Mean Square Error of Approximation) was under 0.08. The parameters above indicated that the questionnaire was appropriate from a psychometric standpoint.
The administrative process of the follow-up survey
- Forming research teams responsible for conducting the follow-up survey
CCRTVU and 44 PRTVUs attached great importance to the administration of the follow-up survey. To ensure the successful operation of the follow-up survey, research teams led by presidents and composed of staff members from relevant departments were formed. They were in charge of designing the plan for implementation, training the participants involved in conducting the follow-up survey, administering the survey, data processing, and analysis and writing of the report. They were also responsible for budgeting for the research to guarantee the smooth operation of the follow-up survey.
- Training the participants involved in conducting the follow-up survey
Various kinds of training sessions were held via different media, including online meeting and video conferencing to guarantee that all the participants understood the whole process well and that the results obtained in this way would be scientific and valid.
- Sampling
The overall population for this study was graduates from 2002 to 2004, which totaled 280,000. While the main body of subjects was chosen from course programmes with a large number of graduates, including Law, Computer Science and Technology, English, Economics and Finance, Accounting and Business Management, all of which were degree courses, some were graduates of Primary Education, Finance, and Hydro Electronics and Water Conservancy, which were diploma courses. To ensure that the subjects with valid questionnaires could fall between 3% and 5% of the overall population, 17,520 graduates, or 6.3%, were selected through stratified sampling and systematic sampling.
- Administration of the questionnaire
1. Delivery of questionnaires to the graduates
All RTVUs selected graduates as subjects according to their registration numbers, by selecting every nth person in the numerically ordered list as a subject. Subjects were then contacted through the address they registered with the study centre, and if anyone was unreachable by this means, then s/he would be replaced by the one with the registration number one before or after hers/his. On contacting the graduates, the researcher would explain to them the purpose and the significance of conducting the follow-up survey. Then the RTVU would assemble the graduates close to the study centre on the weekend for the completion of the questionnaires. The researcher would explain to the graduates how to complete the questionnaires and the questionnaires were handed in immediately after completion. As calculated, an average of 25 minutes was needed for the completion of each questionnaire. For those who were far away from the study centres, questionnaires would be posted to them after a confirmation by telephone or email of their willingness to complete the questionnaire. Together with the questionnaire and guideline for completing the questionnaire, a return envelope and postage were also enclosed to make it more convenient for the graduates.
2. Delivery of questionnaires to their employers
After the graduates were selected and contacted, they were asked to provide the contact details for the personnel department of their employers. Then the researcher would contact the personnel department, who would further ask the manager of the graduate to complete the questionnaire. To ensure the truthfulness and objectivity of the appraisal of the graduates, the questionnaires were delivered in the following three ways:
1) The survey researcher directly took the questionnaire to the appointed person, who would complete the questionnaire and return it immediately after completion.
2) The RTVU assembled some of the subjects for interviews to learn about their comments and recommendations on RTVU graduates, followed by the completion of the questionnaires.
3) For those employers who were far away, questionnaires would be posted to them after a confirmation by telephone or email of their willingness to complete the questionnaire. Together with the questionnaire and guideline for completing the questionnaire, a return envelope and postage were also enclosed to make it more convenient for the employers.
3. Response rate
Thanks to the various measures mentioned above, and especially the joint efforts made by the RTVU system, a relatively high response rate was achieved. Furthermore, to ensure that enough valid data would be collected, more subjects were selected than necessary. Altogether 17,520 questionnaires were delivered to graduates and 17,386 were returned, for a response rate of 99.24%. Among the questionnaires returned, 15,776 were valid, or 90.74%, amounting to 5.63% of the overall population for the research. As for the employer surveys, 9,926 valid questionnaires were returned, or 57.09%, amounting to 3.55% of the overall population.
4. Identification of invalid questionnaires
To guarantee the validity of the questionnaires, guidelines were clearly explained to the subjects. However, it was unavoidable that invalid questionnaires would be collected due to various reasons, including questionnaire fatigue and the subjects' inability to see its usefulness, etc. To make data processing easier, a model for identifying invalid questionnaires was designed.
- Administration of interviews
After the completion of the questionnaires, some of the graduates were selected for interviews to obtain further information or comments on CCRTVU open learning programmes. Also, some of the employers were assembled for comments and recommendations on the overall quality of the graduates.
- Data processing and analysis
The data collected through the follow-up survey was then analysed using SPSS 11.0, and the reliability and validity were also analysed using SPSS 11.0 and Amos 4.0. As mentioned previously, the survey proved to be reliable and valid in terms of the relevant parameters.