Publication
A Guide to the Calculated Grades Data Collection, National Standardisation and Quality Assurance Processes
Published on
Last updated on
This page provides a brief guide to the data collection, national standardisation, and quality assurance processes in the Calculated Grades system.
In making the difficult decision to postpone this year’s Leaving Certificate Examination and replace it with the new system of Calculated Grades, the Department of Education and Skills has set out to develop the best possible system to provide students with fair and accurate results in the absence of them being able to sit the Leaving Certificate this summer.
This was important so that students would have the opportunity to move on with their lives and would be able to use these results to progress to employment, further education, training and apprenticeships and to higher education. It must be acknowledged that the system is not the same as a Leaving Certificate and is not a perfect solution. However, it is the best alternative that could be provided. It is the fairest way possible to tackle the effects that lack of schooling and other problems caused by COVID-19 have had on the Leaving Certificate.
The first stage in this system was to be able to base the Calculated Grades on really good quality information from schools about students’ expected performance in the Leaving Certificate examinations, had their educational experience not been interrupted by COVID-19. While the school data will be subject to adjustment in the national standardisation process, it forms the basis for the Calculated Grades, so a lot of time and effort was spent in seeking to get this part right.
The approach taken centres on international, educational research which tells us that teachers are more likely to make accurate judgments of student performance in a high stakes context and that teachers are good at ranking their students relative to each other. Therefore, schools were asked to provide an estimated mark and a rank order for each student in a class group.
Very clear and comprehensive guidance and training was provided in order that schools could provide their students with the fairest possible estimate of their performance in the Leaving Certificate.
Schools were asked to engage in a robust and rigorous three-stage process to arrive at their best estimates of student performance, which involved:
- teachers applying their professional judgement to various sources of evidence
- a process to align standards among different teachers of the same subject
- oversight of the process by the school principal
The only sound means to achieve fairness across different class groups taking the same subject within a school was through the in-school alignment procedure, so this was a very important part of the process. Specific guidance was also provided on recognising and avoiding unconscious bias that might affect the estimates.
Following the completion of the estimating process, schools had to return the data to the Calculated Grades Executive Office (CGEO) in the Department of Education and Skills, using an online data collection system. Checks and balances in the process ensured that the data was entered correctly and accurately. A similar robust and rigorous quality assurance process has been carried out by the CGEO on receipt of the school data and in subsequent processing.
Further checking is taking place, which will include sampling some of the completed paperwork from randomly selected schools to ensure accuracy.
At this point in time, all of the available school data is being processed by the CGEO in the statistical system that has been developed for us by an expert in this area. We do not have all of the data yet as we are still undertaking reviews with schools in respect of students taking subjects outside of school where the school could not provide an estimated mark. We are also processing applications from students who were studying for their Leaving Certificate but who were not attending a school or other centre of education authorised to hold the Leaving Certificate examinations.
Every effort is being made to provide a Calculated Grade to as many students as possible, whether they are out-of-school learners or taking subjects outside of school.
For the Calculated Grades system to be operated with integrity, and to be fair to all students, an estimated percentage mark, based on credible, satisfactory evidence, can only be accepted from an appropriate source. This work will continue, but in the meantime we can proceed with the statistical standardisation process using the available data and regular data updates will be fed into the standardisation process as they become available.
In the Calculated Grades system, students’ expected performance in a subject and level will be combined with information about how students in the school have fared in this subject or level in recent years in line with national performance standards over time. The performance of this year’s group of students against their overall performance at Junior Cycle will also be reviewed.
The relevant information which will be used to support this process includes:
- national level data for the Leaving Certificate for 2019, 2018 and 2017 and related Junior Cycle data
- school level data for the Leaving Certificate 2019, 2018 and 2017 and related Junior Cycle data
- student level data for the Leaving Certificate 2019, 2018 and 2017 and related Junior Cycle data
- student level data for the Junior Cycle results of the 2020 Leaving Certificate cohort of students
The Leaving Certificate results from any one year are considered equivalent to the results from any other year. This is because the results each year are subject to a process of national standardisation through the marking processes undertaken by the State Examinations Commission.
This year, the national standardisation process will combine the school-sourced data and the historical data to ensure the Calculated Grades reflect standards that are properly aligned across schools and with a national standard.
By collecting and using a range of different types of information, the different sources of data will complement each other, to provide the most accurate and fair set of results within the limitations of the available data. As the school data is only accurate at school level, the final calculated marks, and so Calculated Grades, provided to students, for any subject and level, may be higher or lower than the estimates provided by their school.
It is as a result of this standardisation process that the Calculated Grades will have an equal standing and status with previous and future Leaving Certificate grades. If this is not done, it would undermine the currency and value of Calculated Grades.
The national standardisation process will use the range of Junior Cycle and Leaving Certificate historical data to determine the particular distributions of results that are most likely for each subject and level. The use of such distributions as part of the estimation process is referred to as conditioning.
At school and national level, Junior Cycle examination results are strong predictors of Leaving Certificate performance but are not good at estimating individual student performance. The Junior Cycle data will be used in the model to produce a distribution of the likely performance of the particular group of students taking each subject in each school.
In addition, the historical school data for the last three years of Leaving Certificate performance at each subject and level will be blended with the historical national distributions for each subject at each level, in order to establish further the most likely subject or level patterns of results for the 2020 cohort.
These conditioning distributions will provide a pattern of expected results for the school group at each subject and level. The school-based data will also provide a pattern of expected results for the school group at each subject and level. The school-based data provides valuable student-specific information at a local level and the historical data provides valuable group-level information at a national and school level.
No one pattern of expected results on its own will determine the Calculated Grades, as the model will combine the range of patterns to generate the grades in a way which is as reasonable, fair and accurate to students as possible. While the school-based estimates may move upwards or downwards as a result of the standardisation process, the class rank order, provided by the school, will not be changed. The fact that the distribution of results is not based entirely on any one data set, but capitalises on the distinct strengths of each source of information, contributes to accuracy and fairness in the model.
The statistical process takes account of whether the group of students taking a subject in the school this year is academically stronger (or weaker) than in previous years. This means that the distribution of grades emerging from the school is not pre-ordained to be the same as it was before.
The model does not impose any predetermined score on any individual in a class or in a school. Provided the school has accurately reflected the relative differences between students within the class, there will be no barrier to any excellent student achieving high grades in any school. For example, if there is a high-performing student within a class of low-performing students they will not be unfairly disadvantaged by using the distributional information as described earlier. Provided the school has recognised the atypically high performing student, they will be correctly identified as an outlier student.
Even if the standardisation process shifts the marks upwards (or downwards) for the entire class, the calculation process within the model will take account of the relative standing of the student within the class, as set out in the school-based data, and will allow the estimates for the group as a whole to be properly realigned, if necessary, without losing this information about the degree to which this high-performing outlier student exceeds the norm.
The same will apply if there are outlier students at the lower end of performance. The model is designed to accommodate such outliers.
To ensure that the statistical model is behaving as expected, and is achieving its objectives, there needs to be a related process of model validation. The workings of the statistical model will be reviewed and validated in a number of ways.
There will be a review of the distributions of results for each subject and level.
There will also be a review of the outcomes using a number of different demographic characteristics which will include gender and socio-economic status. This is to ensure that the model is presenting outcomes that are as fair and equitable as possible given its constraints, and in line with previous outcomes as much as is possible.
The purpose of the Calculated Grades system is to arrive at the grade that each student would have achieved if the examinations had taken place as normal. The validation process will check to see if the interactions between these characteristics and the calculated results are similar to the interactions in the historical data between these characteristics and examination results.
For example, in the case of gender, if the performance of female students relative to male students in various subjects turns out to be similar under the Calculated Grades model as was normally the case in previous examination years, then this can be taken as an indicator that the Calculated Grades model is in line with previous years in relation to this.
The purpose of this review of the outcomes of the statistical model is to check whether the Calculated Grades model is resulting in any particular group being advantaged or disadvantaged relative to previous years’ outcomes. It is important to note that this assessment is being made relative to previous years; the validation will check, for example, that disadvantage effects, or gender effects, are not exacerbated within the model. It is also important to note that the system will not correct for relative biases that are internal to the sets of estimates provided by individual schools. Schools should have resolved the issue of any unconscious bias within their own estimating process. This was advised in the Guidance to Schools.
The combination of historical data sets and the data collected from schools will be processed through the statistical model. However, there is a significant level of human intervention required to be able to stand over the statistical model and to approve the issue of the Calculated Grades to students.
The national standardisation process will be overseen by a National Standardisation Group responsible for the implementation of the process and the application, review, and adjustment of the data to arrive at fair and just representations of student performance.
The expertise of the National Standardisation Group is in the field of high stakes examinations and assessment, educational evaluation, and second level education and the organisations and individuals represented on the Group were central to the development of the system of Calculated Grades. The Group comprises representatives from the Calculated Grades Executive Office and the Inspectorate, both from the Department of Education and Skills; Educational Research Centre and the State Examinations Commission. The group will meet regularly to analyse and review the most recent iteration of the model and to discuss the outcomes and review how best to proceed in relation to adjusting the model and also to work through the validation process.
The maintenance of a national standard during the Calculated Grades process is as important as in previous years in order to ensure that the Leaving Certificate 2020 Calculated Grades are of equal standing to the outcomes from previous years. This is in order to ensure equity and fairness for the 2020 cohort but also for previous and future students who may be competing for college places or in the world of work.
National distributions, which range across 79 Leaving Certificate curricular subjects and levels, 18 non-curricular languages and approximately 20 Leaving Certificate Applied subjects and tasks will be reviewed for 2020, as well as against the historical data.
Where distributions are not aligned satisfactorily, the model will be adjusted in order to bring the distributions into the range of previous outcomes so as to ensure comparability and fairness across years. With each adjustment to the model, there will be further review, to check for negative impacts elsewhere in the model, and further adjustment as required until the group is satisfied that a safe, satisfactory and defensible set of result outcomes has been achieved.
In the Calculated Grades system, the Educational Research Centre will provide a data quality assurance and verification service on the data processing and standardisation processes associated with the 2020 Leaving Certificate and Leaving Certificate Applied Calculated Grades. The Centre has expertise in the conduct of independent research, assessment and evaluation studies; and in the development and provision of tests to schools.
Further quality assurance of the Calculated Grades system is provided through an Independent Steering Committee appointed by the Minister to provide assurance of the quality and integrity of the outcomes of the Calculated Grades system including by satisfying itself as to the fairness and accuracy of the outcomes following the national standardisation process. In addition an independent expert unconnected with the design of the Calculated Grades model will provide an external review and validation of the system, including the operation of the appeals system, to check that the process has executed as intended.
