Creative, Collaborative, and Cost-effective: The New Era of Widening Participation Evaluation

 Evaluation is a critical tool needed for improvement.

As educators, we encourage pupils to evaluate their learning and reflect on what they learned, what went well and how they could improve. Evaluation of the impact of our Widening Participation (WP) programmes are just as important. We need to be able to evaluate and understand the impact of these programmes, in terms of how valuable and beneficial they are to pupils.

 

But why is this to vital in work for WP?

On a basic level it is for annual tracking purposes. WP programmes need to showcase to the Scottish Funding Council (SFC) that they have value and produce beneficial outcomes for WP pupils, therefore, must produce data to evidence this. Each year the SFC grant funding to each of the programmes to carry out the interventions they have, so each element must highlight why it is value for money and use evaluation to prove these figures.

But rather than focusing on a policy and funding level, the most important impact educators are concerned about is the impact on the pupils. Of course, those that work with pupils on a daily basis and are there to encourage them to pursue their ambition and reach Higher Education (HE), want effective interventions that produce positive outcomes. This would arguably be the most important reason for evaluation of WP programmes, because it allows for the reflection on the pupil experience. We can understand what the pupil enjoyed most, what elements of the intervention they learned best from, what they perhaps did not enjoy, and highlight any gaps in the system that can be amended.

However, the importance of WP and the need for evaluation was particularly elevated by the Commission on Widening Access (CoWA), which put in place recommendations to improve the achievement of equal access at HE. Recommendation 2 was to work with experts to publish a Scottish Framework for Fair Access by 2018 (Scottish Government, 2016). This framework was published and as a one of the two main pillars it created a fair access toolkit to evaluate and assess the effectiveness of existing interventions (Commissioner for Fair Access, 2022a). This aim of this toolkit is to evaluate and identify the already successful interventions and promote their effectiveness for future interventions to shadow.

 

So, why has it become increasingly important and more prevalent now, in 2022?

John Blake, director for Fair Access and Participation, stated that evaluation is his highest priority in his new role (Blake, 2022) because he believes evidence-based practice is essential for all public services, so that they can make sense to other practitioners and survive the contact with the tough reality of frontline provision. As part of Blakes work, HE institutions will be required to seek a change to the Access and Participation Plans (APPs) of work they cover regarding priority areas of strategic school engagement, and a new cycle of these APPs will begin next year.

With these new practices in place from stakeholders, it is becoming increasing important as practitioners to produce robust evidence of the effectiveness and value of our interventions to promote WP.

 

As we mentioned earlier, the original toolkit was introduced to evaluate the existing interventions and to assess the value in cost and outcomes (Commissioner for Fair Access, 2022a). But education is constantly evolving, so current interventions are adapting, and new interventions are emerging, so these also need evaluating.

Education has shifted majorly over the last two years, mainly due to COVID-19 and the impact of lockdown. When schools closed, online learning was at the forefront of educations priority, as it was the only way for pupils to continue learning from home. As such, all teaching methods and interventions had to adapt from in person delivery, to online delivery. This was a major change to how interventions were previously run, and particularly for the work in WP, it needed to continue to be just as effective, if not more effective in preparing those WP pupils for HE, and successfully completing their school education.

Online learning had its pros and cons, for both educators and the pupils.

Pros: flexible learning style, less travel commitments for commuting, access to the internet and wider resources online and less costly in the delivery format.

Cons: lack of interaction between pupils and teachers, less encouragement and enthusiasm through an online platform, pupils could become more easily disengaged.

With this balance of pros and cons, it is vital for WP interventions to evaluate the value in continuing with the online format of delivery moving forward. Some practitioners have suggested blended learning is the most effective because it allows for both the pros and cons (Pearson, 2021), but we need to do robust evaluation to understand the statistics and impact it has for the pupils and their access into HE.

Not only has online delivery changed the course of action for existing programmes, but completely fresh interventions have emerged over the last few years to tackle the increased level of challenge and new barriers that have surfaced for WP pupils. Lockdown meant the closure of schools, and for a lot of pupils online learning from home was not effective, particularly for the younger pupils who need more encouragement to engage with the school day. Therefore, a lot of these pupils missed out on a structured education in their key stages of schooling, which is why once schools opened up, we as educators, noticed a gap in knowledge and a need to build up confidence of those cohort year groups. New interventions focused on lower secondary workshops that could be interactive and engaging to plant the seed of HE and get them thinking about their upcoming school choices and future careers. These too, need to be evaluated, we need to understand what parts of the workshops are beneficial for the pupils and perhaps what other gaps need to be filled in order to achieve equal access in upper-secondary education.

 

Why hasn’t this already been done? What is holding back WP practitioners in producing this rigorous, robust evidence of evaluation?

Two main challenges for us as WP practitioners have been cost, in terms of finance and time, and lack of expertise.

Firstly, evaluating projects is costly and can require money being spent on resources, data analytics, monitoring and assessment, and for programmes funded by the SFC, this might not be covered by the money given by the Government, so it is a cost the programmes themselves have to account for.

But it isn’t just costly in terms of financial support, evaluation of WP is costly in time commitments being spent on conducting the whole evaluation cycle. For WP practitioners the main purpose of our work is to engage and support pupils in their transition and access into HE, so taking time away from this core aim can inhibit the overall positive impact we have for our pupil’s success.

Secondly, leading on from the time commitment needing spent on evaluation, in order to produce robust evidence, as we mentioned, WP practitioner’s role is to support and encourage pupils to access levels of HE. Therefore, there is a potential lack of expertise in creating and conducting a cohesive and rigorous evaluation plan. If WP programmes aren’t able to produce credible evaluation evidence there is the risk that they aren’t able to demonstrate the true impact they are positively having on the pupils involved and could lose out on future funding.

Not only is the lack of expertise in research a factor in which makes evaluation a challenging process, but the multiple elements within one overarching programme can make it difficult for singling out the impact of an individual project within that. Typically, pupils take part in multiple elements, and of course there are external factors that can contribute to the holistic outcome of a pupil’s access into HE. Therefore, with the lack of experience developing robust, specific evaluation, it can make it challenging for WP practitioners themselves to conduct the evaluation.

 

So, what is the way forward? How can we overcome these challenges, and at the same time produce robust evidence?

 

In line with the recommendations for evaluation proposed by CoWA (Scottish Government, 2016), SCAPP have created an innovative and accessible WP evaluation guide to support practitioners in developing and conducting their evaluation (Commissioner for Fair Access, 2022b). As someone who works within this field, this is a very useful resource that will be extremely beneficial in future development plans of evaluation. There is a clear evaluation cycle, involving 6 key steps, which highlights the way in which credible evidence can be produced.

The WP evaluation guide acknowledges the challenges that are currently in place for evaluation and targets specific support to address these. The first way in which the guide supports WP educators is by providing a clear theory-based approach to creating the initial basis of evaluation. Specifically, SCAPP have identified how significant change has occurred over the last few years to education and WP interventions. Therefore, the guide showcases the success of implementing Theory of Change (Harrison & Waller, 2017) for identifying and evaluating the effectiveness of specific changes to the intervention.

However, arguably the most innovative part of the guide is the use of a collaborative approach to evaluation. By understanding the potential lack of expertise in the research and evaluation domain, WP practitioners can collaborate with research experts to develop and deliver an effective form evaluation which will produce robust and credible results. Moving forward, this collaborative approach will be holistically beneficial because it allows for the sharing of knowledge and information, and the communication between key stakeholders. Therefore, the ultimate aim of increasing progression into HE will have the most potential because the pupils will be getting the most effective form of intervention (developed through rigorous evaluation), and have it delivered by WP educators (with the expertise on WP pupils and the specific barriers they face).

 

Lorna Lamont, FOCUS West Development Worker

 

References:

Blake, J. (2022). Schools and university partnership working: Why evaluation and collaboration is the cornerstone of widening participation. Available at: https://www.officeforstudents.org.uk/news-blog-and-events/blog/schools-and-university-partnership-working-why-evaluation-and-collaboration-is-the-cornerstone-of-widening-participation/. Accessed 25^th May 2022.

 

Commissioner for Fair Access. (2022a). The Framework for Fair Access- an introduction by the Commissioner. Available at: https://www.fairaccess.scot/. Accessed 25^th May 2022.

 

Commissioner for Fair Access (2022b). Widening Participation Evaluation Guide. Available at: https://www.fairaccess.scot/widening-participation-evaluation-guide/. Accessed 26^th May 2022.

 

Harrison, N. & Waller, R. (2017). Evaluating outreach activities: overcoming challenges through a realist ‘small steps’ approach. Perspectives: Policy and Practice in Higher Education, 21(2-3).

 

Pearson. (2021). Thoughts on blended learning after lockdown. Available at: https://www.pearson.com/uk/educators/fe-college-educators/btec-blog/2021/06/thoughts-on-blended-learning-after-lockdown.html. Accessed 25^th May 2022.

 

Scottish Government. (2016). A Blueprint for Fairness: Final Report of the Commission on Widening Access. Available at: https://www.gov.scot/publications/blueprint-fairness-final-report-commission-widening-access/pages/3/. Accessed 25^th May 2022.