Tools: Evaluation of quality apprenticeship programmes
A number of tools for carrying out tracer studies, cost–benefit analysis, assessment of the apprenticeship policy environment and self-assessment of apprenticeships, as well as performance indicators, are given below for both policy-makers and practitioners who are engaged in the evaluation of the apprenticeship system and programmes.
Tools for conducting tracer studies
Online survey for graduate apprentices, EAN
This tool is an example of an online survey for graduate apprentices, to assess their opinions about the quality of apprenticeship programmes. The #AskTheApprentices survey questions cover modules such as: the quality of education, learning materials and teaching methods; apprentices’ rights, responsibilities and protection; apprentices’ contracts; representation in the workplace; promotion of apprenticeships; anti-discrimination measures; and access to information on apprenticeships.
Guide to tracer studies, ETF, ILO and Cedefop
This guide offers a step-by-step introduction to the wide range of different tracer study approaches available and provides detailed instructions on how to design a graduate tracer study, develop the questionnaire and carry out the data analysis. Furthermore, it provides two versions of the questionnaire: a minimal version, which was designed to be used with very few adaptations/changes; and a module version, which is a toolbox of different topics and questions from which elements can be selected as appropriate.
Although this guide primarily targets TVET and higher education, it can easily be adapted to apprenticeship programmes.
A survey of apprentices to assess the system, United Kingdom
This report presents the findings of the apprenticeships evaluation learner survey 2017 in the United Kingdom. The survey, which is carried out on a regular basis, is intended to monitor key progress indicators and assess the impact of recent policy reforms in order to help shape future development of the apprenticeship programme. Specifically, it covers individuals’ motivations for undertaking their apprenticeship, their experience of the training they received, their satisfaction with the apprenticeship and the impact it has had on their careers.
|Source: https://assets.publishing.service.gov.uk/government/uploads/ system/uploads/attachment_data/file/659709/Apprenticeships_evaluation_2017-learners.pdf.|
Tools for enterprises, including cost–benefit analysis
QualiCarte – a tool that allows employers to self-assess how well they manage apprentices, Switzerland
QualiCarte is an instrument for assessing the quality of in-company vocational training in any occupation. It consists of 28 quality requirements, divided into five chapters, covering: monitoring of quality improvement measures and objectives, commitment, introduction to training, training, responsibility of the host company and end of the agreement.
This tool can be used by employers for self-assessment purposes but also functions as an external assessment tool in cases where representatives of cantons or professional associations want to assess the quality of training provided in a given host company.
A review of methodologies for measuring the costs and benefits of in-company apprenticeship training, ILO
Understanding the costs and benefits of apprenticeship training, as well as its outcomes, is an important factor for enterprises, enabling them to make informed decisions relating to the skills development of current and future employees. Existing studies on this topic paint a mixed picture: some reveal net benefits to employers and others, net costs. In addition to the observed variations in apprenticeship systems and practices between countries, a proper international comparison of cost–benefit analyses is hampered by differences in research methods. This report reviews the research methods and findings of major existing studies with the aim of helping to advance discussions on research methods and highlighting areas where knowledge gaps exist.
Assessment of returns on apprenticeship investment, India
This tool provides a questionnaire for assessing returns on investments in apprenticeships.
The publication discusses five case studies in which senior managers from SMEs assess the costs and benefits of the apprenticeships they provide. The case studies provide a clear insight into how the cost–benefit or returns on investment (ROI) approach can be adapted to the case of SMEs in the Indian context and how evidence gained from the ROI approach can be used as an effective instrument for enterprises to base their training and hiring decisions on.
Analysis of the costs and benefits of apprenticeship to employers, Germany
This tool provides information on how dual vocational education and training is financed in Germany and which cost–benefit factors are involved.
The tool answers the following questions:
Set of indicators for assessing work-based learning, IAG
This paper lists a range of indicators developed by the Inter-Agency Working Group (IAG) on Work-based Learning (WBL) to monitor and evaluate WBL. Although the proposed indicators are related to the performance of WBL, they can easily be adapted for the assessment of quality apprenticeships.
The proposed indicators cover two policy areas:
There are also two additional indicators for quality assessment that can be used during a transition period when WBL programmes are not yet fully developed: share of programmes with learning outcomes/objectives for the WBL component and share of programmes with assessment procedures for the WBL component.
Apprenticeship outcomes performance matrix, United States
|In the United States, registered apprenticeship programmes are evaluated using an apprenticeship outcomes performance matrix, which lists indicators for employers, apprentices or workers, and partner organizations.|
|Number of apprentices retained|
|Decrease in recruitment costs|
|Increase in productivity|
|Increase in workforce diversity|
|Increase in workplace safety|
|Number of credentials earned|
|Number of programmes developed|
|Number of individuals trained|
|Increase in under-represented populations engaged (e.g. women in non-traditional occupations, veterans, individuals with disabilities, etc.)|
|Number of apprentices retained|
|Source: United States Department of Labor, n.d.|
Tools for system-level evaluation
The ILO has developed an analytical framework for evaluating and benchmarking a country’s apprenticeship system and programmes. This framework can assist in carrying out detailed, external evaluation of the policy environment and apprenticeship system, leading to practical policy recommendations to overcome identified challenges. It enables a rapid assessment of an apprenticeship system to be carried out and highlights gaps and challenges that are not always obvious to the practitioners themselves.
Evaluation tool for apprenticeship policy and system, ILO
The tool serves three purposes, as illustrated below:
|Source: ILO Evaluation Tool for the Review of a Country’s Apprenticeship Policy and System|
The following two tools are in the form of reports that provide the results of reviews of a country’s apprenticeship system, carried out by CEDEFOP and OECD. These reports illustrate examples of actual reviews, using methodology and tools developed by the two organizations.
Review of the apprenticeship system in Italy, Cedefop
Cedefop has been carrying out in-depth reviews of apprenticeship systems at national level in order to identify their specific strengths and any challenges that they face in order to propose a set of policy recommendations for ensuring quality apprenticeshpis. Such information can also help other countries to reflect on their own practices and implement reforms for quality apprenticeships.
This report includes the key findings, conclusions and recommendations of the thematic country review on apprenticeships in Italy. It also offers a short description of the review’s rationale and methodology.
Evaluation framework for Modern Apprenticeships in Scotland, OECD
This evaluation framework aims to generate insights into the outcomes, benefits and challenges of Modern Apprenticeships in Scotland, thereby strengthening the links between apprenticeships and labour market policies. Furthermore, it can also provide policy-makers in other countries with a concrete example of an evaluation strategy in an actual policy setting.
The evaluation framework proposes recommendations for evaluating Modern Apprenticeships in Scotland, detailing various evaluation activities, scope of evaluation and data options. It outlines both long-term and medium-term evaluation strategies covering various methodologies, such as ex-ante strategy, use of multiple specific control groups, analysis of impact on well-being and on employer productivity, etc.