Evaluation of the SME Funding Schemes · A REVIEW OF RECOMMENDATIONS FROM ... • Prepare recommendations for optimising similar funding schemes in the future. ... Evaluation of the

Final Report

Evaluation of the SME Funding Schemes 2001-02 and 2002-03

June 2004

��

P O Box 159 Sevenoaks Kent TN14 5WT United Kingdom www.cses.co.uk

��

��

��

Evaluation of the SME Funding Schemes – Final Report

CONTENTS

��

��

SECTION PAGE

EXECUTIVE SUMMARY i

1. INTRODUCTION 1

1.1 Aims of the Evaluation

1.2 Methodological Approach and Workplan

1.3 Structure of the Final Report

1

1

3

2. BACKGROUND & KEY EVALUATION ISSUES 5

2.1 European Agency for Safety and Health at Work

2.2 SME Funding Schemes

2.3 Key Evaluation Issues

2.4 Project Typology

5

5

7

10

3. ANALYSIS OF SME FUNDING SCHEMES PROJECT PORTFOLIO 13

3.1 Applications and Awards

3.2 Analysis of Project Activities

3.3 Analysis of Project Funding

3.4 Nature of Partnerships

3.5 Health and Safety Issues and Target Sectors

3.6 Conclusions – Analysis of Project Portfolio

13

14

15

16

18

19

4. MANAGEMENT OF THE SME FUNDING SCHEMES 21

4.1 Promotion, Application and Selection Procedures

4.2 Reporting Requirements, Project Monitoring and Evaluation

4.3 Critical Success Factors in Project Implementation

4.4 Role of Unions, Employers, Workforce, the Agency and National Focal Points

4.5 Publicity and Dissemination

4.6 Programme Management Resources

4.7 Conclusions – Management of SME Funding Schemes

21

24

28

31

34

35

36


CONTENTS

��

��

5. EFFECTIVENESS OF THE SME FUNDING SCHEMES 38

5.1 Project Objectives and Activities

5.2 Financial Additionality

5.3 Assessment of Effectiveness – Achieving Objectives

5.4 Added Value of Project Outcomes

5.5 Sustainability of Projects and Outcomes

5.6 Conclusions – Effectiveness

38

40

43

50

53

56

6. CONCLUSIONS AND RECOMMENDATIONS 58

6.1 Overall Conclusions

6.2 Recommendations (General)

6.3 Recommendations (New Member States)

58

62

65

APPENDICES

PAGE

A REVIEW OF RECOMMENDATIONS FROM 2003 EVALUATION 70

B SURVEY QUESTIONNAIRE - 2001-02 SCHEMES 73

C SURVEY QUESTIONNAIRE - 2002-03 SCHEMES 77

D SURVEY OF SCHEMES WHICH WERE NOT ACCEPTED 83

E INTERVIEW CHECKLIST 86

F INTERVIEW LIST 87


Executive Summary

i

��

��

The Centre for Strategy & Evaluation Services (CSES) was commissioned by the European Agency for Safety and Health at Work to carry out an evaluation of the SME Funding Scheme.

Overall Summary

The SME Funding Scheme is a well-run programme which achieve useful results. It is relevant to the important health and safety issues faced by SMEs and shows a high degree of financial additionality.

Although the scale of the interventions (some �9 million over two years) is modest compared to most EU-funded programmes, the scheme achieves considerable added value, has beneficial impacts on the target group of SMEs, and has wider ‘demonstration’ effects.

In line with the overall conclusions set out above regarding its performance, there should be continued support, preferably on a larger scale, for the SME Funding Scheme on a multi annual basis, concentrating on those trans national projects which achieve the highest Community added value There also needs to be more emphasis on ensuring that the results of projects are disseminated as widely as possible.

Consideration needs to be given to the question of whether the SME Funding Scheme should continue in its current form as a separate scheme, and/or become part of larger EU-funded programmes (for example as a horizontal theme in ERDF, ESF and agriculture and fishery funding programmes).

In any case, the Agency should prepare guidance aimed at policymakers in regional authorities and other agencies and explaining what sorts of SME-related health and safety at work projects could be funded with EU support.

Following EU enlargement, there is a strong case for a special SME Funding Scheme for the New Member States. The report sets out a number of recommendations in how this particular initiative might be taken forward.

The aims of the evaluation, key findings, conclusions and full set of recommendations are summarised on the following pages.


Executive Summary

ii

��

��

1. Evaluation Aims

The main aims of the evaluation were, in summary, to:

• Assess the impact and sustainability of the 2001 projects over one year since their completion;

• Carry out a preliminary evaluation of the projects that were approved under the second scheme (2002);

• Prepare recommendations for optimising similar funding schemes in the future.

Details of the evaluation methodology and work plan are set out in the final report. The research involved a survey of project leaders, case studies, and an interview programme.

2. Key Findings and Conclusions

The overall conclusion of this evaluation is that the SME funding Scheme is a well-run programme that is achieving useful results in the field of health and safety at work.

Although the scale of the interventions is relatively modest compared to most EU-funded programmes, the scheme achieves considerable added value, has beneficial impacts on target group of SMEs, and, perhaps more significantly, has wider ‘demonstration’ effects by highlighting good practices that can be replicated more widely.

1. The SME Funding Scheme performs well in terms of the key programme evaluation issue of ‘relevance’. It achieves this by combining a focus on a high risk group (SMEs) in terms of health and safety at work with types of assistance and delivery mechanisms that are customised to the needs of smaller undertakings. Other factors that make the SME funding scheme highly ‘relevant’ include the non-bureaucratic nature of the procedures used to operate the SME Funding Scheme and this is one of the main advantages of it being a relatively small programme by EU standards.

2. The research suggests that there is a high degree of financial additionality. As indicated in the report, 37 of the 45 organisations (82%) responding to the survey would not have gone ahead on the same scale/timing, or at all, with their projects without Agency funding. The cases studies reinforce these findings since we were able to discuss what, if any, alternative sources of funding had been investigated by project leaders. In most cases, there appear to have been no specific schemes focusing on the promotion of


Executive Summary

iii

��

��

improved health and safety at work practices in SMEs available in the countries where project leaders were located. The SME Funding Scheme therefore fills a gap.

3. The scale of the interventions is relatively modest but nevertheless significant in terms of immediate impacts (‘outputs’ and ‘results’). It is likely that of the order of 700,000 SMEs will have benefited from the scheme in some way, either through direct advice, viewing a website or receiving written information during the 2002-03 period. Some 80,000 SMEs will have received direct advice. As the evaluation has shown, a wide range of SMEs has been involved directly in project activities which typically involved groups of between 10 and 50 undertakings being targeted by project leaders to test and help develop new ideas to tackle health and safety at work issues. Feedback provided in the ‘Final Activity Reports’ on workshops, conferences and advisory support provided to SMEs suggests that the immediate project outputs have been generally well received. Because of the relatively short duration of projects and the general absence of follow-up surveys and monitoring of ‘results’ there is, however, not much evidence available on the actions actually taken by target groups to improve health and safety practices, and the effect of this on the operations of the organisations concerned.

4. Notwithstanding the limited direct impacts, given the wide dissemination of results, ‘demonstration’ effects are likely to be considerable. As argued in the report, the real value of the SME Funding Scheme lies in the ‘demonstration’ effects it has. With annual funding of only around �4-5 million, the Scheme cannot hope to have extensive immediate impacts on SMEs – it is simply not operating on the scale across the EU required to achieve this sort of outcome. However, through an effective dissemination of project outcomes the Scheme can – and, the research suggests, does – reach a far larger number of SMEs. The evaluation suggests that there are various critical success factors in this respect. These include: building dissemination activities into the design of projects from the outset, producing best practice guidance and other material in a format that can be easily used by SMEs, translation of this material into different languages, etc. At a national and regional level, the structure of health and safety support organisations also has a bearing on how effectively project outcomes from the SME Funding Scheme are disseminated.

5. The added value of project outcomes is varies but particularly evident where projects have a trans-national character and/or there is an emphasis on a wide dissemination of results. As part of this study, we developed a project typology based on the key factors determining added value – the sort of health and safety at work issues addressed by projects and the way in which this is done. The case study work carried out for this


Executive Summary

iv

��

��

evaluation suggests that many projects are innovative in tackling issues that have not previously been considered, at least in an SME context; other projects are less innovative in this respect but still demonstrate added value by testing and developing particularly effective ways of delivering existing health and safety guidance to target groups. Projects that combine these attributes – new ideas and innovative ways of translating them into practical assistance for SMEs – are the ones that maximise added value. A significant proportion of the projects we examined in the case studies fell into this category. These tended to be the projects that had a trans-national character with partners in different countries and which placed emphasis on a wide dissemination of project outcomes through partner organisations, web sites, multi-language brochures, conferences, etc.

6. A key issue arising from the evaluation is whether the SME Funding Scheme should be targeted more narrowly on particular types of health and safety at work risks and/or beneficiaries. There are arguments for and against a more precise focusing of the SME Funding Scheme in terms of health and safety at work issues and/or target groups. Most of those we interviewed favoured a more targeted approach, arguing that because of the limited scale of the programme, this would maximise impacts. It was also pointed out that Agency’s national representative network (FOPs and others involved in implementation of the scheme could concentrate their (limited) resources more effectively. The alternative view is that the SME funding Scheme should be experimental and seek to test new ideas across a wide range of health and safety issues, focusing then on those that are appropriate to pursue in a more focused way. In practice, this latter course of action appears, however, to only be feasible if the project holder secures funding for a second, follow-up period of activity, either from the Agency or from another source (the case study work highlights examples of both these situations). On balance, the conclusion of this study is that the former approach is preferable, i.e. a greater thematic focus from the outset. This was a feature of the 2001-02 SME Funding Scheme (which focused on accident at work prevention) but not of subsequent programmes.

7. With respect to ‘effectiveness’, most projects achieve their immediate objectives but it is difficult to assess the final impacts on beneficiaries and these effects are uncertain. Analysis of the ‘Final Activity Reports’ on projects, feedback from the survey, and the case studies, suggests that in most cases project do achieve their stated objectives, typically carrying out research with a number of SMEs, producing a guide to good practice in the particular field of health and safety at work covered by the project, and disseminating the results. However, these objectives whilst important in their own right, are somewhat limited. Relatively few project holders carried out follow-up surveys and the other types of evaluation activities required to assess the longer term impacts on


Executive Summary

v

��

��

SMEs in terms of improved health and safety at work practices. Where this type of assessment was undertaken, the impacts appear to have been beneficial but there is insufficient evidence to generalise about the effectiveness of the SME Funding Scheme as a whole.

8. The research feedback strongly suggests that the one-year period for completion of projects is too short. The feedback from the research is clear and unambiguous in this respect. The main problem of timing faced by project leaders is that there is not enough time after completion of their schemes to disseminate the results or to undertake any evaluation of the outcomes achieved, in particular the impacts on SMEs and other beneficiaries.

9. Projects themselves, and the outcomes being achieved by them, demonstrate considerable sustainability. This is particularly important given the relatively short period allowed for completion of projects. The survey feedback provides strong evidence that project activities continue beyond the period of Agency support, and that partnership structures are durable. In the report we draw a distinction between the sustainability of project activities themselves and the lasting effects as far as raising awareness amongst SMEs of health and safety at work issues are concerned. On this latter point, the survey work also suggests a pronounced degree of sustainability (e.g. development of websites publicising good practices, reprints and renewed dissemination of brochures) although, as noted earlier, there is insufficient evidence from the research to draw firm conclusions on the extent to which SMEs adopt good practices and the longer term effect this has on their operations.

10. There is scope to further develop the role of National Focal Points although this would require additional resources being made available. Feedback from project leaders suggests that after the application procedures and appraisal of tenders is completed, most have little if any contact with FOPs. At the same time, the research suggests that a greater continuing involvement of FOPs in project activities could enhance the effectiveness of the SME Funding Scheme, particularly through more intensive trans-national networking, the relationship with national schemes and support for the dissemination of project outcomes, etc. The extent to which FOPs are engaged in such activities varies across countries for the reasons explained in the report. The Agency, Commission, and national authorities should review what can be done to ensure that sufficient resources are available to FOPs.

11. The SME Funding Scheme is administered largely on a centralised basis and although some tasks could be decentralised and undertaken by FOPs, the basic model


Executive Summary

vi

��

��

seems appropriate. Feedback from the research suggests that the SME Funding Scheme is being efficiently managed by the Agency. However, in line with the earlier conclusion (Point 10), some programme management tasks could be undertaken on a more decentralised basis. This includes the monitoring of project implementation activities as well as the initial screening of applications.

3.1 Recommendations (General)

12. In line with the overall conclusions set out above regarding its performance, there should be continued support, preferably on a larger scale, for the SME Funding Scheme. Increased funding is, in our view, justified given the positive contribution that the scheme is making to improving health and safety at work and because of EU enlargement. In addition to providing support for project activities, there is a need to ensure that the Agency itself receives the funding that is needed to carry out programme management tasks without having to divert resources from other sources for this purpose.

13. However, the Agency and EU institutions should consider whether the SME Funding Scheme should continue in its current form, i.e. as a separate scheme, or become part of a larger EU-funded programme. As argued earlier, these two options each have advantages and disadvantages. Feedback from those we have interviewed suggests strong support for the SME Funding Scheme to continue in its present form as a separate programme. We support this view but argue below (Points 18, 19 and 20) that, at the same time, there is a need for health and safety at work issues and support to be more comprehensively ‘mainstreamed’ in other EU programmes.

14. If the SME Funding Scheme continues in its present form, the funding arrangements should be altered to allow projects to be supported on a multi-annual basis. The previous sub-section, and the research findings set out in the report as a whole, explain the reasons why this is desirable. If the adoption of a multi-annual programming approach is not considered feasible, an alternative would be to place greater emphasis on supporting projects that build on the results of initiatives that are supported in earlier years. A variation on this would be to adopt a two-tier system – providing support in the first year for research and project development, and (conditional on the results of the first stage) then providing support during the second year specifically for dissemination activities.

15. Similarly, if the SME Funding Scheme continues in its present form, there should be a greater focus on the types of projects that deliver the highest Community added value. As noted earlier, added value tends to be maximised where projects combine a focus on


Executive Summary

vii

��

��

examining health and safety at work issues that have not been investigated before with innovative ways of disseminating project outcomes and engaging SMEs. Applicants for support should be specifically requested to explain how their projects are likely to achieve added value in these ways. Priority should be given to trans-national projects (and national projects that have the potential to be developed trans-nationally) since these seem, from our research, to lead to the highest added value.

16. There is a need for a ‘consolidation’ exercise pulling together the health and safety at work best practice lessons to be learnt from projects supported by the SME Funding Scheme. Over 140 projects have been supported since the Scheme was launched with an enormous amount of material being produced on how SMEs can improve health and safety practices. Although useful summarises are available on the Agency’s website, we recommend that an effort should be made to combine material concerning similar issues/sectors so that best practice guidelines and examples are brought together in a single publication. These could complement the material already produced by the Agency.

17. There also needs to be more emphasis on ensuring that the results of projects are disseminated as widely as possible. As argued earlier, the SME Funding Scheme cannot be expected to have extensive direct impacts given the modest scale on which it operates. However, the ‘demonstration effects’ of the scheme and indirect impacts on SMEs can be considerable if project outcomes are disseminated as widely as possible and in a form that makes it possible to do this trans-nationally. A number of initiatives might be considered: firstly, networking between project leaders from different countries, and between FOPs, should be further developed as a way of providing the necessary support structures for more extensive dissemination activities; secondly, there is a case for additional resources being made available specifically to ensure that outcomes from a project in one country are, if necessary, modified so that they can be applied in other countries; thirdly, and as argued above, consideration might be given to producing best practice guides (organised on a thematic basis, e.g. by sector or type of health and safety at work issue) that draw on the experience gained from projects.

18. Future interventions under the SME Funding Scheme should have a narrower thematic focus with support more clearly targeted on particular types of SMEs and health and safety at work issues. As noted in the report, the 2001-02 SME Funding Scheme had a relatively narrow thematic focus but this was subsequently broadened out. Feedback from the research suggests that a narrower focus leads to the programme being easier to manage and increases the likelihood of positive impacts. It would also make the task of


Executive Summary

viii

��

��

identifying good practices less complicated since the results of projects would be easier to compare.

19. Steps should be taken to ensure that synergies with other EU funded networks and programmes are maximised so that there is a leveraging of funding and resources from other sources to support projects. In carrying out the study, we found some evidence of projects combining support form the SME Funding Scheme with financial assistance that is available from other EU programmes (for example ESF, FEOGA and INTERREG). The Structural Funds (in particular, the European Social Fund – ESF) are an obvious source of additional funding for health and safety at work projects and initiatives and project leaders should be encouraged to make full use of such resources assuming they satisfy the eligibility requirements. A related recommendation is that more use should be made of other EU-funded networks (Euro Info Centres, European BICs, the Rural Carrefour, Innovation Relay Centres, etc) that focus on SMEs to promote the SME Funding Scheme.

20. Irrespective of the precise arrangements for operating the SME Funding Scheme in the future (see Point 13), health and safety at work should ideally be developed into a ‘horizontal’ theme in major EU-supported programmes such as the Structural Funds. The Structural Funds currently have three ‘horizontal’ or cross-cutting themes (equal opportunities, information society, and environmental sustainability). Project managers who receive ERSF and ESF grant aid are expected to explain at the application stage how they will address these themes in their projects and implementation of the horizontal priorities is subsequently a key feature of programme monitoring and evaluation activities. Ideally, health and safety at work should be treated in a similar way in the new (post 2006) Structural Fund programming period. In practice, this may be difficult to achieve because of the competing claims of different policy lobbies. It may therefore be more realistic to focus on ensuring that health and safety at work issues are addressed by particular Priorities and Measures that relate to SMEs (this would in any case by a necessary starting point to promoting health and safety at work as a ‘horizontal’ theme).

21. To help ensure that sufficient account is taken of health and safety at work issues in developing Structural Fund Priorities and Measures relating to SMEs, the Agency should prepare guidance aimed at policymakers in regional authorities explaining what sorts of health and safety projects are eligible for support. An analysis of some current Structural Fund programme documents shows that there is no direct reference to health and safety at work issues although many of the measures could have included such a reference. Programme documents are usually drawn up by policymakers who are generalist rather


Executive Summary

ix

��

��

then experts in any particular subject and they may well find such guidance helpful. The Agency should discuss with DG Regio the content of any proposed guidance and could well use its network of FOPs to ensure that managing authorities are aware of such guidance.

22 The Agency should investigate the possibility of similar guidance being included in other major EU funding initiatives, in particular the agricultural and fishery funds. Both these funds, which account for substantial Community expenditure, provide funding to industries which are dominated by SMEs and which have serious health and safety issues. There may be a possibility of leveraging further the use of Community funding to promote health and safety.

3.2 Recommendations (New Member States)

23. Whilst a transfer of know-how from the Agency’s EU15 SME Funding Scheme to the New Member States (NMSs) is desirable, this should be a two-way process. Although the NMSs have not operated the same sort of scheme as in EU15, they nevertheless have valuable experience of tackling health and safety at work issues.

24. Following EU enlargement, there is a strong case for a special SME Funding Scheme for the New Member States. Whilst such a programme could operate in the same way as the scheme has in EU15, i.e. providing support for new initiatives to develop better health and safety at work practices in SMEs, an alternative approach would be to devote the available resources exclusively to initiatives aimed at transferring ‘good practices’ that have already been developed under previous annual programmes to the New Member States. We would recommend that this alternative approach is adopted. Suggestions on how this approach might be taken forward are provided below.

25. An important early task should be to establish health and safety at work baselines in the New Member States for key industrial sectors, types of risk, and with regard to the relevant institutional/policy set ups. Having a clear picture of the situation in the New Member States with regard to safety and health at work practices, policies and institutional frameworks is vital if EU-funded interventions are to be appropriately targeted. The Agency has an important role to play in providing guidance on what sort of information is needed to compile the baselines, making contact with the authorities to obtain the required inputs, and then in analysing the results and identifying priorities. This is a potentially very demanding task and is probably best approached in stages by focusing on high risk safety and health at work issues and sectors in the first instance.


Executive Summary

x

��

��

26. There is a need to review experience from EU15 to identify ideas and good practices in the safety and health at work field that are especially relevant to the New Member States. As noted earlier (Point 16), there is an enormous amount of information available from EU15 project outcomes that is potentially relevant to the New Member States. Apart from the guidelines and other outputs produced directly by the EU15 projects themselves, the Agency’s website also includes useful information. The problem is that neither source is ideal, as things stand, as a guide to good practice – in the first case because there is too much information and in the second case because only project summaries are available. Best practice guides are needed on specific issues that combine the results of different projects in a concise but informative way with references to more detailed information and useful contacts. Consideration also needs to be given to translation into the various national languages in the New Member States to assist dissemination. Following on from the last recommendation, we suggest that the focus should initially be on types of safe and health at work issues and sectors that are especially high risk in the New Member States.

27. Support should be provided, where necessary, to help develop safety and health at work institutional capacity and policies in the New Member States. Extension of the Agency’s activities to cover the New Member States presupposes that appropriate supporting structures are in place in the various countries. This includes not only FOPs but also the wider structures needed to promote safety and health at work across different sectors and regions in the New Member States. In addition to the basic structures, there may be a need for capacity building and this should be one of the Agency’s key priorities in developing its role in the enlarged EU.

28. We also recommend that there should be a review of EU programmes now operating in the New Member States to establish (a) whether health and safety issues are being addressed, and (b) if not, the scope under existing programmes to do so. For example, a check should be made to see if the project appraisal criteria being used in Structural Fund programmes in the New Member States include reference to health and safety at work issues. A review by us of the programme documents in one New Member State suggests that there is virtually no focus on such issues. If this is typical of all 10 countries, then a key priority for the Agency should be to provide guidance to national authorities on how project appraisal criteria and procedures can be adapted to include health and safety at work issues, as described in 21 and 22 above. This exercise could involve producing a best practice exemplar/template and then holding workshops with programme managers from the New Member States, especially those responsible for implementing SME measures, to explain how the criteria and procedures should be applied in practice.


Executive Summary

xi

��

��

29. Steps should be taken to strengthen trans-national networks of safety and health at work organisations so that they include representatives from the New Member States and can be used to help transfer know-how. In addition to the networks and other health and safety at work structures that exist at a national level, the SME Funding Scheme has helped to develop trans-national linkages. This experience is especially relevant given other recommendations concerning the transfer of good practices set out in this section. One idea, for example, might be to support a mentoring initiative under which EU15 partners who have benefited from the Agency’s support are encouraged to transfer their know-how to similar organisations in the New Member States. We found some evidence of initiatives by particular EU15 project leaders to work more closely with partners in the accession countries but to be effective, this type of activity needs to be coordinated with perhaps limited funding being made available to cover travel expenses . This is a role that the Agency, supported by National Focal Points, could undertake.

30. Many of the suggested improvements to the SME Funding Scheme that has operated in EU15 are especially relevant to the New Member States and should be implemented there if a scheme is to be launched that goes beyond transferring best practices. If support is provided directly to organisations in the New Member States in a similar way to the EU15 SME Funding Scheme, then feedback on the schemes evaluated in this report should be taken into account, especially with regard to the time allowed for completion of projects, targeting of particular health and safety at work themes/risks, emphasis on projects that add value, and importance attached to dissemination and ‘demonstration’ effects.

The final report includes a suggested prioritisation of the above recommendations. Overall, priority should in our view be given to initiatives to tackle SME-related safety and health at work issues in the New Member States (Recommendations 23 to 30). Although recommendations relating to the current EU15 SME Funding Scheme would improve it, the most pressing need is to extend the scheme to member states that are not presently covered by it.


Section

Introduction

1

1

��

��

This document contains the final report prepared by the Centre for Strategy & Evaluation Services (CSES) for the assignment ‘Evaluation of SME Funding Scheme’ for the European Agency for Safety and Health at Work (the Agency).

1.1 Aims of the evaluation

The main aims of the evaluation were, in summary, to:

• Assess the impact and sustainability of the 2001 projects over one year since their completion;

• Carry out a preliminary evaluation of the projects that were approved under the second scheme (2002);

• Prepare recommendations for optimising similar funding schemes in the future.

The results of this evaluation are designed to satisfy a dual requirement – the need to demonstrate accountability and, looking forwards, to identify lessons from past experience that can be used to help improve future activities.

With the enlargement of the European Union occurring while the research was underway, particular emphasis was placed on assessing the implications of the evaluation for future activities in the 10 New Member States.

1.2 Methodological Approach and Work Plan

The evaluation was carried out over a six-month period. The diagram below shows the main steps in the evaluation. We describe these in more detail in the paragraphs following the diagram:

Phase 1 Phase 2 Phase 3

Preparatory TasksPreparatory Tasks Survey Work & InterviewsSurvey Work & Interviews Analysis & ReportingAnalysis & Reporting

� Set up meeting� Desk research� Methodology

� Survey of 103 projects� Case studies with sample of

35 projects� Interviews with Agency staff

and National Focal Points

� Analysis of research findings

� Preparation of draft final report

Inception Report Interim Report Final Report

January 30 March 21 June


Section

Introduction

1

2

��

��

The fieldwork for this evaluation combined a survey, face-to-face case studies, and an interview programme.

1.2.1 Project Survey

Three surveys were undertaken covering:

• the 51 projects which went ahead under the 2001/02 scheme;

• the 52 projects which went ahead under the 2002/03 scheme;

• 500 project applications from the two years which were not selected by the Agency for support.

Questionnaires were drafted in English and translated into four other EU languages (French, German, Italian, and Spanish). Copies of the questionnaires (English versions) are contained in Appendices B, C and D of this report.

The questionnaires were sent out in March 2004 by e-mail and also by post with a four-week deadline for completion. They were accompanied by a draft letter of introduction from the Agency. A telephone follow-up was undertaken to increase the response rates. The responses to each of the surveys were follows:

Table 1.1: Summary of Project Survey Responses

Survey Questionnaires sent out

Questionnaires returned

Response Rate (%)

2001-02 51 35 69

2002-03 51 46 90

Rejected applications 500 43 9 Source: CSES analysis of survey responses

In the case of the survey of the 2001-02 projects, where appropriate we included similar questions to those asked as part of the earlier evaluation1 so that comparisons could be made.

1 ‘Evaluation of the Small and Medium Sized Enterprises Accident Prevention Funding Scheme 2001-02’, IDOM Consultancy, 2003.


Section

Introduction

1

3

��

��

1.2.2 Case Studies with a Sample of Completed Projects

Based on a preliminary analysis of existing data, the survey responses, and discussions with the Agency, we selected 15 projects (i.e. a third of the total) from the first and second SME Funding Schemes as a sample for the interview programme for face-to-face interviews. Further project case studies were undertaken by telephone. The telephone interviews focused on specific issues that emerged from the in-depth face-to-face case study work and were used to determine the extent to which the case study findings were representative of the project portfolio as a whole.

The selection of projects for case studies was based on the following criteria:

• Two thirds were selected from the earlier 2001-2 scheme and one third from the 2002-3 scheme so as to concentrate on projects with a longer history and therefore to obtain more information on impacts, sustainability and additionality;

• Approximately two thirds of projects were national, and the remainder transnational (this represents a bias towards transnational projects);

• There was a spread across EU member states (for the face to face fieldwork, the 15 case studies were chosen from eight countries) and in terms of subject matter of the project (i.e. training, information and communications, best practice).

In Appendix D we set out the details of the projects covered by the case study work. A more detailed write-up of the case studies is provided separately.

As part of the evaluation, CSES also carried out interviews with National Focal Points, Agency staff and external stakeholders such as the European Commission. Details of these interviews are contained in Appendix F and the interview checklist is provided in Appendix E.

1.3 Structure of the Draft Final Report

The final report is structured as follows:

• Section 2: Background and Key Issues - describes the background to the SME Funding Scheme and sets out the evaluation framework and key issues;


Section

Introduction

1

4

��

��

• Section 3: Analysis of SME Funding Scheme Project Portfolio – examines key features of the supported projects including the number of applications and awards, distribution of funding, and beneficiaries;

• Section 4: Management of the SME Funding Scheme - examines the management of the 2001-02 and 2002-03 SME Funding Schemes and addresses issues relating to the key evaluation issue of ‘efficiency’;

• Section 5: Effectiveness of the SME Funding Scheme - evaluates the effectiveness of the projects and examines related issues such as financial additionality, project impacts and added value, and sustainability;

• Section 6: Conclusions and Recommendations - sets out the findings and recommendations from the evaluation. The recommendations are subdivided into two categories – general and those specifically relating to the New Member States.

The main body of the report is supported by a number of appendices containing an analysis of the current status of recommendations from the earlier evaluation (Appendix A), the survey questionnaire and other details relating to the evaluation methodology (Appendices B to E), and a list of external interviews (Appendix F).


Section

Background & Key Evaluation Issues 2

5

��

��

We begin by examining the background to the SME Funding Scheme (Section 2.1), key features of the 2001-02 and 2002-03 schemes (Section 2.2) and then consider key evaluation issues and related questions such as the project typology (Section 2.3).

2.1 European Agency for Safety and Health at Work

The European Agency for Safety and Health at Work was established under Council Regulation (EC) No 2062/94 of 18 July 1994 as subsequently amended2. The regulation sets out its objectives as follows:

’In order to encourage improvements, especially in the working environment, as regards the protection of the safety and health of workers as provided for in the Treaty and successive action programmes concerning health and safety at the workplace, the aim of the Agency shall be to provide the Community bodies, the Member States and those involved in the field with the technical, scientific and economic information of use in the field of safety and health at work.’

The Agency is given a specific remit in respect of small and medium-sized enterprises. For example, one of the tasks set out in the Regulation is to “supply the Community bodies and the Member States with the objective available technical, scientific and economic information they require to formulate and implement judicious and effective policies designed to protect the safety and health of workers; to that end, provide the Commission in particular with the technical, scientific and economic information it requires to fulfil its tasks of identifying, preparing and evaluating legislation and measures in the area of the protection of the safety and health of workers, notably as regards the impact of legislation on enterprises, with particular reference to small and medium-sized enterprises”; 2.2 SME Funding Schemes The European Parliament and the European Commission invited the Agency to run a series of annual funding schemes to encourage SMEs to take an active role in improving their levels of occupational safety and health. These schemes represent an acknowledgement at European level of the distinctive problems faced by SMEs and are an attempt to demonstrate that a small business initiative can be of value and be cost-efficiently organised.

2 Council Regulations 1643/95 and 1654/03 amend Regulation 2062/94.


Section


6

��

��

Three schemes have been run so far, in 2001-02, 2002-03 and 2003-04. The first scheme focused on accident prevention. The second scheme had risk reduction at its centre, and the third scheme currently up and running continues this theme. A total of �9 million has been allocated to the first two schemes. A third scheme for 2003/04 is currently underway. There were 769 eligible applications for support under the first two schemes leading to 104 projects being approved. A further 649 applications were submitted under the current 2003-04 scheme with 40 projects approved. The projects fall into two broad categories – trans-national and national projects. Grants are in the range �10,000 to �200,000. The objectives of the SME Funding Scheme are defined as follows:

Figure 2.1 SME Funding Scheme Objectives (2001-02)

• Raise awareness of accident risks and of the heavy load borne by workers and their families, as well as the considerable economic consequences for SMEs;

• Promote the development and the identification of effective good practice examples that reduce accident risks in SMEs, and facilitate their dissemination across Europe;

• Promote the development of risk evaluation and prevention practice as embodied in safety and health directives and the provision of early intervention/access to diagnosis and treatment, in particular in SMEs;

• Motivate duty holders to take action and to develop preventive measures;

• Encourage the development of sustainable and European added value occupational safety and health (OSH) activities involving employers, workers and their representatives or partnership programmes developed in particular by intermediaries who are working directly with SMEs to improve their practices, for example organising intervention programmes;

• Contribute to the reduction of the number and seriousness of work-related accidents in SMEs;

• Promote OSH at work as part of business thinking and organisational development and demonstrate to SMEs that ‘Good Safety and Health is Good Business’;

• Address the diversity of SMEs in Europe and respond to the specific needs of SMEs.

Source: Agency web site The 2002-03 scheme had similar objectives and provided co-funding for initiatives that encourage SMEs to adopt good occupational safety and health at work practices.


Section


7

��

��

Grants for projects are aimed at SMEs’ specific needs in one of three categories - training related to the prevention of health and safety risks; information and communication on health and safety prevention; and development of good practices that reduce health and safety risks. The types of projects that have been supported include workshops on risk management, training initiatives, publications, projects to develop good practices in the field of health and safety at work, and initiatives focusing on particular industrial sectors (agriculture, construction, printing, etc). In the section 3 of this report we have provided an analysis of the types of projects being carried out. An external evaluation of the first scheme was undertaken between November 2002 and March 2003.3 This research, which involved a survey of the 51 approved projects and National Focal Points, came to broadly favourable conclusions. In particular, the survey feedback suggested that in almost two-thirds of cases, project objectives had been achieved. Amongst the outcomes that were quantified were 250 publications, 500 information and communications activities, and an estimated 500,000 SMEs that had benefited either directly or indirectly from the project outcomes. In view of the brief period of time that had elapsed since the projects were completed, the evaluators indicated that it was not possible to fully assess wider impacts.

2.3 Key Evaluation Issues

In designing the evaluation framework for the SME Funding Scheme, we sought to ensure that the approach was consistent with guidelines on the evaluation of EU programmes generally.4 These identify five key evaluation issues:

Relevance – the extent to which the objectives of a programme are relevant to the needs of the target group(s), in this case to what extent are the measures implemented and targets achieved appropriate given the policy objectives of the SME funding schemes?

Effectiveness – the extent to which the outcomes achieved by interventions are in line with specific project objectives and overall programme goals to what extent is the SME funding schemes helping achieve the specific programme objectives and contributing towards the achievement of wider Commission aims with regard to reform;

3 ‘Evaluation of the Small and Medium Sized Enterprises Accident Prevention Funding Scheme 2001-02’, IDOM Consultancy, 4 See, for example, ‘Interim and Ex Post Evaluation of EU Expenditure Programmes’, DG Budgets, 1997, the MEANS documentation and other evaluation guidelines and working papers.


Section


8

��

��

Efficiency – the relationship between the financial inputs and outcomes, and value for money (whether the same level of financial inputs could have achieved more outcomes, or whether the same outcomes could have been achieved with lower financial inputs) how efficiently has the SME funding schemes been implemented;

Impacts and added value – the effect of interventions on beneficiaries and the extent to which, in the absence of EU support, these outcomes would have been possible;

Sustainability – the extent to which outcomes prove to be long-lasting, i.e. last beyond the period of EU support. Sustainability applies both to the schemes themselves and to the individual projects which have been supported under the schemes.

The overall evaluation framework for this assignment was summarised in a diagram contained in CSES’s tender document, reproduced below:

Figure 2.2: Evaluation Framework

P ro jec ts

T ra in ingT ra in in g In fo rm a tio nIn fo rm atio n G o od P ra c ticeG o o d P ra ctice

O u tp u ts

R esu lts

Im p a cts

N u m b er o f tra in in g a c tiv ities

N u m b er o f S M E s/w o rke rs b en e fitin g fro m tra in in g

N u m b er o f p u b lic a tio n s

N u m b er o f S M E s/su p p o rt o rg an isa tion s re ce iv in g in fo rm ation

� In creased S M E a w aren ess o f acc iden t p rev en tion and hea lth an d sa fe ty issues

� A ctio n s to im p ro ve p ractice s in S M E s

� B etter S M E p ra c tices lead in g to redu ced acc id en ts , b ette r sa fe ty an d hea lth p roced ures

D e ve lo p m en t o f g o o d p ra c tic e g u id es

N u m b er o f S M E s re ce iv in g g o o d p ra c tice g u id a n ce

Indirect effects of projects

Promotion and dissemination of project results

Employer and employee

organisations

Health and safety agencies and

business support organisations

Government and other public sector

organisations

SME associations and other networks

SME sector

Reduced absence from work

Better staff moraleReduced staff

turnover

Improved productivityImproved productivity

Improved business performance

Direct effects on targeted SMEs


Section


9

��

��

Reflecting the Commission’s evaluation guidelines, the above diagram distinguishes between ‘outputs’, ‘results’ and ‘impacts’ in tracing through the desired effects of the SME Funding Scheme. The evaluation of these effects has been approached in the following way:

• Direct outputs have been quantified using the Agency’s monitoring data with feedback from the Phase 2 interview programme being used to assess indirect outputs;

• Information to quantify the results and impacts of the funding scheme has come mainly from the case studies. Whilst some results and impacts have been possible to quantify, others have needed to be assessed in a more qualitative way.

The table below lists a number of specific questions covered as part of this evaluation.

Table 2.1: Key Evaluation Issues

Evaluation issues Key Questions

Efficiency • What is the relationship between financial inputs and outputs i.e. overall costs of the SME funding schemes/ number SMEs benefiting

• Are administrative requirements proportionate to size of grant awarded? And are they reasonable in relation to the size of the grant? How can administrative requirements be further simplified?

• How efficiently are funds disbursed?

• Are there administrative issues (eg annuality) which reduce the efficiency of projects

Effectiveness • What have been the impacts on intermediaries, ie those bodies which provide information or advise SMEs on health and safety at work issues?

• What have been the direct and indirect impacts on SMEs

• What has been the contribution of the social partners

• Are SME funding schemes an effective means of encouraging innovation and sharing of best practice?

• Is there any evidence on the relative effectiveness of various types of projects, for example training, information and communication or the provision of best practice?

• Is there any evidence on the relative effectiveness of national or EU projects?

• What opportunities are there for sharing experiences in SME funding schemes as a means of improving health and safety at work?

• Has technical assistance to projects in partner-finding, project preparation, application and


Section


10

��

��

project management a) been made available b) been effective?

Relevance and Utility

• From a workplace perspective, have SME funding schemes projects been incorporated into working practices?

• To what extent have SME funding schemes projects been relevant and useful from the perspective of SMEs participating in projects?

Sustainability • To what extent are SME funding schemes likely to continue beyond the life of the project? How long are benefits likely to be received?

• What are the characteristics of sustainable projects?

• What is the incidence of withdrawal from SME funding schemes projects by partners? How can this be minimised?

Value-Added • To what extent have projects encouraged innovation and creativity in health and safety at work practices?

• How do projects add value to work already carried out at national level, for example by encouraging cross border exchange of information?

• To what extent have the funds provided by the Agency attracted additional funding from other sources

• What is the value added provided by the Agency, particularly in respect of communication and publicity

2.4 Project Typology The Agency classifies projects that receive support under the SME Funding Scheme in a number of ways. Apart from the subject matter (in particular, type of risk) addressed by the project, type of intervention (training, information and communications, development of best practice), the year when funding is approved, the country where the project activities take place, a key distinction is made between ‘national’ and ‘trans-national’ projects. Underlying these factors is a more fundamental difference between projects in terms of two key variables that have a key bearing on added value - the health and safety issues being addressed and the project methodology used to tackle them (including the approach adopted to disseminating results). Figure 2.3 on the next page summarises the project typology.


Section


11

��

��

Figure 2.3: Typology of Project Added Value

In the above diagram, there are four possibilities:

• Scenario A - in this case, the project addresses issues that have not been investigated before using a ‘conventional’ methodology to tackle the issues;

• Scenario B - here, the health and safety issues have not been examined before and at the same time an innovative project methodology is adopted. In this case, added value is likely to be maximised;

• Scenario C – i.e. the reverse case where neither the issues nor the project methodology are new. Added value, at least at a Community level, is likely to be minimal;

• Scenario D – where the health and safety issues that have already been investigated but the project involves an innovative methodology, either in terms of achieving the project outcomes and/or disseminating these outcomes beyond the immediate target group.

The following examples illustrate this typology. Taking ‘Scenario B’ first, a significant proportion of the Agency-supported projects involve examining health and safety issues that have not been investigated before. Where this is combined with project methodology

A B

C D

New

Existing

NewExisting

Health & Safety Issues

Project Methodology


Section


12

��

��

that involves EU-wide dissemination of the results, and the approach to tackling health and safety issues is designed to be replicable, the added value of the Agency’s support is likely to be very high. The extent to which project outcomes can be replicated depends on a number of factors including whether the recommendations arising from project activities are generalisable beyond the immediate context, translation of material into other languages, but also on the extent to which the project outcomes are relevant given variations across countries in the regulatory environment for health and safety, and other differences.

In the case of ‘Scenario C’, none of these ‘critical success factors’ are likely to be demonstrated. Although there will be minimal added value at a Community level, it is still however conceivable that project results have benefits to the immediate target group and perhaps to others within the same region or country. The two other scenarios (A and D) involve situations that fall between those outlined above in terms of added value. For example, in the case of ‘Scenario D’, the health and safety issue addressed by a project may be well known, and ‘good practice’ guidelines may indeed already exist, but the project may still demonstrate added value by developing an innovative and more effective way of raising awareness of and tackling risks than was previously available. An example of this from our case studies was a project that developed a particularly innovative way of publicising safety and health at work risks using TV advertising.

Underpinning the typology outlined above is the argument that given the very modest financial support available to the SME Funding Scheme, added value lies less in extending the scale of intervention beyond what would be possible in the absence of the scheme and more in the ‘multiplier’ or ‘demonstration’ effects. The extent to which these effects come about depends crucially on ensuring that project outcomes are disseminated as widely as possible beyond the immediate and necessarily limited target group.

Later in this report, we draw on the results of the case study work to apply this typology of project added value.


Section

Analysis of SME Funding Scheme Project Portfolio

3

13

��

��

In this section, we examine the type of projects supported under the SME Funding Scheme during the 2001-02 and 2003-04 programming periods.

3.1 Project Applications and Awards

During the period covered by this evaluation, a total of 103 projects involving total costs of some �14.7 million received support from the Agency. Table 3.1 provides an analysis.

Table 3.1: SME Funding Scheme Applications (2001-04)

2001-02 Number of

eligible projects

Number of awarded projects

Total cost of awarded

projects (�m)

Agency contribution

(�m)

Intervention rate %

National 281 35 4.53 2.21 48.8 Transnational 129 16 3.58 2.28 63.6 Total 410 51 8.11 4.49 55.3 2002-03 Number of

eligible projects



projects (�m)

Agency contribution

(�m)

Intervention rate %

National 248 41 4.54 2.17 47.8 Transnational 111 12 2.03 1.51 74.5 Total 359 535 6.575 3.685 56.1

2003-04 Number of eligible projects



projects (�m)

Agency contribution

(�m)

Intervention rate %

National 437 26 3.10 1.63 52.7 Transnational 212 14 2.61 1.95 74.6 Total 649 40 5.71 3.58 62.7

Source: Agency report to EP Employment and Social Affairs Committee, January 2004

5 For 2002-03, the totals shown are in respect of the 53 projects initially awarded. Subsequently, two projects did not proceed. The data in the remainder of this section is in respect of the remaining 51 projects.


Section


3

14

��

��

The above analysis indicates that during the 2001-02 and 2002-03 periods:

• Relatively few applications were accepted - whereas 12% were accepted under the 2001-02 scheme, this rose slightly (14%) the following year;

• Taking the two schemes together, national projects had a slightly higher probability of being accepted than transnational projects (14% compared to 11%);

• The Agency’s intervention rate averaged around 55% with a significantly higher proportion of total transnational project costs being covered (67%) than was the case for national projects (54%).

3.2 Analysis of Project Activities

The Agency distinguishes between three basic types of projects (information and communication, good practice development, and training projects) and we have used these categories for the analysis. The principal activity recorded for each project is shown in Table 3.2 below.

Table 3.2: Analysis of Primary Project Activity

Type of activity 2001-02 2002-03 Total

Agency contribution

(�m) %

Agency contribution

(�m) %

Agency contribution

(�m) %

Information and communication 2.17 49 1.04 30 3.21 40

Good practice development 1.54 34 1.46 42 3.00 38

Training 0.78 17 0.99 28 1.77 22

Total 4.49 100 3.49 100 7.98 100 Source: CSES analysis of Agency data. See footnote 5.


Section


3

15

��

��

In practice, many projects cover more than one type of activity and the table above provides only a general indication of the split between different types of activities. Some two-fifths of funding has gone to each of information and communications, and good practice and development. One-fifth has gone to training.

3.3 Analysis of Project Funding

Table 3.3 provides an analysis of the cost of the projects and, more specifically, the size of the Agency’s contribution. As can be seen:

• Whereas national projects received funding of up to �90,000, transnational projects generally received amounts in excess of this, up to �200,000;

• In the case of the national projects, there was a shift between 2001-02 and 2003-03 towards smaller projects (68% received less than �60,000 in the second period compared with 37% during the first period);

• In the case of transnational projects, there was no significant change in the pattern of grant awards between the two programming periods.

Table 3.3: Distribution of Agency Contributions by Size

Funding Bands National Projects Transnational Projects 2001-02 2002-03 2001-02 2002-03 No. % No. % No. % No. % �10,000 - 49,999 8 23 15 39 0 0 0 0 �50,000 - 59,999 5 14 12 29 0 0 0 0 �60,000 - 69,999 5 14 2 5 1 6 0 0 �70,000 - 79,999 5 14 5 12 0 0 0 0 �80,000 - 89,999 11 32 6 15 1 6 2 18 �90,000 - 99,999 1 3 0 0 1 6 0 0 �100,000 - 119,999 0 0 0 0 0 0 3 27 �120,000 - 139,999 0 0 0 0 1 6 2 18 �140,000 - 159,999 0 0 0 0 6 38 1 9 �160,000 - 179,999 0 0 0 0 4 25 3 28 �180,000 - 200,000 0 0 0 0 2 13 0 0 Total 35 100 40 100 16 100 11 100

Source: CSES analysis of Agency data. See footnote 5.


Section


3

16

��

��

There are no national ‘quotas’ and applications for support from the SME Funding Scheme are assessed purely on their merits. As Table 3.4 shows, the number of applications from different countries – and the proportion accepted – has varied considerably with a similar pattern in this respect for both the 2001-02 and 2002-03 SME Funding Schemes.

Table 3.4: Comparison between Applications and Grant Awards

Country National Projects EU Projects 2001-02 2002-03 2001-02 2002-03 Applied Agreed % Applied Agreed % Applied Agreed % Applied Agreed %

Austria 9 2 22 6 2 33 4 1 25 4 1 25

Belgium 5 2 40 3 1 33 13 5 38 3 1 33

Denmark 5 1 20 14 3 21 2 1 50 4 1 25

Finland 3 2 67 3 2 67 2 1 50 0 0 0

France 19 3 16 25 4 16 8 1 13 5 0 0

Germany 15 2 13 23 4 17 13 2 15 9 2 22

Greece 9 2 22 11 3 27 13 1 8 9 0 0

Ireland 7 3 43 7 3 43 1 0 0 3 0 0

Italy 53 4 8 43 3 7 20 1 5 22 1 5

Luxembourg 2 0 0 3 1 33 2 1 50 1 0 0

Netherlands 6 2 33 5 1 20 2 0 0 3 0 0

Portugal 21 3 14 9 3 33 5 0 0 5 0 0

Spain 106 5 5 75 5 7 34 2 6 34 5 15

Sweden 1 1 100 4 1 25 2 0 0 1 0 0

United Kingdom

20 3 15 17 4 24 8 0 0 5 0 0

EU 0 0 0 0 0 0 0 0 0 3 0 0

Total 281 35 12 248 40 16 129 16 12 111 11 10 Source: CSES analysis of Agency data. See footnote 5 .


Section


3

17

��

��

3.4 Nature and Scope of Partnerships Projects supported under the SME Funding Scheme generally have a number of different partners (in the case of transnational projects this is of course a precondition). The analysis indicates that:

• In the case of national projects, there was an average of 1.2 partners per project (or 2.2 including the project leader) taking the two programming periods together;

• Not surprisingly, transnational projects had a larger number of partners per project (averaging 4.0 per project for the 2001-03 period as a whole);

• There was a tendency for the size of partnerships to increase during the period under review, this being especially pronounced with transnational projects where the number increased from an average of 3.1 during the first period to 5.2 during the second.

This last point is interesting and may reflect the fact that as the thematic focus of the SME Funding Scheme broadened, there was a need to draw in a greater variety of organisations to support the implementation of planned activities. Table 3.5 provides an analysis of the number of partners per project (it should be noted that the project leader is not included in the totals). Apart from more detail on the size of partnerships, the analysis also sheds further light on the geographical scope of project implementation activities.

Table 3.5: Analysis of Number of Partners per Project (excluding Project Leader)

Countries National Projects Transnational Projects 2001-02 2002-03 2001-02 2002-03 Projects Partners Projects Partners Projects Partners Projects Partners Austria 2 0 2 10 1 4 1 2 Belgium 2 4 1 3 5 13 1 4 Denmark 1 0 3 4 1 1 1 4 Finland 2 0 2 7 1 2 0 0 France 3 5 4 5 1 3 0 0 Germany 2 4 4 0 2 5 2 16 Greece 2 0 3 2 1 3 0 0 Ireland 3 1 3 2 0 0 0 0 Italy 4 7 3 6 1 5 1 7 Luxembourg 0 0 1 0 1 9 0 0


Section


3

18

��

��

Netherlands 2 3 1 2 0 0 0 0 Portugal 3 2 3 0 0 0 0 0 Spain 5 5 5 6 2 6 5 25 Sweden 1 1 1 0 0 0 0 0 United Kingdom 3 9 4 2 0 0 0 0 Total 35 41 40 49 16 51 11 58


3.5 Health and Safety Issues and Target Sectors

There are various other ways in which SME funding Scheme grant disbursements can be analysed. Perhaps the most interesting is by type of health and safety at work risk. Table 3.6 provides a breakdown in these terms for the 2002-03 scheme. Again, the analysis is not straightforward since some projects addressed more than one type of health and safety at work issue (‘General H&S’ accounts for the largest single category in the following analysis).

Otherwise, the most obvious conclusion to be drawn is that projects supported under the SME Funding Scheme addressed a very diverse range of subjects. The analysis below is limited to the 2002-03 scheme since the earlier programme focused exclusively on the accident prevention.

Table 3.6: Types of Safety and Health at Work Risks Addressed by Projects (2002-03)

Types of Risk National Transnational No. % No. %

General H & S 11 28 1 13 Accidents 9 22 2 25 Asbestos 1 3 0 0 Chemical, solvents 6 15 0 0 Ergonomic/stress 4 10 5 37 Other risks 9 22 3 25 Total 40 100 11 100



Section


3

19

��

��

With regard to the types of industrial sector that were targeted by projects, it is clear that taking the two years together the SME Funding Scheme covered a very wide range.

During the 2001-02 period, the scheme tended to focus on sectors with particularly high accident rates (for example, six projects targeted the construction industry) whereas in the second period, a wider range of sectors benefited from project activities. Table 3.7 provides an analysis.

Table 3.7: Types of Sectors Benefiting from SME Funding Scheme Projects

Sector National 2002-03 Transnational

No. % No. % Agriculture 2 5 1 9 Auto repair 4 10 0 0 Construction 4 10 0 0 Craft industry 2 5 0 0 Manufacturing 2 5 0 0 Metal 2 5 0 0 Textile 2 5 0 0 All/Various 9 22 10 91 Others* 13 33 0 0

Total 40 100.0 11 100.0 Source: CSES analysis of Agency data. See footnote 5 above Note: * this category includes sectors where in each case there was only one beneficiary, e.g. fishing industry.

3.6 Conclusions – Analysis of Project Portfolio

There are a number of conclusions to be drawn from the analysis in this section:

• Financing - the SME funding Scheme has operated on a financially modest basis. The total cost of the 103 projects accepted under the 2001-02 and 2002-03 schemes was some �14.68 million, of which �8.17 was made available by the


Section


3

20

��

��

Agency6. This underlines the importance of not overestimating the likely scale of impacts.

• Relatively few applications were accepted - 12% under the 2001-02 scheme and 14% the following year – whilst a low acceptance rate reflects the limited funding available, it also suggests that high standards were set in appraising applications.

• Whereas national projects received funding of up to �90,000, transnational projects generally received amounts in excess of this and up to �200,000. The Agency’s intervention rate was 55.6% with a significantly higher proportion of total transnational project costs being covered (67%) than was the case for national projects (54%).

• ‘Information and communications’ project activities accounted for the largest single proportion under the 2001-02 SME Funding Scheme. During the 2002-03 period, there was a quite marked shift towards ‘good practice development’ although, taking the two periods together, information and communication projects remained the largest category of projects receiving support. Given the limited scale of the Scheme, and the need to disseminate project outcomes as widely as possible, this emphasis on information and communications seems appropriate.

• There was a tendency for the size of partnerships to increase during the period under review, this being especially pronounced with transnational projects where the number increased from an average of 3.1 during the first period to 5.2 during the second. A possible explanation for this was the broadening out of the Scheme’s focus and consequent need to engage a more diverse range of organisations in projects.

6 These amounts include projects which did not proceed


Section

Management of the SME Funding Schemes

4

21

��

��

We turn in this section to programme management issues. After examining the way in which the SME Funding Scheme has been promoted and applications processed (Section 4.1), the assessment considers other aspects of programme management as well as the question of ‘critical success factors’ for projects.

4.1 Promotion, Application and Selection Procedures

We begin by considering the way in which the SME Funding Scheme is promoted, and the application and project selection procedures.

4.1.1 Promotion of the SME Funding Schemes

Most of those who have participated in the scheme heard of the SME Funding Scheme from the Agency’s website. The table below shows where successful applicants first heard of the 2002-03 scheme:

Figure 4.1: How did you hear about the Funding Scheme?

Source: survey of 2002-03 projects Feedback from interviews confirms that the Agency website has been the main source of information about the Scheme, with the number of potential applicants also coming forward as a result of seeing a Call for Proposals in the Official Journal and/or through

0

5

10

15

20

25

Official Journal

Website FOP Trade Assoc

Other


Section


4

22

��

��

word of mouth and press publicity. But as shown in Figure 3.1, National Focal Points (FOPs) have also had an important role in publicising the SME Funding Scheme. For example some FOPs contacted businesses directly to see if they would be interested in bidding for support from the SME Funding Scheme. The role of FOPs in this respect appears, however, to vary considerably from one country to another, partly reflecting the resources available. Discussions with National Focal points suggest that in some countries, they have been reluctant to publicise the scheme too much because of the limited opportunities available for funding. As the analysis in the previous section showed, the number of applicants has been substantial and most have had to be turned down - for the two schemes reviewed in this report, less than 15% of total applications were awarded.

4.1.2 Application Procedures

The application form used for the SME Funding Scheme consists of five sections covering aspects such as the project aims, partners, funding, and expected outcomes. The form is relatively brief by the standards of some EU programming documentation (for example, ERDF application forms) which is undoubtedly helpful since one of the main complaints made by organisations applying for EU funding is that application forms and procedures are unnecessarily complicated and time-consuming. The form has already been improved during the life of the various schemes.

The survey results and feedback from interviews suggested several ways in which the application form might be further improved:

• The question on added value (Section 4.3) could be more precise in asking applicants to explain how their project will add value to existing (non EU-funded) initiatives in their country/sector;

• It would be helpful to have a question in Section 4 asking applicants to explain what performance indicators will be used to assess the project’s success.

The earlier evaluation of the 2001-02 SME Funding Scheme also suggested improvements to the application form, in particular avoiding repetitive questions to shorten it. The evaluation also argued in favour of improved indicators to allow projects


Section


4

23

��

��

to be more effectively monitored and evaluated. Most of the earlier recommendations, especially with regard to financial procedures, have been adopted.

4.1.3 SME Funding Scheme Timescales

Feedback from the case studies and other interviews suggests that the timescales for preparing proposals have been rather tight, especially for SMEs and particularly where the projects have involved putting together partnerships covering several EU Member States. But it is difficult to see how these timescales can be extended without impinging on the time available to carry out the work.

Figure 4.1 summarises the timelines for the SME Funding Scheme’s operations.

Figure 4.1: Timeline for SME Funding Scheme Operations

1 2 3 4 5 6 7 8 9 10 11 12 1 2 3 4 5 6 7 8 9 10 11 12

Agr

eem

ents

sig

ned.

Pay

men

ts c

ompl

eted

.

Year 1 Year 2

Work period

Bud

get f

inal

ised

.

OJ

adve

rtise

men

t.

Ret

urn

of p

ropo

sals

.S

elec

tion

com

plet

ed .

Rep

ort.

Agr

eem

ents

sig

ned.

Pay

men

ts c

ompl

eted

.

Work period

Bud

get f

inal

ised

.

OJ

adve

rtise

men

t.

Ret

urn

of p

ropo

sals

.S

elec

tion

com

plet

ed .

Rep

ort.

Agr

eem

ents

sig

ned.

Pay

men

ts c

ompl

eted

.

Work period

Bud

get f

inal

ised

.

OJ

adve

rtise

men

t.

Ret

urn

of p

ropo

sals

.S

elec

tion

com

plet

ed .

Rep

ort.

Each Scheme takes place over a period somewhat in excess of two years. However, the various necessary administrative procedures mean that the time available to carry out individual projects is limited to about nine months in most cases. During the first year of the scheme, following the finalisation of the budget, a Call for Proposals is advertised in the Official Journal. A six month period then exists for the return of proposals and selection of projects to be funded. It is then necessary to complete contracts in time for work to start at the beginning of the second year. The last three months of the second year are taken up with the finalisation of reports, so the time available for field work and research is limited in practice to nine months.


Section


4

24

��

��

National and trans-national projects followed slightly different selection procedures, with the former first being assessed by National Focal Points and the latter being assessed only by an European jury. At the final stage the Agency makes an overall selection. We discussed the selection procedures with a number of National Focal Points and European jury members. Both appear to adopt appropriate criteria for the selection of projects. More generally, the application and selection are seen as working well. Indeed, the relatively non-bureaucratic nature of these procedures is seen as one argument for retaining a smaller programme rather than integrating the SME Funding Scheme into a larger EU programme.

4.2 Reporting Requirements, Project Monitoring and Evaluation

Overall, feedback from the research confirms that in contrast to many other EU programmes, the SME Funding Scheme is seen as relatively un-bureaucratic in terms of the requirements with regard to reporting, project monitoring and evaluation.

4.2.1 Monitoring and Reporting Requirements

Project activities are monitored by the Agency against the work plans and budgets set out in the applications. As part of the programme monitoring arrangements, project leaders are required to submit interim and final reports. A considerable amount of information on project activities, outcomes and other issues is provided in the Interim and Final Activity Reports. These reports are completed by the project leader and consist of two main sections:

• Part 1: General data – contact details for project leader, project title, names of partners, etc;

• Part 2: Detailed activity reports - activities undertaken with an emphasis on public events, nature of the ‘product’, role of key partners, feedback from beneficiaries, critical success factors in carrying out the project, etc.;

• Detailed financial statements – indicating how the grants provided to beneficiaries have been used and other information.


Section


4

25

��

��

As part of the evaluation, we reviewed the activity reports relating to the 2001-02 scheme. In general, it seems to us that the amount and quality of feedback on projects is of a high standard. The reports typically run to around 5-7 pages (excluding attachments such as copies of best practice guides, leaflets, conference agendas, etc) with a lot of useful information being provided on various aspects of the projects. This applies especially to the qualitative information (description of project activities, nature of products, etc) which is generally extremely detailed and comprehensive. Quantification of outcomes is also good at the level of ‘outputs’ (e.g. number of seminars and participants, number of publications disseminated, etc) but less so with ‘results’ and ‘impacts’ (e.g. number of SMEs benefiting, feedback from them on the benefits of the project activities, etc). National Focal Points appear to have had a very limited role, if any, in monitoring projects. This stems partly from the limited resources available to FOPs to monitor what are in some cases quite large project portfolios but also from the fact that because contracts are with the Agency, it is here that responsibility for monitoring is seen as lying.

4.2.2 Evaluation of Project Outcomes

There was a very mixed picture with regard to the evaluation of project outcomes. Whilst efforts were made to obtain feedback from SME beneficiaries immediately after the delivery of project activities (e.g. circulating questionnaires after a training seminar), and factors such as the number of publications disseminated and web site hits were monitored, there was far less emphasis on evaluating the benefits to target groups, in particular assessing whether changes in health and safety practices had occurred in SMEs.

The case study feedback provides further insights and confirms a very mixed picture with regard to evaluation of project outcomes.


Section


4

26

��

��

Evaluation of Project Impacts – Case Study Examples

Case Study 1 - Multicultural Health at Work – Migrants and Marginalised Groups (AU) The outcome of the project was a publication setting out the problems arising from stress at work in NGOs that help marginalised groups, and ideas on how these problems might be tackled. About 600 copies of the publication have been disseminated via conferences and other events in the three countries. In addition, copies have been downloaded from the project website (the number of downloads is not known). The project leader has not, however, undertaken any follow-up work to assess whether NGOs have acted on the ideas set out in the publication.

Case Study 2 - Prevention of knee disorders in the floor-laying trade (DK) Evidence of SME action as a result of the training can be found in the feedback survey, which shows that 85% had used the new tools in their work after the course and 43% used them weekly or daily. More than half found that work is faster with the new method and neck, shoulder, back and knee pain has been reduced significantly. A copy of the survey was forwarded to Bilbao. The survey took place 3 months after the courses and some 75% of respondents replied (although a number of telephone interviews had to be made). The project leader also proposed a proper 12 months feedback survey on project take-up. Whilst the Agency welcomed this suggestion, it did not fall within the activities that the Agency can fund in the time available to carry out the project work.

Case Study 3 - Implementation of Solvents Regulation in the Vehicle Repair Sector (D) No steps have been taken to assess whether the action plans are being implemented by the 20 SMEs directly participating in the project. The reason for this is that it is only in 2005 that the intermediate targets for a reduction in solvent emissions have to be met, and further visits to the SMEs are not scheduled until then.

A common factor limiting the scope for evaluation activities cited by interviewees was the one year duration of projects. This did not allow time for impacts to become apparent, let alone to be evaluated.

Generally, feedback from the research undertaken confirms that there is a very positive perception of the way in which the SME Funding Scheme is managed by the Agency. This is particularly so given that the funding package made available by the Agency did not provide for additional resources to cover administrative overheads and these have had to be covered in other ways.

In the survey we asked several more general questions about the application, project management and reporting requirements.


Section


4

27

��

��

Figure 4.2: Were the procedures formulated in a clear and understandable way, providing sufficient information for

potential applications?

40

41

42

43

44

45

46

Call for projectproposal

Applicationform/criteria

Technicalspecif ications

Reportingrequirements

Source: survey of 2002-03 projects

There was some difference between the reaction of respondents in respect of the call for proposals and the application form and criteria, which most project leaders participating in the survey considered easy to understand, and the technical specifications and reporting requirements which were seen as less understandable.7

4.2.3 Feedback on Reasons for Unsuccessful Applications

As part of the research, we surveyed a number of project leaders whose applications had not been accepted. The reason for doing this was partly to obtain further views on the Agency’s procedures for the SME Funding Scheme and partly to help shed light on the question of financial additionality (see Section 5.2).

Taking the first of these issues, a common theme was that unsuccessful applicants would have liked more feedback on the reasons for rejection of their projects. The Agency has provided a substantial amount of feedback where requested by unsuccessful project 7 These findings are broadly in line with the 2003 evaluation of the 2001-02 scheme when the survey established that 88% of respondents agreed that the various programme documents had been drawn up in a clear and comprehensible way.


Section


4

28

��

��

applicants and in some instances this has led to a long exchange of letters. But the general policy is not to provide detailed feedback in the first instance. With the large number of rejected applications, this is understandable given the considerable workload involved and the Agency’s limited resources.

4.2.4 Project Deadlines

Turning to project deadlines, we asked about their appropriateness both at the application stage and towards the end of projects.

Table 4.2: Were the deadlines appropriate for the following?

Response options Yes No % Yes Presenting the application 42 4 91

Achieving project objectives, outcomes, and targets 31 15 67

Submitting the final report 30 16 65


The deadlines for the application process were seen as reasonable by most project leaders, but those for completion of projects and reporting were only acceptable to about two thirds of survey respondents.

Feedback from the case studies and interviews reinforced the message that the one year period for projects is too short, especially if the results of projects are to be effectively evaluated and disseminated.8 In several cases, this problem was compounded by the fact that contracting procedures were only finalised one or two months into the project period. This feedback on the duration of projects is also to be found in the ‘Final Activity Reports’. We return to this issue elsewhere in the report.

4.3 Critical Success Factors in Project Implementation

A key issue addressed by the evaluation is how well projects were managed and the critical success factors. It is clear from the survey feedback, case studies, and other

8 In one case, the project leader tackled this problem is by obtaining support from the Agency for a second follow-up project under the 2003-04 SME funding programme.


Section


4

29

��

��

research that most projects achieved their stated objectives, in whole or in part. Feedback from project leaders shows the following position in respect of the 2002/03 scheme:

Figure 4.3: To what extent have the objectives

of your project been achieved?

0

5

10

15

20

25

30

Fully achieved Partially achieved Not achieved atall


The proportion of projects fully meeting their objectives – some two thirds - appears higher than for the equivalent question asked in the earlier evaluation of 2001/02 scheme when only 29% of the 2001-02 projects that had reached their objectives.

In the survey, we asked project leaders who had participated in the SME Funding Scheme what they saw as being the most important factors in ensuring successful project outcomes. The survey findings indicate that:

• The most important factor was having clear project aims and work plans;

• Other important factors were the support of partners and networking and sharing of good practice;

• Perhaps surprisingly, project management and achieving good publicity were rated as being less important.


Section


4

30

��

��

Table 4.3: Factors that are critical in ensuring successful project outcomes

Response option 2001-02 2002-03 No. % No. %

Having clear project aims 27 77.1 35 76.1

Support of partners 20 57.1 17 37.0

Funding arrangements 16 45.7 23 50.0

Project management 8 22.9 13 28.3

Networking/sharing good practice 17 48.6 18 39.1

Achieving good publicity 15 42.9 24 52.2

Source: survey of 2002-03 and 2001-02 projects

Viewed across the two programming periods, ‘having clear project aims’ remained the key factor but there was some difference in the perceived importance of other considerations. Whilst ‘achieving good publicity’ and ‘project management’ were seen as more important by 2002-03 project leaders than previously, other considerations are ranked less highly.

Feedback from the case study research sheds additional light on the factors that contribute to successful project outcomes. Carrying out research as part of the project design process to identify previous initiatives and publications covering a similar field was important in many cases in ensuring that project activities were innovative and added value to existing know-how. This was illustrated by a case study in Spain focusing on ergonomic stress: whilst several publications on the subject already existed, the project sponsor concluded that they did not adopt a sufficiently practical approach and more straight-forward guidance was needed for SMEs. Another critical success factor highlighted by the case studies was the need to work through business support intermediaries (e.g. chambers of commerce) to reach as many SME beneficiaries as possible. In one case, the supply chain was also being targeted as a way of having an impact on SMEs.

Ensuring that project outcomes could be disseminated across different countries also emerged as a key factor in maximising impacts. For example, in the case of one project, involving the production of TV adverts to publicise health and safety issues, messages were conveyed without using speech so that the film could be more easily used in


Section


4

31

��

��

different countries. In other cases, typically involving the preparation of good practice brochures, an emphasis was placed on illustrations as a way of getting simple messages across. More generally, the research suggests that an emphasis on disseminating project outcomes was the single most important critical success factor in ensuring positive impacts. As noted earlier, time constraints often meant that there was only limited scope to disseminate project outcomes during the period of the contract with the Agency but in many cases, efforts have continued beyond this to make use of the results.

4.4 Role of Unions, Employers, the Workforce, the Agency and National Focal Points

The analysis in the previous section highlights the importance of partnership working to ensure successful project outcomes.9 Many if not most of the ‘critical success factors’ depend on this.

4.4.1 Role of Social Partners

Many projects involved one or more social partners and an important issue is the extent to which these partners were fully involved in various stages of the projects. The survey findings indicate that:

• A relatively high percentage of all social partners received information on projects;

• Not surprisingly, the main form of participation by the workforce was in training, although 53% of workforce representatives participated directly in the project activities as well. Participation by unions was slightly lower although a third were involved in planning projects;

• The participation of employers was higher at each stage in the project cycle than was the case of other social partners except with regard to training.

9 The importance of partnership working to successful project outcomes was also a key conclusion from the earlier evaluation of the 2001-02 SME Funding Scheme.


Section


4

32

��

��

Table 4.4: To what extent have Social Partners been involved in projects?

Response options Workforce Unions Employers No. % No. % No. %

Participating in the planning of projects 15 33 19 41 26 57

Participating in project implementation 28 61 18 39 31 67

Training 29 63 11 24 21 46

Receiving information on projects 37 80 26 57 35 76

Participating in seminars 26 57 20 43 27 59

Other 6 13 6 13 8 17


4.4.2 Role of the Agency and National Focal Points

The Agency received high ratings in the survey for its role in supporting project activities and generally administering the SME Funding Scheme. With very few exceptions indeed, the feedback was very positive with project leaders stating that they had had a very good relationship with the Agency.10 Apart from its role in awarding grants and managing the Scheme generally, this relationship with project leaders typically involved responding to e-mails and telephone calls to clarify queries.

The survey feedback indicates a much lower involvement of National Focal Points in project activities and administration of the scheme following the selection procedures. Indeed, some of the projects leaders we talked to were not aware of the role of FOPs and did not know who they were. However, where FOPs did have an involvement, this was welcomed and generally felt to be helpful.

The role of FOPs in the SME Funding Scheme was investigated by us in more depth in the case studies and interviews. From the FOP perspective, a view expressed was that in many cases, there are simply not the resources available to become closely involved in projects.

10 There were also positive conclusions in the 2003 evaluation regarding the role of the Agency with two-thirds of the survey respondents stating that it provided an ‘excellent’ service.


Section


4

33

��

��

This point was made strongly by several FOPs who had other tasks to undertake and were only able to spend a limited proportion of his time on Agency work. The role of FOPs is also influenced by the health and safety system in different countries: where responsibilities are largely decentralised, the task of liaising with contacts is far more time-consuming and this makes it difficult to do many other things that might be expected of a FOP than in countries where more centralised structures exist. The research suggests that there are a number of other factors influencing the role played by FOPs in different countries – in particular, the seniority of the official concerned and, linked to this, the number of support staff available. Another factor, as noted earlier, is that because project leaders have a contract with the Agency (rather than the FOP), they inevitably look to this source for support.

Table 4.5 summarises the survey feedback on the role of the Agency and FOPs.

Table 4.5: How would you describe the help and advice received from the Agency and the National Focal Point?

Response option Agency Excellent Good Fair Poor Very poor Total

% Fair - Excellent

Answering questions 17 18 4 3 1 43 90.7

Giving additional information 15 16 10 3 0 44 93.2

Monitoring projects 13 18 8 4 2 45 86.7

Response option National Focal Points Excellent Good Fair Poor Very poor Total

% Fair - Excellent

Answering questions 2 10 4 1 1 18 88.9

Giving additional information 3 8 5 0 1 17 94.1

Monitoring projects 3 8 2 0 2 15 86.7


It is particularly noticeable that only about 15 to 18 project holders were able to rate FOPs, as compared to 43 to 45 who rated the Agency. The interviews we carried out suggest that many project holders simply do not know about the FOP system.


Section


4

34

��

��

4.5 Publicity and Dissemination

The importance of disseminating project outcomes has been noted earlier as a critical success factor. Given the modest scale of the SME Funding Scheme, direct impacts on SMEs are inevitably limited but through wider dissemination, indirect effects can be considerable.

The table below shows that the most common way of doing this was by using a web site to publicise results. In practice, a combination of dissemination methods were adopted to publicise project outcomes.

Figure 4.4: What type of information has been produced by the project?

0

5

10

15

20

25

30

35

Web site Brochure Guidelines


Our interviews with project leaders suggest that web sites are likely to be looked at by intermediaries, rather than by SMEs themselves unless a business was seeking to obtain an answer to a very specific question. The discussions suggested that for the purposes of raising awareness directly with SMEs, the best route was written material or face to face contact (e.g. training schemes, workshops) either with advisers or through SME support organisations.

This is an important issue because although a project can be effective in producing and testing ideas, to have an impact beyond those participating directly in a project, it is essential that the project results are effectively disseminated to as wide a range of SMEs


Section


4

35

��

��

as possible. Apart from the activities of project leaders themselves, the Agency also carries out a number of activities to publicise project outcomes. In particular, it prepares summaries of project outcomes both in hard copy format and on its website. The Agency has recently redeveloped its website and there are now a series of pages providing details of the results of the projects linked to the main SME Funding Scheme page http://sme.the Agency.eu.int/index_en.htm. Pages are available or are being developed in all EU languages.

We asked participants in the 2001-02 projects to rate the Agency’s help in publicising their project and disseminating outcomes. The results of this are shown in the table below. It can be seen that most participants rated the Agency’s help as either good or average. Relatively few respondents considered the Agency’s role in disseminating project outcomes as ‘excellent’ but it needs to be borne in mind that this feedback was obtained before the changes to the website referred to above.

Table 4.6: How would you rate the Agency’s help in publicising this project?

2001-02 SME Funding Scheme 2002-03 SME Funding Scheme Rating No. % Rating No. %

Excellent 6 17.1 Excellent 8 19.0

Good 19 54.3 Good 20 47.6

Average 10 28.6 Average 11 26.2

Poor 0 0.0 Poor 3 7.1

Total 35 100.0 Total 42 100.0


4.6 Programme Management Resources

The SME Funding Schemes are managed by a small unit within the Agency and also draws on the resources of National Focal Points. When the schemes were set up, no additional resources were provided for administration and the resources necessary to manage the scheme had to be provided by the Agency from its existing budget.

In practice, two or three Agency staff have been occupied on a full-time equivalent basis in the administration of the scheme, although because there are a number of other tasks carried out by the unit, and because the tasks are spread across the Agency to some


Section


4

36

��

��

extent, it is difficult to provide precise estimates of administrative inputs. However, it is likely that the full cost of administration is equivalent to around 5% of the SME Funding Scheme’s budget. CSES’s experience of carrying out evaluations of other SME support schemes suggests that this is a relatively low administrative cost.

4.7 Conclusions - Management of the SME Funding Schemes

In respect of the management of the SME Funding Scheme, we can draw the following conclusions:

• The Agency website is a major source of information about the scheme, although the numbers of applicants well exceeds the funds which have been available. Some National Focal Points are reluctant to publicise the scheme too far because of the lack of funding available

• There are some further modifications which could be made to the application form to improve it, in addition to those that have been made already;

• Most successful project applicants thought that the calls for proposals and subsequent procedures were clear. Feedback from some project leaders suggests that more time is needed to prepare project proposals;

• Unsuccessful project applicants would have welcomed more feedback, but there are considerable administrative burdens in doing this on a comprehensive basis;

• The support provided by the Agency was rated highly, as was the support received from FOPs. However, many project holders did not appear to be aware of the FOP system;

• With regard to timing, about a third of project holders would have welcomed a longer period for completion of projects. There are difficulties in running projects on a one year time frame, particularly with regard to the dissemination of project outcomes. The research also suggests that the relatively short time for projects makes it difficult to evaluate the impacts on target groups.

• Project leaders felt that clear project aims were important in achieving successful outcomes. Perhaps surprisingly, publicity was rated rather lower in the list of critical success factors. Most projects achieved their aims. The position of


Section


4

37

��

��

2002-03 Scheme in this respect would appear to be better than suggested in the previous evaluation

• The most common project outputs were web sites and brochures containing safety and health at work good practice guidelines. There is a clear need for a dissemination strategy for projects to ensure that the results are distributed as widely as possible;

• The research feedback suggests that the SME funding Scheme is well managed. Administration costs of the schemes appear low, and no additional funding was provided to cover these overheads. There is a need to consider the administrative burden of any future scheme.


Section

Effectiveness of the SME Funding Scheme

5

38

��

��

This section of our report sets out our findings on the effectiveness of the SME Funding Scheme, both in respect of the effect on the organisations carrying them out and on ultimate beneficiaries (SMEs).

5.1 Project Objectives and Activities

Although each project was different, they tended to follow a broadly similar course of action:

• A meeting of partners to refine the project aims and to plan activities;

• Development of a health and safety at work ‘product’ for SMEs in a particular sector;

• Dissemination material and events (training courses, seminars, conferences, etc) to publicise project outcomes and to encourage SMEs to improve health and safety at work practices.

The nature of the health and safety project activities supported under the 2001-02 and 2002-03 SME Funding Schemes varied considerably. In the first programme, the focus was on accident prevention. However, in the second programme, the scope of project activities was broadened out with a far wider range of activities being supported. Feedback from the research suggests mixed views about the merits and drawbacks of these differing approaches.

Some of those we interviewed argued that there needed to be a clearer focus on particular types of health and safety at work risks in order to maximise the benefits of the SME Funding Scheme. Thus, at the appraisal stage, the evaluation of project applications and their respective merits was, according to this view, easier if the projects focused on broadly similar themes. Similarly, the reduced multiplicity of project activities and official contacts in the selected health and safety field makes it easier to preserve a management overview, and to promote networking and a sharing of experiences and outcomes amongst project beneficiaries. The counter argument (expressed by fewer interviewees) was that the SME Funding Scheme is essentially experimental, tackling problems that are often new and developing innovative approaches to tackling them and,


Section


5

39

��

��

as such, it is appropriate to cover a broad range of health and safety issues. This approach was also seen as being more demand-led.

Examples of the project activities and outcomes from projects are shown below:

Project Activities and Outcomes – Case Study Examples

Case Study 1 - Prevention of knee disorders in the floor-laying trade (DK) Knowledge was developed into good practice by this training project, which selected 10 ‘super-users’ among the most respected artisans within the trade on a broad geographical basis. These experts went through a ‘train-the trainer’ programme in the use of the new working methods and tools that allow the job to be done in an up-right position. Subsequently, the new trainers carried out 47 training courses for 292 SME representatives in the floor-laying sector. This number corresponds to approximately 1/3 of the total potential target audience. A wide dissemination strategy has gone hand in hand with the training element to inform some 600-700 organisations about the new methods and the opportunity for training. 8 regional meetings were organised in the early parts of the project to sensitise the sector. A number of press articles have been written for the professional press and the project is described on the website of the union TIB (www.tib.dk/presse). A brochure explaining the project results was sent out to all floor-laying firms, trade unions and advisers on occupational health. The full project report was distributed to relevant authorities and Arbejdstilsynet.

Case Study 2 - Multicultural Health at Work – Migrants and Marginalised Groups (AT) A literature review carried out as part of one project established that whilst there is a considerable amount of previous research on stress at work, its causes and ways of tackling the problem, there is very little research specifically relating to organisations working with marginalised groups. It was argued that such organisations face a situation in which ‘client symptoms’ are transferred to professional workers. In particular, as with their clients, the professionals working with refugees and migrants tend to constitute a low status and often marginalised group. Additional pressures leading to stress amongst staff include insecurity caused by funding problems, and often the despair and hopelessness felt by refugees themselves. Following a survey of the staff of organisations from three countries, a publication was produced out the problems arising from stress at work in NGOs that help marginalised work, and ideas on how these problems might be tackled. About 600 copies of the publication have been disseminated via conferences and other events in the three countries.

Case Study 3 – Best Practice for Built Facilities throughout their Life Cycle (EU) This project was carried out by the social partners in the construction industry. In order to develop the best practice guide, an advisory group was formed which brought together European social partners. The project involved developing and testing ways of improving safety on construction sites. Considerable emphasis was placed by the project leader on disseminating the project outcomes as widely as possible. Some 1,500 hard copies of a Best Practice guide were printed with around 250 copies of each guide being translated into six different languages. The Guide was also presented formally to the European Construction Forum.


Section


5

40

��

��

5.2 Financial Additionality A key issue in evaluating the SME Funding Scheme is the extent to which it demonstrates financial additionality, in other words the extent to which grant aid is necessary to enable projects to go ahead. There are a number of possibilities:

• Full additionality – a situation where the project would not have gone ahead at all without Agency funding;

• Partial additionality – where the project would have gone ahead in the absence of Agency funding but on a reduced scale, in a different form, or at a later stage;

• Deadweight – a situation where in the absence of Agency funding, the project would have gone ahead anyway.

There is of course an additionality test in the application form for the scheme. But in the survey we also asked project leaders if they would have gone ahead without Agency funding. It will be seen from the table below that 37 out of the 45 2002-03 project leaders (82%) responding to the survey would not have gone ahead on the same scale/timing, or at all, without Agency funding.

Table 5.1: Would the project have gone ahead without Agency funding?

Response options No. % Yes, the project would have gone ahead without funding 2 4.3 The project would have gone ahead but on a reduced scale 12 26.1 The project would have gone ahead but on a reduced scale 3 6.5 No, the project would not have gone ahead at all without funding 26 56.5 Don’t know 3 6.5 Total 46 100.0


Agency funding has therefore been instrumental in enabling most projects to proceed. Only two of the 45 projects surveyed would have gone ahead in full without Agency funding. The survey feedback suggests that the additionality is higher for transnational projects than for purely national projects. National projects were more likely to have gone ahead anyway. An analysis is provided below in Table 5.2. As can be seen, whereas 53% of those who applied for Agency support for a national project stated that they would have not gone ahead without a grant, this figure was substantially higher (70%) for


Section


5

41

��

��

transnational projects. One factor that needs to be taken into account, of course, is that a higher intervention rate (80%) has been offered for trans-national projects compared with national projects (50%). The case studies provide further evidence to support the conclusion that the SME Funding Scheme demonstrates high levels of financial additionality.

Table 5.2: Analysis of Financial Additionality by Type of Project (2002-03)

Response National Transnational All Projects Number % Number % Total Total % Yes, with other funding 2 5.6 0 0.0 2 4.3 Yes, on a reduced scale 10 27.8 2 20.0 12 26.1 Yes, but at a later date 2 5.6 1 10.0 3 6.5 No, it would not have gone ahead 19 52.8 7 70.0 26 56.5 Don't know 3 8.3 0 0.0 3 6.5 Total 36 100.0 10 100.0 46 100.0


Financial Additionality - Case Study Examples

Case Study 1 - Implementation of Solvents Regulation in the Vehicle Repair Sector (DE) According to the project managers, the project would not have gone ahead at all without Agency funding. The reason for this is two-fold: firstly, although there are several schemes in Germany to help businesses comply with environmental legislation relating to the use of solvents, none specifically target SMEs or focus on EU rules that came into effect in 2001; and, secondly, in Berlin, there is no longer any support for such initiatives because of the difficult budgetary situation facing the city’s authorities. This view was confirmed by the FOP. Case Study 2 - Prevention of knee disorders in the floor-laying trade (DK) The project is unlikely to have gone ahead if funding from the Agency had not been obtained. Initially, the project leader tried to obtain national funding, but the available schemes were aimed at a much broader range of projects in the building and construction sector, and did not cater for projects specifically of this nature. The Department was informed about the SME Funding Scheme by the Danish authority that houses the FOP, who on a regular basis circulates project opportunities available through the Agency. The Scheme was targeted at exactly the type of project they had in mind and the eligibility criteria and application procedures were significantly simpler than for many other programmes. Case Study 3 - Multicultural Health at Work – Migrants and Marginalised Groups (AU) The project leader wanted to continue with existing research but lacked the funds to do so and could not obtain further support from national sources. The project would not therefore have gone ahead at all without Agency funding. It was argued that NGOs working in Austria with marginalised groups, in particular refugees and migrants, generally receive only modest financial support from the Austrian authorities and not enough to carry out additional activities such as the project supported by the Agency. Instead, most activities are funded from project income with a high proportion coming from various EU contracts.


Section


5

42

��

��

5.2.1 Feedback from Projects that did not go ahead

Further insights into the extent of financial additionality can be obtained by finding out what happened in cases where project leaders applied for funding from the Agency but whose applications were rejected.

As shown below, the majority of projects falling into this category (60%) did not go ahead with a further 26% going ahead only in part. Only 14% went ahead in full. The table below analyses the survey feedback from a survey of project leaders whose applications were rejected under the 2001-02 or 2002-03 schemes.

Table 5.3: What happened to the project?

Response No. % Project went ahead in full as planned 6 14

Project went ahead in part 11 26

Project did not go ahead 26 60

Total 43 100

Source: survey of 2001-02 and 2002-03 schemes

The responses to this question support the conclusion that SME Funding Scheme is critical in enabling projects to go ahead. The paragraphs below consider in more detail what happened to each category of project.

• Deadweight - two of the projects which subsequently went ahead in full were funded by a later Agency SME scheme. All the other projects which went ahead in full said that they used their own funding11 rather than any public source of funding. For this relatively small number of projects the Agency scheme would not have provided any additionality.

• Partial additionality - of the 11 projects which went ahead in part, 8 were still seeking further funding and three were not. Such a response is consistent with the suggestion that the SME Funding Scheme demonstrates a high level of additionality.

11 A larger number of projects answered this question than the numbers which said projects went ahead in full. However, all said they used their own funding


Section


5

43

��

��

• Full additionality - 26 projects did not go ahead and respondents indicated that in most cases they were still seeking funding. As shown in the table below, some three quarters of project teams (19 out of 26) were still seeking funding.

In a related question, we asked unsuccessful applicants whether they were still working together with their partners. The responses are summarised below alongside those relating to funding.

Table 5.4: If the project did not go ahead

Response Yes No Project team is still working together 19 17

Seeking further funding 7 9

Total 26 26


Overall, there seems little doubt that the Agency schemes have been instrumental in ensuring that a large number of projects have gone ahead. Without the SME Funding Scheme’s support, most projects would not have gone ahead.

5.3 Assessment of Effectiveness – Achieving Objectives

We now consider the question of effectiveness, in other words, the extent to which projects have achieved their aims and had the desired impacts. The overall objectives of the schemes were described earlier and can be summarised as follows:

‘The overall objective of the first SME scheme run by the Agency was to identify, communicate and support activities and projects with effective added value that motivate, develop, support and sustain effective organisation of OSH management in SMEs.’ (Source: the Agency call for proposals for the 2001-02 and 2002-03 Schemes.)

In considering the effectiveness of the two schemes, it is necessary to put the scale of the scheme into context. The 2001-02 Scheme had a budget of �5 million and the 2002-03 scheme a budget of �4 million. There are over 20 million SMEs in the EU and it would be unrealistic to suppose that the schemes could have any measurable effect on safety and health outcomes in the SME sector as a whole. In considering effectiveness, it is therefore necessary to rely on intermediate output measures for individual projects rather than any macro-level performance indicators.


Section


5

44

��

��

Another way of looking at the SME Funding Scheme is to view it as providing support for a number of ‘demonstration’ type projects. As such, large-scale impacts are not to be expected. However, to be ‘effective’, the results of projects need to be replicable on a wider scale and the mechanisms to ensure dissemination on an extensive basis have to be in place.

5.3.1 Benefits to Project Holders

In terms of immediate impacts (as opposed to those arising from a wider ‘demonstration’ effect), a distinction can be made between the benefits to project holders and those for target groups (SMEs).

As part of the survey, we asked whether organisations felt they had benefited from project activities. Details of responses are shown in the tables below. Generally, there is a similar picture for both schemes being evaluated.

Figure 5.1: To what extent do you think your organisation has benefited from the project?

2001-02 2002-03

0

5

10

15

20

25

30

A Lo t To s o meextent

No t a t a ll0

5

10

15

20

25

30

A lot To some extent Not at all


Feedback from case studies strongly confirmed this view. Two specialist trade associations we interviewed argued that they had been able to offer a useful additional service to their members, and that this had benefited the perception of the association. One of the associations found from surveying its members that they were now seen as


Section


5

45

��

��

offering an expert view of safety and health. In neither case had the associations had significant previous work in the safety and health area. Another smaller body we interviewed considered that it had benefited greatly from the project. It had carried out a project under the 2003-04 scheme as well and the outcome of this project had enabled it to bid successfully for ESF funding for other health and safety related work.

At the European level, several large representative organisations said that they strongly supported the scheme and, for this reason, felt that they had to offer projects which would support its overall aims. However, the direct benefit to those large organisations was not so marked as to small trade associations.

5.3.2 Reaching SMEs through Intermediaries

An important issue is the effect of the schemes on those organisations or bodies which advise SMEs on safety and health issues. Because of the limited scale of the SME Funding Scheme, ‘multiplier’ effects are likely to come about if intermediaries (and not just project holders and partners) act as a delivery mechanism for project outcomes.12 Intermediaries in this context are principally business support organisations but the category also includes larger companies and organisations that work on a commercial basis with SMEs.

The interview feedback suggested that one of the issues arising from projects was the degree to which projects were able to disseminate information, as compared to developing new ideas. We spoke to project holders who had put together web sites but who felt that the best way of reaching SMEs was through intermediaries who provided local training or contacts. These include business advisers, chambers of commerce, local trade associations and other similar bodies. Against this, there was a view from employers’ organisations that many SMEs are not members, or at least active members. The unions felt similarly, with a low level of workforce participation in SMEs.

12 Research feedback on the benefits to intermediaries is only available from the case study interviews and it might have been helpful to have specifically targeted a number of intermediate organizations to find out their views on the scheme.


Section


5

46

��

��

Reaching SMEs

One interesting example of achieving ‘multiplier’ effects by working closely with intermediaries is provided by a case study where a trade union orientated college had targeted SMEs in the retail trade through larger companies in the same area. Also, in the construction industry, small suppliers had been required to adopt particular safety and health practices. Safety and health workshops had been held at these larger bodies and had been well attended.

A further example came from the woodworking machinery industry where larger suppliers encouraged smaller distributors to become involved in safety and health issues by using their leverage throughout the supply chain. As customers bought woodworking machinery, so they are given safety and health advice based on the project part funded by the Agency.

5.3.3 Benefits to SMEs and Other Final Beneficiaries

SMEs are defined as the final beneficiaries of the Agency’s scheme. In this context, the term ‘SME’ covers a wide range of entities – most obviously small firms employing fewer than 250 people but also non-private sector organisations such as smaller community and voluntary organisations, trade bodies, and so on.

The best view of the benefits to SMEs comes of course from those organisations that were reached by the various projects. The scope of this study did not extend to surveying final beneficiaries directly, but we asked project leaders whether they had carried out a survey of beneficiaries, or were proposing to do so.

Table 5.5: Has a follow-up survey been carried out to assess project effects on SMEs?

2001-02 Scheme 2002-03 Scheme Responses No. % Responses No. %

Yes, survey completed 7 20.0 Yes, survey completed 7 15.6

Planned but not completed 10 28.6 Planned but not completed 13 28.9

Not planned 18 51.4 Not planned 25 55.6

Total 35 100.0 Total 45 100.0


It can be seen that seven project leaders from the 2001-02 Scheme and seven from the 2002-03 Scheme stated that they had carried out a follow up survey. We contacted these


Section


5

47

��

��

organisations to ask for details of the follow-up research but unfortunately only one of them provided information.

The case study that provided information on impacts was responsible for supplying relief workers to farms. They keep records of accidents to their staff and there is evidence that the project (to promote a preventative culture and risk assessments by farmers) which took place in 2001 may have helped reduce serious accidents in the past two years. We were told that the project had helped develop a culture of safety and this, in turn, had led to a reduction in accident rates. The serious accident rate has declined as follows:

Table 5.6: Serious Accident Rate - Relief Farmers, compared with National Average

Comparison 1998 1999 2000 2001 2002 2003

Case study 2.17 1.34 2.63 1.19 0.76 0.59

National 1.07 0.92 1.10 1.01 1.37 na Source: CSES case study interview

Feedback from interviews and case studies suggests that the benefits of the SME Funding Scheme are very diverse (and consequently difficult to measure):

• A case study in Ireland indicated that whilst the project had been highly effective in targeting a small niche industry (craft butchers) it was doubtful if the results of the project would be used in related industries or in other EU member states;

• In Belgium, one of the project leaders indicated that whilst a large number of good practices had been identified and publicised in a brochure and on a web site, the main objective to implementation was actually reaching the SMEs. During the case study, the comment was made that the project leader had earlier participated in health and safety work financed by ERDF funding, and that the funding had enabled a larger number of SMEs to be reached.

• One project leader had been able to negotiate a favourable insurance premium for businesses that went through the training course developed under the project. Given the cost of health and safety insurance claims, this point raises an interesting issue about the leverage to be obtained from tying safety and health training to insurance costs.


Section


5

48

��

��

More typically, the outcomes of projects – best practice guidelines disseminated via publications, web sites and events – will potentially have reached a very large number of SMEs. In many cases, however, the ‘products’ generated as a result of projects were targeted at intermediary organisations and there is consequently no way of estimating the number of final beneficiaries. Nevertheless, in the next section we provide broad estimates based on questions asked in the survey of project leaders.

5.3.4 Number of SMEs Benefiting from the Scheme

In carrying out the survey, we also asked about the numbers of SMEs benefiting directly and indirectly from projects. In the case of the 2001-02 Scheme, analysis of the survey results suggests that:

• 141,000 SMEs were provided with advice during the course of projects;

• There were over five million hits on web sites and information was provided in other ways to a further 175,000 SMEs;

Only about two-thirds of respondents were able to provide this data so the above returns are almost certainly underestimates of the numbers of SMEs contacted. There are also complications in interpreting some of the information provided by respondents.13 The 2002 -03 Scheme provided over 42,000 SMEs with advice, had 210,000 web site hits and provided 186,000 SMEs with written information. Table 5.7 provides a summary

Table 5.7: Estimated number of SMEs that have benefited from the Agency project

2001-02 Scheme 2002-03 Scheme

Responses A B C D Responses A B C

Received advice 23 6,143 141,000 66,000 Received advice 27 1,574 42,000

Viewed website 12 430,492 5,166,000 166,000 Viewed website 15 13,978 210,000 Received written info 20 8,744 175,000 50,000 Received written info 34 5,468 186,000

Total 5,482,000 282,000 Total 438,000

13 In respect of the 2001-02 scheme, one project indicated that they had provided 75,000 SMEs with advice, had five million hits on their website and had provided information to 125,000 SMEs. It is very likely that this includes areas other than those covered in the project they carried out for the Agency.


Section


5

49

��

��


Note: A = Number of respondents; B = Average number of beneficiaries claimed by respondents; C = Total. For the 2001-02 scheme a further total D is shown which excludes one Netherlands scheme where the data includes beneficiaries from other non Agency work

If we scale up the results of the 2002-03 scheme to allow for non respondents, it is likely that of the order of 700,000 SMEs will have benefited from the scheme in some way, either through direct advice, viewing a website or receiving written information. Of these 700,000 SMEs of the order of 80,000 SMEs will have received direct advice. We also carried out an analysis of ‘Final Activity Reports’. An analysis of the 40 documents (27 national and 13 EU) available for the 2001-02 SME Funding Scheme is summarised below.

Table 5.8: Summary Analysis of Outcomes – Final Activity Reports (2001-02)

Activities National EU Total

Conferences, workshops, etc 70 60 130

SMEs directly involved in project activities 8,400 900 7,300

Publications, CD-ROMS, etc, distributed 176,000 41,000 217,000 Source: analysis of final activity reports

Comparisons between Transnational and National projects

We also looked at the average number of SMEs benefiting from trans-national and national projects. National projects tended to be smaller and closer to individual SMEs, whilst by their nature of transnational projects are larger and often more conceptual.


Section


5

50

��

��

Figure 5.2: Beneficiaries of Trans-national and National Projects

Received specific advice

499

1762

0200

400600

8001000

1200

14001600

18002000

EU Nat

View ed Website

8143

30023

0

5000

10000

15000

20000

25000

30000

35000

EU Nat

Received w ritten information

5951

3213

0

1000

2000

3000

4000

5000

6000

7000

EU Nat


5.4 Added Value of Project Outcomes

We now consider the added value of the various projects carried out under the SME Funding Schemes. In theory, examples of added value could include:

• The project addressed an area of health and safety that that not been examined before;

• Alternatively, the project examined a familiar field of health and safety but nevertheless added value by developing innovative methods for disseminating guidance to SMEs;

• Or the project might have added value with regard to both subject matter and the method of dissemination. But this may have been in a purely national context, and added value may have been demonstrated by transferring ideas to other countries.

In Section 2.3, we suggested a project typology that combines the two key features contributing to added value – the nature of the health and safety at work issues being addressed and, secondly, the methodology used to tackle these issues.


Section


5

51

��

��

Figure 5.3: Project Typology

In applying this methodology, we have relied mainly on the case study feedback to arrive at conclusions. This has involved an independent assessment of the merits of different projects (e.g. in some cases, project leaders argued that their schemes were innovative but our conclusions from knowledge of the programme as a whole were different). It should be stressed that given the limited scale of this assignment, we were not able to investigate individual project activities and outcomes in depth or to form judgements applying beyond the case studies.

Notwithstanding these cautions, the basic conclusion of the research is that the 2001-02 and 2002-03 projects are distributed widely across the diagram’s quadrants. The majority of projects, however, fall within the bottom half of the diagram - Quadrants C in terms of health and safety issues but because many projects demonstrate innovative approaches to tackling these issues, this places them in this respect in Quadrant D.

The most innovative methodologies tended to be those developed in trans-national projects with partners from different countries bringing fresh ideas to tackling issues; similarly, where project outcomes were widely disseminated (again a tendency most apparent with trans-national projects), added value often lay in introducing SMEs to

A B

C D

New

Existing

NewExisting

Health & Safety Issues

Project Methodology


Section


5

52

��

��

practices that may not have been innovative in a different context. On the next page, we illustrate these findings with some examples from the case studies.

Added Value – Case Study Examples

Case Study 1 - Safety in Metal Working (IT) This project dealt with H&S in the metalwork industry before in the context of increasing the sector’s general awareness about the EU Directive on Safety. The scope of this project went much further, though, by providing practical training in the elementary rules and practices to follow to avoid explosions when handling high danger metallic substances. According to the project leaders, this type of information had not been provided to companies before, at least in Italy. The project is believed to have added value both with regard to the subject matter and the way in which it was disseminated in multimedia form.

Case Study 2 - Implementation of Solvents Regulation in the Vehicle Repair Sector (DE) The project leaders, argued that the added value of this project lay in encouraging SMEs operating in the vehicle paint-spraying sector to adopt measures to comply with EU health and safety regulations, specifically on reducing solvent emissions, and that there are no other initiatives in the Berlin-Brandenburg area seeking to do this. they argued that there is a particular need to help SMEs because they are more likely to put economic before environmental considerations, especially given the difficult trading conditions prevailing in Germany at present. Under the Agency agreement, visits were made to 20 SMEs. These were mostly existing clients but the Agency project made it possible to spend more time with them. The time was used to establish baselines on the use of solvents and then to prepare an action plan aimed at achieving the targets set out in the EU regulations. Case Study 3 – Learning from Near Accidents (AT) Although research has been undertaken previously on near accidents, the project leader had identified a gap in terms of appropriate guidance for SMEs. In particular, it was felt that more emphasis needed to be placed on providing learning tools so that SMEs have a structured framework for translating experience with near accidents into practical guidance that can be applied in the workplace. The project also demonstrated added value through its transnational features. The project leader linked up with the University of Lund in Sweden which had particular expertise in the field of near accident research.

Case Study 4 – Best Practice for Built Facilities throughout their Life Cycle (EU) In order to develop the best practice guide, an advisory group was formed which brought together trade unions, employer’s organisations and industry federations. While the project leader already had a good relationship with its partners at a national level, the project has improved co-operation considerably in respect of joint working on health and safety issues.

The development of the guide was thought to add considerable value by providing a bridge between EU directives and national legislation in respect of health and safety issues in the construction sector, on the one hand, and those working at the sharp end on building sites on the other. It was argued that there was a great need for practical and easily implementable guidance on how to apply basic health and safety principles at building site level. The Guide was therefore produced in a form that construction workers could easily relate to, e.g. with heavy pictorial content.


Section


5

53

��

��

An important issue is that of Community added value, or the added value that EU intervention has over and above interventions by initiatives supported by national safety and health bodies. One pointer in this area is the degree to which projects transfer experience from one country to another or one sector to another. Clearly, trans-national projects are more likely to be helpful in transferring experience and the data from the survey supports this contention. In respect of transferability, 87.5% of trans-national project holders said that their project outcomes could be easily transferred to other SMEs or sectors compared to only 29% of national project outcomes. That is not to say that in niche sectors, projects in a national area might not be transferable to another area.

Most projects involved, as was indicated at the beginning of this section, a research phase and a publicity or dissemination phase. A recurring theme of the case study interviews that we carried out was the need to concentrate more on dissemination. There are many studies carried out by others who bring together research on safety and health and indeed the Agency carries out work in this field and produces a series of guides. And there are without doubt gaps which need to be filled and areas where cross border cooperation would help improve practice. But the greatest need appears to be for greater dissemination and the time to ensure that this is carried out within the project.

5.5 Sustainability of Projects and Outcomes

Given that the SME Funding Scheme only makes support available for one year, the question of sustainability beyond this relatively short period is important. The question of sustainability applies both to the projects themselves and the outcomes being achieved, i.e. whether these are likely to be of lasting benefit to SMEs.

5.5.1 Sustainability of Projects

In the research, we asked whether project activities were continuing, if the project partners were still working together, and the source of any further funding.

Of the 18 project leaders that responded to this question as part of the 2001-02 scheme survey, 12 claimed to be continuing work. This is a good indicator that the subject matter of the project is continuing after the involvement of the Agency had ceased.


Section


5

54

��

��

Table 5.9: The extent to which the project partners still work together

Response options Regularly Sometimes Never On the subject matter of the project 19 5 2 On another OSH project 9 11 3 On other matters 13 7 4

Source: Analysis of 2001-02 projects

In terms of funding, most projects had sought further funding either from the own resources of the partners, or from other public or private sector sources

Table 5.10: Have you sought further funding to enable project activities to continue after

the Agency’s support ended?

Response options Yes % Own resources 24 68.6 Public funding 14 40.0 Private sector sources 10 28.6

Source: Analysis of 2001-02 projects Several interviewees told us that the source of further funding to continue work had included other EU funding. In particular: • One project had been able to obtain ESF funding to digitise a health and safety

training course;

• Another project had received funding under INTERREG 3c in France, Luxembourg and Belgium which had enabled it to transfer the results of a national project to the other countries covered by the programme;

• A third project leader was aware of FEOGA funding in the agricultural sector which could be used for SME advice in the health and safety field.

In the paragraphs below, we have described examples of financing used by other case studies to continue the work started by Agency projects:


Section


5

55

��

��

Project Sustainability – Case Study Examples

Case Study 1 - Multicultural Health at Work – Migrants and Marginalised Groups (AU) Because Omega have obtained funding from the Agency for a follow up project, activities that build on the 2002-03 project will continue until at least the end of 2004. The emphasis will be on improving the publication, producing a CD version, and on achieving wider dissemination.

Case Study 2 - Inside Sweden – a simple model of exchanging experiences designed for SMEs (SW) The result of an evaluation showed that the implementation time of the project was too short, that marketing was insufficient and that there were not enough companies in the database. All of these aspects could have been remedied by a longer project period than 1 year. Subsequently, the project leaders have persuaded VINNOVA, the Swedish Agency for Innovation Systems, to finance the continuation of the project in a third phase running up till August 2005. Case Study 3 – Best Practice for Built Facilities throughout their Life Cycle (EU) The project is likely to demonstrate long-term sustainability. The project director is contemplating revising the best practice guide every few years to ensure that it remains up to date and relevant to those working in the construction and wood working sectors as part of the their commitment to continuous improvement in the field of health and safety. The process of continuous improvement will be facilitated by cooperation between the employers, unions and workforces.. They have invited key partners from a variety of disciplines including construction, engineering, architecture etc. to submit formal comments on how the good practice guide might be improved in future.

5.5.2 Sustainability of Impacts on SMEs

As far as SMEs are concerned, we asked project leaders about the extent to which they are still benefiting from results of projects. The views of those carrying out the projects are that SMEs continue to benefit, substantially or to some extent. Again, the position is similar for both years.

Figure 5.4: To what extent are SMEs still benefiting from the results of the project? 2001-02 2002-03

0

5

10

15

20

25

A Lo t To s o m ee xte nt

No t a t a ll 0

5

10

15

20

25

30

A lo t To s o m e e xte nt No t a t a ll



Section


5

56

��

��

There was also a general view amongst those carrying out projects that SMEs would continue to benefit for more than a year, suggesting that they had a view that benefits would be sustainable. Again, the position was similar for both years.

Table 5.11: How long will SMEs continue to benefit from the project?

Responses 2001-02 Scheme Responses 2002-03 Scheme

No. % No. %

Up to a year 2 6.1 Up to a year 3 6.8

More than a year 31 93.9 More than a year 41 93.2

Total 33 100.0 Total 44 100.0

Source: Analysis of 2001-02 projects and 2002-03 projects

5.6 Conclusions - Effectiveness

Some of the issues which are emerging from the evaluation are:

• A good level of financial additionality, supported both by results from projects which have gone ahead and those which have not;

• Positive outcomes for project holders, particularly the smaller organisations carrying out projects. Trade associations saw projects as a means of providing additional support to their members and some other bodies saw the projects as a means of extending their involvement with OSH issues;

• Less information is available on the effect on SME support organisations generally. It is likely that intermediaries will be a key aspect of any strategy to disseminate results to SMEs and in any future scheme could well be a key target group;

• Interesting examples of how to approach SMEs through the supply chain and through reduced insurance premiums. SMEs often have low involvement with trade associations and have workers who are not members of a union, so cannot be reached though those routes. It is therefore necessary to use innovative marketing strategies to reach SMEs;


Section


5

57

��

��

• Sustainability – strong evidence that project activities continue beyond the period of Agency support and a view from project holders that SMEs will continue to benefit from their work into the future;

• The ability of project holders to access other EU funding should be encouraged, for example ESF, FEOGA and INTERREG to carry out health and safety work and in some cases to extend the work carried out under an Agency project;

• Evidence of large-scale awareness raising – based on the survey returns, an estimated 438,000 SMEs have been reached by the 2002-03 scheme. When this is scaled up for non respondents, an estimated 700,000 SMEs will have benefited from the scheme in some way, either through direct advice, viewing a website or receiving written information;

• The equivalent figure for the 2001-02 SME Funding Scheme is much higher but most of this is accounted for by one project which had of the order of five million hits to a website which also included other OSH issues;

• Evidence that there is greater Community added value from trans-national projects rather than national projects, underlining the need for effective dissemination strategies.


Section

Conclusions & Recommendations

6

58

��

��

In this final section we provide a resume of the overall conclusions of the evaluation of the 2001-02 and 2002-03 SME Funding Schemes and then present recommendations for the future of the programme. 6.1 Overall Conclusions

The overall conclusion of this evaluation is that the SME funding Scheme is a well-run programme that is achieving useful results in the field of health and safety at work.

Although the scale of the interventions is relatively modest compared to most EU-funded programmes, the scheme achieves considerable added value, has beneficial impacts on target group of SMEs, and, perhaps more significantly, has wider ‘demonstration’ effects by highlighting good practices that can be replicated more widely. 1. The SME Funding Scheme performs well in terms of the key programme evaluation issue of ‘relevance’. It achieves this by combining a focus on a high risk group (SMEs) in terms of health and safety at work with types of assistance and delivery mechanisms that are customised to the needs of smaller undertakings. Other factors that make the SME funding scheme highly ‘relevant’ include the non-bureaucratic nature of the procedures used to operate the SME Funding Scheme and this is one of the main advantages of it being a relatively small programme by EU standards.

2. The research suggests that there is a high degree of financial additionality. As indicated in the report, 37 of the 45 organisations (82%) responding to the survey would not have gone ahead on the same scale/timing, or at all, with their projects without Agency funding. The cases studies reinforce these findings since we were able to discuss what, if any, alternative sources of funding had been investigated by project leaders. In most cases, there appear to have been no specific schemes focusing on the promotion of improved health and safety at work practices in SMEs available in the countries where project leaders were located. The SME Funding Scheme therefore fills a gap. 3. The scale of the interventions is relatively modest but nevertheless significant in terms of immediate impacts (‘outputs’ and ‘results’). It is likely that of the order of 700,000 SMEs will have benefited from the scheme in some way, either through direct advice, viewing a website or receiving written information during the 2002-03 period. Some 80,000 SMEs will have received direct advice. As the evaluation has shown, a wide range of SMEs has been involved directly in project activities which typically involved groups of between 10 and 50 undertakings being targeted by project leaders to test and help


Section


6

59

��

��

develop new ideas to tackle health and safety at work issues. Feedback provided in the ‘Final Activity Reports’ on workshops, conferences and advisory support provided to SMEs suggests that the immediate project outputs have been generally well received. Because of the relatively short duration of projects and the general absence of follow-up surveys and monitoring of ‘results’ there is, however, not much evidence available on the actions actually taken by target groups to improve health and safety practices, and the effect of this on the operations of the organisations concerned. 4. Notwithstanding the limited direct impacts, given the wide dissemination of results, ‘demonstration’ effects are likely to be considerable. As argued in the report, the real value of the SME Funding Scheme lies in the ‘demonstration’ effects it has. With annual funding of only around �4-5 million, the Scheme cannot hope to have extensive immediate impacts on SMEs – it is simply not operating on the scale across the EU required to achieve this sort of outcome. However, through an effective dissemination of project outcomes the Scheme can – and, the research suggests, does – reach a far larger number of SMEs. The evaluation suggests that there are various critical success factors in this respect. These include: building dissemination activities into the design of projects from the outset, producing best practice guidance and other material in a format that can be easily used by SMEs, translation of this material into different languages, etc. At a national and regional level, the structure of health and safety support organisations also has a bearing on how effectively project outcomes from the SME Funding Scheme are disseminated. 5. The added value of project outcomes is varies but particularly evident where projects have a trans-national character and/or there is an emphasis on a wide dissemination of results. As part of this study, we developed a project typology based on the key factors determining added value – the sort of health and safety at work issues addressed by projects and the way in which this is done. The case study work carried out for this evaluation suggests that many projects are innovative in tackling issues that have not previously been considered, at least in an SME context; other projects are less innovative in this respect but still demonstrate added value by testing and developing particularly effective ways of delivering existing health and safety guidance to target groups. Projects that combine these attributes – new ideas and innovative ways of translating them into practical assistance for SMEs – are the ones that maximise added value. A significant proportion of the projects we examined in the case studies fell into this category. These tended to be the projects that had a trans-national character with partners in different


Section


6

60

��

��

countries and which placed emphasis on a wide dissemination of project outcomes through partner organisations, web sites, multi-language brochures, conferences, etc. 6. A key issue arising from the evaluation is whether the SME Funding Scheme should be targeted more narrowly on particular types of health and safety at work risks and/or beneficiaries. There are arguments for and against a more precise focusing of the SME Funding Scheme in terms of health and safety at work issues and/or target groups. Most of those we interviewed favoured a more targeted approach, arguing that because of the limited scale of the programme, this would maximise impacts. It was also pointed out that Agency’s national representative network (FOPs and others involved in implementation of the scheme could concentrate their (limited) resources more effectively. The alternative view is that the SME funding Scheme should be experimental and seek to test new ideas across a wide range of health and safety issues, focusing then on those that are appropriate to pursue in a more focused way. In practice, this latter course of action appears, however, to only be feasible if the project holder secures funding for a second, follow-up period of activity, either from the Agency or from another source (the case study work highlights examples of both these situations). On balance, the conclusion of this study is that the former approach is preferable, i.e. a greater thematic focus from the outset. This was a feature of the 2001-02 SME Funding Scheme (which focused on accident at work prevention) but not of subsequent programmes. 7. With respect to ‘effectiveness’, most projects achieve their immediate objectives but it is difficult to assess the final impacts on beneficiaries and these effects are uncertain. Analysis of the ‘Final Activity Reports’ on projects, feedback from the survey, and the case studies, suggests that in most cases project do achieve their stated objectives, typically carrying out research with a number of SMEs, producing a guide to good practice in the particular field of health and safety at work covered by the project, and disseminating the results. However, these objectives whilst important in their own right, are somewhat limited. Relatively few project holders carried out follow-up surveys and the other types of evaluation activities required to assess the longer term impacts on SMEs in terms of improved health and safety at work practices. Where this type of assessment was undertaken, the impacts appear to have been beneficial but there is insufficient evidence to generalise about the effectiveness of the SME Funding Scheme as a whole. 8. The research feedback strongly suggests that the one-year period for completion of projects is too short. The feedback from the research is clear and unambiguous in this


Section


6

61

��

��

respect. The main problem of timing faced by project leaders is that there is not enough time after completion of their schemes to disseminate the results or to undertake any evaluation of the outcomes achieved, in particular the impacts on SMEs and other beneficiaries. 9. Projects themselves, and the outcomes being achieved by them, demonstrate considerable sustainability. This is particularly important given the relatively short period allowed for completion of projects. The survey feedback provides strong evidence that project activities continue beyond the period of Agency support, and that partnership structures are durable. In the report we draw a distinction between the sustainability of project activities themselves and the lasting effects as far as raising awareness amongst SMEs of health and safety at work issues are concerned. On this latter point, the survey work also suggests a pronounced degree of sustainability (e.g. development of websites publicising good practices, reprints and renewed dissemination of brochures) although, as noted earlier, there is insufficient evidence from the research to draw firm conclusions on the extent to which SMEs adopt good practices and the longer term effect this has on their operations. 10. There is scope to further develop the role of National Focal Points although this would require additional resources being made available. Feedback from project leaders suggests that after the application procedures and appraisal of tenders is completed, most have little if any contact with FOPs. At the same time, the research suggests that a greater continuing involvement of FOPs in project activities could enhance the effectiveness of the SME Funding Scheme, particularly through more intensive trans-national networking, the relationship with national schemes and support for the dissemination of project outcomes, etc. The extent to which FOPs are engaged in such activities varies across countries for the reasons explained in the report. The Agency, Commission, and national authorities should review what can be done to ensure that sufficient resources are available to FOPs. 11. The SME Funding Scheme is administered largely on a centralised basis and although some tasks could be decentralised and undertaken by FOPs, the basic model seems appropriate. Feedback from the research suggests that the SME Funding Scheme is being efficiently managed by the Agency. However, in line with the earlier conclusion (Point 10), some programme management tasks could be undertaken on a more decentralised basis. This includes the monitoring of project implementation activities as well as the initial screening of applications.


Section


6

62

��

��

6.2 Recommendations (General) 12. In line with the overall conclusions set out above regarding its performance, there should be continued support, preferably on a larger scale, for the SME Funding Scheme. Increased funding is, in our view, justified given the positive contribution that the scheme is making to improving health and safety at work and because of EU enlargement. In addition to providing support for project activities, there is a need to ensure that the Agency itself receives the funding that is needed to carry out programme management tasks without having to divert resources from other sources for this purpose. 13. However, the Agency and EU institutions should consider whether the SME Funding Scheme should continue in its current form, i.e. as a separate scheme, or become part of a larger EU-funded programme. As argued earlier, these two options each have advantages and disadvantages. Feedback from those we have interviewed suggests strong support for the SME Funding Scheme to continue in its present form as a separate programme. We support this view but argue below (Points 18, 19 and 20) that, at the same time, there is a need for health and safety at work issues and support to be more comprehensively ‘mainstreamed’ in other EU programmes. 14. If the SME Funding Scheme continues in its present form, the funding arrangements should be altered to allow projects to be supported on a multi-annual basis. The previous sub-section, and the research findings set out in the report as a whole, explain the reasons why this is desirable. If the adoption of a multi-annual programming approach is not considered feasible, an alternative would be to place greater emphasis on supporting projects that build on the results of initiatives that are supported in earlier years. A variation on this would be to adopt a two-tier system – providing support in the first year for research and project development, and (conditional on the results of the first stage) then providing support during the second year specifically for dissemination activities. 15. Similarly, if the SME Funding Scheme continues in its present form, there should be a greater focus on the types of projects that deliver the highest Community added value. As noted earlier, added value tends to be maximised where projects combine a focus on examining health and safety at work issues that have not been investigated before with innovative ways of disseminating project outcomes and engaging SMEs. Applicants for support should be specifically requested to explain how their projects are likely to


Section


6

63

��

��

achieve added value in these ways. Priority should be given to trans-national projects (and national projects that have the potential to be developed trans-nationally) since these seem, from our research, to lead to the highest added value. 16. There is a need for a ‘consolidation’ exercise pulling together the health and safety at work best practice lessons to be learnt from projects supported by the SME Funding Scheme. Over 140 projects have been supported since the Scheme was launched with an enormous amount of material being produced on how SMEs can improve health and safety practices. Although useful summarises are available on the Agency’s website, we recommend that an effort should be made to combine material concerning similar issues/sectors so that best practice guidelines and examples are brought together in a single publication. These could complement the material already produced by the Agency. 17. There also needs to be more emphasis on ensuring that the results of projects are disseminated as widely as possible. As argued earlier, the SME Funding Scheme cannot be expected to have extensive direct impacts given the modest scale on which it operates. However, the ‘demonstration effects’ of the scheme and indirect impacts on SMEs can be considerable if project outcomes are disseminated as widely as possible and in a form that makes it possible to do this trans-nationally. A number of initiatives might be considered: firstly, networking between project leaders from different countries, and between FOPs, should be further developed as a way of providing the necessary support structures for more extensive dissemination activities; secondly, there is a case for additional resources being made available specifically to ensure that outcomes from a project in one country are, if necessary, modified so that they can be applied in other countries; thirdly, and as argued above, consideration might be given to producing best practice guides (organised on a thematic basis, e.g. by sector or type of health and safety at work issue) that draw on the experience gained from projects. 18. Future interventions under the SME Funding Scheme should have a narrower thematic focus with support more clearly targeted on particular types of SMEs and health and safety at work issues. As noted in the report, the 2001-02 SME Funding Scheme had a relatively narrow thematic focus but this was subsequently broadened out. Feedback from the research suggests that a narrower focus leads to the programme being easier to manage and increases the likelihood of positive impacts. It would also make the task of identifying good practices less complicated since the results of projects would be easier to compare.


Section


6

64

��

��

19. Steps should be taken to ensure that synergies with other EU funded networks and programmes are maximised so that there is a leveraging of funding and resources from other sources to support projects. In carrying out the study, we found some evidence of projects combining support form the SME Funding Scheme with financial assistance that is available from other EU programmes (for example ESF, FEOGA and INTERREG). The Structural Funds (in particular, the European Social Fund – ESF) are an obvious source of additional funding for health and safety at work projects and initiatives and project leaders should be encouraged to make full use of such resources assuming they satisfy the eligibility requirements. A related recommendation is that more use should be made of other EU-funded networks (Euro Info Centres, European BICs, Innovation Relay Centres, etc) that focus on SMEs to promote the SME Funding Scheme. 20. Irrespective of the precise arrangements for operating the SME Funding Scheme in the future (see Point 13), health and safety at work should ideally be developed into a ‘horizontal’ theme in major EU-supported programmes such as the Structural Funds. The Structural Funds currently have three ‘horizontal’ or cross-cutting themes (equal opportunities, information society, and environmental sustainability). Project managers who receive ERSF and ESF grant aid are expected to explain at the application stage how they will address these themes in their projects and implementation of the horizontal priorities is subsequently a key feature of programme monitoring and evaluation activities. Ideally, health and safety at work should be treated in a similar way in the new (post 2006) Structural Fund programming period. In practice, this may be difficult to achieve because of the competing claims of different policy lobbies. It may therefore be more realistic to focus on ensuring that health and safety at work issues are addressed by particular Priorities and Measures that relate to SMEs (this would in any case by a necessary starting point to promoting health and safety at work as a ‘horizontal’ theme). 21. To help ensure that sufficient account is taken of health and safety at work issues in developing Structural Fund Priorities and Measures relating to SMEs, the Agency should prepare guidance aimed at policymakers in regional authorities explaining what sorts of health and safety projects are eligible for support. An analysis of some current Structural Fund programme documents shows that there is no direct reference to health and safety at work issues although many of the measures could have included such a reference. Programme documents are usually drawn up by policymakers who are generalist rather then experts in any particular subject and they may well find such guidance helpful. The Agency should discuss with DG Regio the content of any proposed guidance and could


Section


6

65

��

��

well use its network of FOPs to ensure that managing authorities are aware of such guidance. 22 The Agency should investigate the possibility of similar guidance being included in other major EU funding initiatives, in particular the agricultural and fishery funds. Both these funds, which account for substantial Community expenditure, provide funding to industries which are dominated by SMEs and which have serious health and safety issues. There may be a possibility of leveraging further the use of Community funding to promote health and safety. 6.3 Recommendations (New Member States)

23. Whilst a transfer of know-how from the Agency’s EU15 SME Funding Scheme to the New Member States (NMSs) is desirable, this should be a two-way process. Although the NMSs have not operated the same sort of scheme as in EU15, they nevertheless have valuable experience of tackling health and safety at work issues.

24. Following EU enlargement, there is a strong case for a special SME Funding Scheme for the New Member States. Whilst such a programme could operate in the same way as the scheme has in EU15, i.e. providing support for new initiatives to develop better health and safety at work practices in SMEs, an alternative approach would be to devote the available resources exclusively to initiatives aimed at transferring ‘good practices’ that have already been developed under previous annual programmes to the New Member States. We would recommend that this alternative approach is adopted. Suggestions on how this approach might be taken forward are provided below.

25. An important early task should be to establish health and safety at work baselines in the New Member States for key industrial sectors, types of risk, and with regard to the relevant institutional/policy set ups. Having a clear picture of the situation in the New Member States with regard to safety and health at work practices, policies and institutional frameworks is vital if EU-funded interventions are to be appropriately targeted. The Agency has an important role to play in providing guidance on what sort of information is needed to compile the baselines, making contact with the authorities to obtain the required inputs, and then in analysing the results and identifying priorities. This is a potentially very demanding task and is probably best approached in stages by focusing on high risk safety and health at work issues and sectors in the first instance.


Section


6

66

��

��

26. There is a need to review experience from EU15 to identify ideas and good practices in the safety and health at work field that are especially relevant to the New Member States. As noted earlier (Point 16), there is an enormous amount of information available from EU15 project outcomes that is potentially relevant to the New Member States. Apart from the guidelines and other outputs produced directly by the EU15 projects themselves, the Agency’s website also includes useful information. The problem is that neither source is ideal, as things stand, as a guide to good practice – in the first case because there is too much information and in the second case because only project summaries are available. Best practice guides are needed on specific issues that combine the results of different projects in a concise but informative way with references to more detailed information and useful contacts. Consideration also needs to be given to translation into the various national languages in the New Member States to assist dissemination. Following on from the last recommendation, we suggest that the focus should initially be on types of safe and health at work issues and sectors that are especially high risk in the New Member States.

27. Support should be provided, where necessary, to help develop safety and health at work institutional capacity and policies in the New Member States. Extension of the Agency’s activities to cover the New Member States presupposes that appropriate supporting structures are in place in the various countries. This includes not only FOPs but also the wider structures needed to promote safety and health at work across different sectors and regions in the New Member States. In addition to the basic structures, there may be a need for capacity building and this should be one of the Agency’s key priorities in developing its role in the enlarged EU.

28. We also recommend that there should be a review of EU programmes now operating in the New Member States to establish (a) whether health and safety issues are being addressed, and (b) if not, the scope under existing programmes to do so. For example, a check should be made to see if the project appraisal criteria being used in Structural Fund programmes in the New Member States include reference to health and safety at work issues. A review by us of the programme documents in one New Member State suggests that there is virtually no focus on such issues. If this is typical of all 10 countries, then a key priority for the Agency should be to provide guidance to national authorities on how project appraisal criteria and procedures can be adapted to include health and safety at work issues, as described in 21 and 22 above. This exercise could involve producing a best practice exemplar/template and then holding workshops with programme managers from the New Member States, especially those responsible for implementing SME measures, to explain how the criteria and procedures should be applied in practice


Section


6

67

��

��

29. Steps should be taken to strengthen trans-national networks of safety and health at work organisations so that they include representatives from the New Member States and can be used to help transfer know-how. In addition to the networks and other health and safety at work structures that exist at a national level, the SME Funding Scheme has helped to develop trans-national linkages. This experience is especially relevant given other recommendations concerning the transfer of good practices set out in this section. One idea, for example, might be to support a mentoring initiative under which EU15 partners who have benefited from the Agency’s support are encouraged to transfer their know-how to similar organisations in the New Member States. We found some evidence of initiatives by particular EU15 project leaders to work more closely with partners in the accession countries but to be effective, this type of activity needs to be coordinated with perhaps limited funding being made available to cover travel expenses . This is a role that the Agency, supported by National Focal Points, could undertake. 30. Many of the suggested improvements to the SME Funding Scheme that has operated in EU15 are especially relevant to the New Member States and should be implemented there if a scheme is to be launched that goes beyond transferring best practices. If support is provided directly to organisations in the New Member States in a similar way to the EU15 SME Funding Scheme, then feedback on the schemes evaluated in this report should be taken into account, especially with regard to the time allowed for completion of projects, targeting of particular health and safety at work themes/risks, emphasis on projects that add value, and importance attached to dissemination and ‘demonstration’ effects. In Table 6.1 we provide a list of the recommendations set out above and suggest how they might be prioritised. Overall, priority should in our view be given to initiatives to tackle SME-related safety and health at work issues in the New Member States (Recommendations 23 to 30). Although recommendations relating to the current EU15 SME Funding Scheme would improve it, the most pressing need is to extend the scheme to member states that are not presently covered by it.

Table 6.1: Suggested Prioritisation of Recommendations


Section


6

68

��

��

No. Recommendations - General Priority

12. In line with the evaluation’s overall conclusions regarding its performance, there should be continued support, preferably on a larger scale, for the SME Funding Scheme.

High

13. However, the Institutions should consider whether the SME Funding Scheme should continue in its current form, i.e. as a separate scheme, or become part of a larger EU-funded programme.

High

14. If the SME Funding Scheme continues in its present form, the funding arrangements should be altered to allow projects to be supported on a multi-annual basis.

Medium

15. Similarly, if the SME Funding Scheme continues in its present form, there should be a greater focus on the types of projects that deliver the highest Community added value.

High

16. There is a need for a ‘consolidation’ exercise pulling together the health and safety at work best practice lessons to be learnt from projects supported by the SME Funding Scheme.

Medium

17. There also needs to be more emphasis on ensuring that the results of projects are disseminated as widely as possible.

High

18. Future interventions under the SME Funding Scheme should have a narrower thematic focus with support more clearly targeted on particular types of SMEs and health and safety at work issues.

Medium

19. Steps should be taken to ensure that synergies with other EU funded networks and programmes are maximised so that there is a leveraging of funding and resources from other sources to support projects.

Medium

20. Irrespective of the precise arrangements for operating the SME Funding Scheme in the future (see Point 13), health and safety at work should ideally be developed into a ‘horizontal’ theme in major EU-supported programmes such as the Structural Funds.

High

21 To help ensure that sufficient account is taken of Health and Safety in developing Regional Funds Priorities and Measures relating to SMEs, the Agency should prepare guidance aimed at policymakers in regional authorities explaining what sorts of health and safety projects could be included.

High

22 The Agency should investigate the possibility of similar guidance being included in other major EU funding initiatives, in particular the agricultural and fishery funds.

High

Recommendations – New Member States

23. Following EU enlargement, there is a strong case for a special SME Funding Scheme for the New Member States

High

24. An important early task should be to establish health and safety at work baselines in the New Member States for key industrial sectors, types of risk, and with regard to

Medium


Section


6

69

��

��

the relevant institutional/policy set ups.

25. There is a need to review experience from EU15 to identify ideas and good practices in the safety and health at work field that are especially relevant to the New Member States.

High

26. Support should be provided, where necessary, to help develop safety and health at work institutional capacity and policies in the New Member States.

High

27. We recommend that there should be a review of EU programmes now operating in the New Member States to establish (a) whether health and safety issues are being addressed, and (b) if not, the scope under existing programmes to do so.

High

28 Steps should be taken to strengthen transnational networks of safety and health at work organisations so that they include representatives from the New Member States and can be used to help transfer know-how.

High

29. Many of the suggested improvements to the SME Funding Scheme that has operated in EU15 (see Section 6.2) are especially relevant to the New Member States and should be implemented there if a scheme is to be launched that goes beyond transferring best practices.

Medium


Appendix

Review Of Recommendations From 2003 Evaluation

A

70

��

��

The interim evaluation of the 2001-02 SME funding Scheme, published by the Agency in 2003, contained a number of ‘priority’ and ‘specific’ recommendations. It is clearly important to review these recommendations in light of further experience and research to ascertain whether (a) they are still relevant; and (b), if so, whether the recommendations have been implemented and had the desired effects. Below, we provide our assessment.

‘Priority’ Recommendations (2001-02 Interim Evaluation)

Recommendations Comment 1. To restructure the SME Accident Prevention Funding Scheme into a Pluri-annual Programme in order to avoid time constraints.

The current research confirms that this remains a priority.

2. To heighten the visibility of the SME Accident Prevention Funding Scheme in relation to other Community Programmes.

No longer relevant but the general point with regard to raising awareness applies although there is a danger of demand for support outstripping available resources.

3. To establish a co-ordination mechanism to assure the complementarity between different Community institutions in occupational safety and health matters, thereby avoiding overlaps and activities with ones having similar contents.

This remains a priority. Perhaps more important is to achieve a mainstreaming of health and safety in other EU programmes.

4. To assure transparency and impartiality in the evaluation process. Subcontracting the evaluation of applications to external specialists will prevent problems arising in the evaluation procedure.

Not accepted and not implemented

5. Should the Funding Scheme continue to operate on an annual basis, it would be advisable to slim down the entire procedure. The Application Form should be simplified as much as possible. This would reduce the work involved for potential project applicants as well as for those taking part in the task of project evaluation.

Our assessment confirms that further steps could be taken to simplify the Application Form.

6. The Funding Scheme should facilitate the participation of Eastern European countries, conceding greater preference to those projects including partners from Eastern countries. This will contribute towards reducing the differences in Occupational Safety and Health concerns between EU and Eastern countries.

This remains a priority although we argue in the report that emphasis should be on ‘mainstreaming’ SME health and safety as a theme in other EU programmes.

7. To dedicate more staff to management the SME Accident Prevention Funding Scheme. The Campaigns and Programmes Task Force managed to put this Scheme into operation with a team made up, basically, of only four people. The team worked in a high workload environment under a great deal of

The research confirms that the SME Funding Scheme is efficiently managed. However, if the scope of the programme is extended to new EU Member States, there will need to be an increased level


Appendix


A

71

��

��

pressure in order to comply with the Evaluation Scheme’s objectives and tight deadlines. In view of the Scheme’s timetable, allocating additional staff to handle the Funding Scheme is recommended.

of management resources.

8. Documents disseminated across Europe should be published in the highest number of languages possible. In order to do this, the possibility of collaborating with National Focal Points in the translation of these documents should be considered.

This remains a priority and we recommend in our report that translation costs should be included in the project budget.

9 .Barely six months have elapsed since completion of the Funding Scheme projects, something that has made assessing the impact of these projects a difficult task. Simple monitoring of projects from the 2001-2002 Funding Scheme is recommended.

Our research suggests that the SME monitoring programme is being effectively monitored and evaluated. We see no requirement for additional initiatives in this respect.

‘Specific’ Recommendations (2001-02 Interim Evaluation)

Recommendations Comment 1. The need for more time to complete the projects. Project holders had difficulties implementing their projects within the deadlines, which reduced project effectiveness.

The current research confirms this but the constraints on timescales arising from the nature of the EU funding remain.

2. To stress the sustainability of the projects, thereby avoiding confusion between sustainability and dissemination of project results.

There is a high level of awareness of the need for project sustainability and our research suggests this is being achieved.

3. To improve the transferability of the results of projects. A start-up, initial meeting with project holders might well be a first step towards reinforcing the transfer of results, together with the need to edit a publication contemplating the results of the 2001-2002 Funding Scheme in the highest number of languages possible.

This remains a high priority although our evaluation suggests that additional steps need to be taken to ensure transferability beyond those mentioned in the earlier evaluation.

4. To encourage the participation of SMEs, associations and consortiums in order to increase the number of SME project holders.

The research suggests that there has been more emphasis on this type of target group.

5. The Agency should organise an initial, “kick-off” meeting with project holders in order to clarify evaluation scheme procedures and to allow project holders to establish contact with both the Agency and other project holders.

We do not see this as a priority, except perhaps in the case of the largest projects. Communication by e-mail and telephone with the Agency is good and meetings would add unnecessary costs.

6. The need to support potential applicants in creating partnerships. The Agency should create and disseminate a database containing details of those applicants who

Whist the need for closer networking remains, this ‘facilitation’ function could be more effectively achieved by FOPs.


Appendix


A

72

��

��

submitted applications to the Funding Scheme. 7. To improve the definition of each project category. The Funding Scheme co-funded three different types of activities: Training, Information and Communication Activities and the Provision of Good Practices. These categories caused a certain degree of confusion among project holders, some of whom did not really know in which category they should participate.

In practice, most projects we have examined combine these activities and we would not support a distinction of this type between different projects.

8. Much stricter eligibility criteria should be applied in order to ensure the technical and administrative quality of the projects submitted. This would stop the Agency’s receiving applications having little likelihood of being accepted.

The research feedback suggests that the initial screening undertaken by FOPs now achieves this.

9. To improve the clarity of the Call for Project Proposals and the Application Form. To include reference indicators in the Application Form in order to allow for projects to be monitored and evaluated.

The survey feedback suggests that there is now sufficient clarity. The need to improve performance indicators remains.

10. To extend the Call for Project Proposals publicity campaigns in order to improve the diffusion of the message.

Given the limited scale of the programme, the level of publicity given to the Call for Proposals seems appropriate to us.

11. To establish a minimum number of promotion activities by all National Focal Points in order to counter-balance the dissemination and publicity of the Call for Project Proposals.

Developing the ‘promotional’ role of FOPs remains a priority although the limited resources available needs to be taken into account.

12. 12. To use the Agency’s website as a means for channelling all the information produced as a result of the Funding Scheme. The SME Accident Prevention Funding Scheme should have a heightened, more visible presence on the Agency's website.

This recommendation has been fully implemented and the website now provides a good source of information on project outcomes.


Appendix

SURVEY OF 2001-02 SCHEME

B

73

��

��

This appendix contains the questionnaire used to survey the 2000-02 scheme

Questionnaire for Project Holders

The European Agency for Safety and Health at Work (the Agency) has commissioned the Centre for Strategy & Evaluation Studies (CSES) to undertake an external evaluation of the SME funding schemes. Your project was supported under the 2001-02 scheme, and you have already kindly provided information which helped in an earlier evaluation. The following short questionnaire is to ask about the current position of the project and whether it is still active, and the results it has achieved. We would be most grateful if you could complete this questionnaire and return it by email to ([email protected]) or ([email protected]) or by fax to +441932 855638 or by post to CSES, PO Box 159, Sevenoaks TN14 5WT, United Kingdom by …………………………. If you have any queries please contact Sonia Cook on +44 (0)1227 763180. 1. Basic Information 1.1 Project Holder 1.2 Country 1.3 Contact Name 1.4 Telephone number 1.5 E-mail 1.6 Project Title 1.7 Project Number 1.8 Number of Partners 2. Current situation with the project 2.1 Please tell us the extent to which the project partners still work together: (Please tick one box for each) Regularly Sometimes Never

a) On the subject matter of this project b) On other OSH projects c) On other matters

Comments: 2.2 To what extent do you think your organisation has benefited from the project? (Please tick one box)


Appendix


B

74

��

��

A lot Some extent Not at all 2.3 To what extent are SMEs still benefiting from the results of the project? (Please tick one box) A lot Some extent Not at all 2.4 How long will SMEs continue to benefit from the project? (Please tick one box)

Up to a year More than a year

2.5 Have you sought further funding to enable project activities to continue after the Agency’s support ended? Own resources Yes….. No Public funding Yes….. No Private sector sources Yes….. No

Other (please specify)

3. Effectiveness How many SMEs have benefited from the project? If possible, please estimate the total number of SMEs which have benefited from the project through all the project partners. 3.1 Please estimate the number of SMEs that have benefited from the Agency project (Please supply numbers)

Received specific advice Received written

information

Viewed website Other (please specify below)

Please explain any entries in the ‘other’ box: Please indicate if the above estimate includes SMEs benefiting from contacts through all partners in the project, or just through your own organisation

Estimate includes SMEs benefiting Estimate includes just SMEs


Appendix


B

75

��

��

through all partners in the project benefiting through our own organisation

3.2 How does the project seek to measure its effectiveness? Please explain: 3.3 In your view, what sort of factors are critical in ensuring successful project outcomes? Please rank the following possibilities from 1=critical to 4 = not important. 1. Having clear project aims 4. Achieving good publicity 2. Support of partners 5. Networking/sharing good practice 3. Funding arrangements 6. Project management Other factors – please explain below

4. Promotion 4.1 How would you rate the Agency’s help in publicising this project? (Please tick one box)

Excellent Good Average Poor Very poor

5. Evaluation

5.1 In order of importance, what do you consider the three most important achievements realised by your project? (Please list briefly)

1


Appendix


B

76

��

��

2

3

5.2 Has a follow-up survey been carried out to assess the effect of the project on SMEs? (Please tick one box) (If yes, we would be grateful if you could send a copy to CSES)

Yes, survey done Planned but not yet

completed Not planned

6. It may be appropriate for CSES to contact you again in connection with this evaluation. Please indicate if you are willing for us to do this.

Willing to be contacted Not willing to be contacted

Thanking you for completing this questionnaire.

Please return it by email to ([email protected]) or ([email protected]) or by fax to +441932 855638 or by post to CSES, PO Box 159, Sevenoaks TN14 5WT, United Kingdom by ………………………….


Appendix

SURVEY OF 2002 -03 SCHEMES

C

77

��

��

This appendix shows the questionnaire used to survey the 2002-03 scheme

Questionnaire for Project Holders

The European Agency for Safety and Health at Work (the Agency) has commissioned the Centre for Strategy & Evaluation Studies (CSES) to undertake an external evaluation of the SME funding schemes. Your project was funded under the 2002-03 programme. Now that the project is completed, we would like to find out your views on various issues relating to the project. We would be most grateful if you could complete this questionnaire and return it by email to ([email protected]) or ([email protected]) or by fax to +441932 855638 or by post to CSES, PO Box 159, Sevenoaks TN14 5WT, United Kingdom by …………………………. If you have any queries please contact Sonia Cook on +44 (0)1227 763180. 1. Project Information 1.1 Project Holder 1.2 Country 1.3 Contact Name 1.4 Telephone number 1.5 E-mail 1.6 Project Title 1.7 Project Number 1.8 Number of Partners 2. Application Procedures 2.1 How did you hear about the Funding Scheme?

a) Official Journal c) National Focal Point b) Website d) Trade Association e) Other Please explain below:

2.2 Were the following formulated in a clear and understandable way, providing sufficient information for

potential applications? Yes No


Appendix


C

78

��

��

a) Call for project proposal

b) Application form/criteria

c) Technical specifications

d) Reporting requirements

2.3 Were the deadlines appropriate for the following? (If No, please explain) Yes No a) Presenting the application…………………… b) Achieving the project objectives, products and targets c) Submitting the final report…………………………

2.4 Would the project have gone ahead without Agency funding? a) Yes, the project would have gone ahead with other funding b) Yes, but on a reduced scale c) Yes, but at a later date d) No, the project would not have gone ahead e) Don’t know 2.5 Please let us have any comments on how the application criteria/process could be improved

3. During the Project 3.1 To what extent have the workforce, unions, and employers been involved in the following? Workforce Unions Employers

a) Participating in the planning of the project b) Participating directly in implementation of the project c) Receiving training d) Receiving some information e) Participating in seminars f) Other (Please specify)


Appendix


C

79

��

��

3.2 How would you describe the help and advice received from the Agency and the National Focal Point during the

project implementation period in relation to the following aspects? (Please rate: 1:Very poor, 2: Poor, 3: Fair, 4: Good, 5: Excellent)

Type of assistance Agency (1-5)

National Focal Points (1-5)

a) Answering questions b) Giving additional information c) Monitoring your project d) Other (Please specify) 3.3 If your project was carried out with partner organisations, how would you evaluate their performance and

added value? 1. Excellent 2. Good 3. Fair 4. Poor 5. Very poor 3.4 Please let us have any comments on how the management of the project by the Agency could be improved.

4. Results of the Project 4.1 To what extent have the objectives of your project been reached? (If not achieved, please explain why) Fully achieved Partially achieved Not achieved at all

We would now like to ask about the number of SMEs which have benefited from the project. If possible, please estimate the total number of SMEs which have benefited from the project through all the project partners 4.2 Please estimate the number of SMEs that have benefited from the Agency project (Please provide numbers):


Appendix


C

80

��

��

a) Received specific advice c) Received written information

b) Viewed website d) Other (please specify below) Please explain any entries in the ‘other’ box: 4.3 Please indicate if the above estimate includes SMEs benefiting from contacts through all partners in the project, or just through your own organisation

Estimate includes SMEs benefiting through all partners in the project

Estimate includes just SMEs benefiting through our own organisation

4.4 In your opinion, how effectively is the Funding Scheme contributing to improving health and safety in SMEs? Very effective Quite effective Quite ineffective Not effective at all

4.5 Could your project outcomes be easily transferred to other SMEs or sectors? Yes No

If Yes, please explain which outcomes and to what extent other SMEs or sectors could benefit 4.6 To what extent do you think your own organisation has benefited from the project? (Please tick one box)

A lot To some extent Not at all 4.7 To what extent are SMEs still benefiting from the results of the project? (Please tick one box) A lot To some extent Not at all

4.8 How long will SMEs continue to benefit from the project? (Please tick one box)

Up to a year More than a year 4.9 In your view, what sort of factors are critical in ensuring successful project outcomes? Please rank the

following possibilities from 1=critical to 4 = not important.


Appendix


C

81

��

��

1. Having clear project aims 4. Achieving good publicity 2. Support of partners 5. Networking/sharing good practice 3. Funding arrangements 6. Project management Other factors – please explain below 5. Publicity 5.1 What type of information has been produced by the project? (Please tick all the apply)

a) Web site d) Other (please explain):

b) Brochure c) Guidelines

5.2 What actions, if any, are you taking to publicise the project and the outcomes being achieved? 5.3 How would you rate the Agency’s help in promoting your project activities? (Please tick one box)

1. Excellent 2. Good 3. Fair 4. Poor 5. Very poor 6. Evaluation 6.1 What measures, if any, are being taken to evaluate the effectiveness of the project?


Appendix


C

82

��

��

6.2

Has a follow-up survey been carried out to assess the effect of the project on SMEs? (Please tick one box) (If yes, we would be grateful if you could send a copy to CSES)

Yes, survey done Planned but not yet

completed Not planned

5.5 It may be appropriate for CSES to contact you in connection with this evaluation. Please indicate if you are willing for us to do this. Willing to be contacted Not willing to be contacted

Thank you for completing this questionnaire

Please return it by email to ([email protected]) or ([email protected]) or by fax to +441932 855638 or by post to CSES, PO Box 159, Sevenoaks TN14 5WT, United Kingdom by ………………………….


Appendix

SURVEY OF REJECTED PROJECTS

D

83

��

��

This appendix shows the questionnaire used to survey schemes which were rejected

Questionnaire for projects that were not selected

The European Agency for Safety and Health at Work (the Agency) has commissioned the Centre for Strategy & Evaluation Studies (CSES) to undertake an external evaluation of the SME funding schemes. We realise that your project did not receive support from the Agency. Nevertheless, we do hope that you will be able to provide information on issues such as whether the scheme went ahead in whole or in part, and your experience in dealing with the application procedures. We would be most grateful if you could complete this questionnaire and return it by email to ([email protected]) or by fax to +44 1932 855638 or by post to CSES, PO Box 159, Sevenoaks TN14 5WT, United Kingdom by …………………………. If you have any queries please contact Sonia Cook on +44 (0)1227 763180.

1. Basic Information 1.1 Applicant organisation 1.2 Country 1.3 Contact Name 1.4 E-mail 1.5 Project Title 1.6 Application number 2. What happened to the project? a) Went ahead in full (Please go to Question 3.)

b) Went ahead in part (Please go to Questions 4)

c) Did not go ahead (Please go to Question 5) 3. If the project went ahead in full, what sources of funding were used? a) Own resources b) National public funding

c) Other (Please specify)

4. If the project went ahead but only in part, are you still seeking further funding? Yes No 5. If the project did not go ahead:


Appendix

SURVEY OF REJECTED PROJECTS

D

84

��

��

a) Is the project team still working together? b) Is the project still seeking funding? Yes No Yes No

c) Are there any other reasons for the project not going ahead? (If so, please explain)

6. Please let us have any comments on the way your application was handled by the Agency:

Thank you for completing this questionnaire. Please return it by email to ([email protected]) or by fax to +441932 855638 or by post to CSES, PO Box 159, Sevenoaks TN14 5WT, United Kingdom by ………………………….


Appendix

INTERVIEW CHECKLIST

E

85

��

��

There is a requirement for an evaluation of Evaluation of SME funding schemes

This evaluation is to be carried out for

As part of the evaluation being undertaken by CSES, interviews are being carried out with Commission officials. This checklist provides a guide to key issues to be discussed in the interviews, but is of a general nature only. Different officials will play different roles – for example policy setting, administration, financial control, etc, and each interview will concentrate on the roles played by the relevant official.

Your Role

We begin by asking a few questions about your role. Please briefly describe your role

1.1 Please describe the work you carry out in and the main focus of this work

1.2 Do your own objectives specifically include work on OSH issues in SMEs

Activities and Progress towards Objectives

A key aim of the evaluation is to establish the extent to which the SME funding schemes are achieving their objectives. Part 2 of the checklist asks for your view on the results achieved so far by the SME funding schemes

2.1 What is your understanding of the SME funding scheme’s objectives?

2.2 How, in your view, should its success be judged?

2.3 What is your view on the arrangements coordinating and monitoring the SME funding schemes? How can procedures be improved?

2.4 How effectively does the SME funding schemes use resources? How can this be measured?

2.5 Overall, to what extent are the results that have been achieved to date by the SME funding schemes? Please comment specifically on:

- the setting up of the SME funding schemes

- the SME funding scheme’s objectives

- Other outputs achieved to date by the SME funding schemes

3. Relationship of SME funding schemes to other programmes


Appendix

INTERVIEW CHECKLIST

E

86

��

��

We now ask for comments on the relationship of the SME funding schemes to other Commission programmes dealing with related issues.

3.1 Are there other Commission programmes with similar objectives to the SME funding schemes?

3.2 Are there other National programmes with similar objectives to the SME funding schemes?

3.3 How should the SME funding schemes relate to work carried out at National level

4. Lessons from Experience so Far and Future Priorities

In addition to evaluating the SME funding scheme’s past and current activities, an important aim of the study is to learn from experience and help to define future priorities.

4.1 In your view, what are the main lessons to be learnt from experience so far, i.e. what is the SME funding schemes doing well/less well and why? 4.2 What should the SME funding scheme’s (or a successor programme’s) priorities be for coming years 4.3 What is your view on the SME funding scheme’s future resource requirements? 4.4 What is the scope for improving current operational processes to use resources more efficiently?


Appendix

INTERVIEW LIST

F

87

��

��

This appendix lists the external interviews carried out during the evaluation. In addition, interviews were carried out with the Director and staff of the Agency

External interviews Paul Glynn DG Empsoc Ann Elisabet Jensen MEP Stephen Hughes14 MEP Marc Sapir ETUC/TUTB Janine Delahaut ETUC/TUTB Liliane Volozinskis UEAPME Bertil Remaeus Agency Board Luis F do Nascimento Lopos UGT Kai Stryczynski DG Regio Jose Ramon Biosca de Sagastuy DG Empsoc

Interviews with National Focal Points Ireland Health and Safety Authority France Ministère de l'Emploi et de la Solidarité Germany Bundesministerium für Wirtschaft und Arbeit Denmark Arbejdstilsynet Spain Instituto Nacional de Seguridad e Higiene en el Trabajo

14 Telephone interview with PA