Nuove frontiere e prospettive di finanziamento per la ricerca europea nel periodo 2007-2013

13/05/2005 G. Abbate Dip. di Sc. Fisiche Università di Napoli Federico II

1

Nuove frontiere e prospettive di finanziamento per la ricerca europea nel periodo 2007-2013

Valutazione delle proposte e probabilità di successo

FP6 IST

Giancarlo Abbate


2

Sommario

La valutazione delle proposte Area Tematica del 6 PQ: Tecnologie per una società dell’informazione

Le procedure di valutazione I criteri ed i sottocriteri di valutazione per i

differenti “strumenti” di finanziamento Il processo di selezione delle proposte

Gli esperti indipendenti. Lettura individualeConsensus meetingLettura incrociata. Panel meeting. Classifica. Audizione

Commenti sul processo di selezione Paragone con le procedure di valutazione di

alcuni altri programmi.


3

Le procedure di valutazione

Procedura a due stadi: esempio FET-Open STREP

ftp://ftp.cordis.lu/pub/ist/docs/fet/int-o-1.pdf


4




5




6




7

Procedura a singolo stadio: Esempio FP6-IST-2002-2.3.2.2 Optical, opto-electronic, & photonic functional components

http://www.europa.eu.int/comm/dgs/information_society/evaluation/pdf/5_y_a/ist_5ya_final_140105.pdf

The scientific and technical evaluation of proposals is carried out by external experts from the public and private sector. Full proposals are evaluated, ranked and selected. The IST theme under FP5 and FP6 generally has a high oversubscription of the budget – e.g. the first call for the IST Priority Theme in FP6 attracted 1400 proposals requesting € 6.2 B, whereas the available budget was only € 1.07 B.


8

I criteri ed i sottocriteri di valutazione per i differenti “strumenti” di finanziamento

Common evaluation criteria for evaluating proposalsA number of evaluation criteria are common to all the programmes of the Sixth Framework Programme and are set out in the European Parliament and the Council Regulations on the Rules for Participation (Article 10). These are:

a) Scientific and technological excellence and the degree of innovation;

b) Ability to carry out the indirect action successfully and to ensure its efficient management, assessed in terms of resources and competences and including the organisational modalities foreseen by the participants;

c) Relevance to the objectives of the specific programme;d) European added value, critical mass of resources

mobilised and contribution to Community policies;e) Quality of the plan for using and disseminating the

knowledge, potential for promoting innovation, and clear plans for the management of intellectual property.


9


Furthermore, in applying paragraph (d) above, the following criteria are also to be taken into account:

a) For networks of excellence, the scope and degree of the effort to achieve integration and the network’s capacity to promote excellence beyond its membership, as well as the prospects of the durable integration of their research capabilities and resources after the end of the period covered by the Community’s financial contribution;

b) For integrated projects, the scale of the ambition of the objectives and the capacity of the resources to make a significant contribution to reinforcing competitiveness or solving societal problems;

c) For integrated initiatives relating to infrastructure, the prospects of the initiative’s continuing long term after the end of the period covered by the Community’s financial contribution.


10


Horizontal issues Are there gender issues associated with the subject of the

proposal? If so, have they been adequately taken into account?

Have the applicants identified the potential ethical and/or safety aspects of the proposed research regarding its objectives, the methodology and the possible implications of the results? If so, have they been adequately taken into account in the preparation of the proposal?

To what extent does the proposal demonstrate a readiness to engage with actors beyond the research community and the public as a whole, to help spread awareness and knowledge and to explore the wider societal implications of the proposed work?

Have the synergies with education at all levels been clearly set out?

If third country participation is envisaged in the proposal, is it well justified and the participation well integrated in the activities?


11


1. Relevance (threshold score 3 out of 5) The extent to which the proposed project addresses the objectives

of the work programme. 2. Potential impact (threshold score 3 out of 5)The extent to which:

the proposed project is suitably ambitious in terms of its strategic impact on reinforcing competitiveness (including that of SMEs) or on solving societal problems.

the innovation-related activities and exploitation and/or dissemination plans are adequate to ensure optimal use of the project results.

the proposal demonstrates a clear added value in carrying out the work at European level and takes account of research activities at national level and under European initiatives (e.g. Eureka).

3. S&T excellence (threshold score 4 out of 5)The extent to which:

the project has clearly defined objectives. the objectives represent clear progress beyond the current

state-of-the-art. •the proposed S&T approach is likely to enable the project to

achieve its objectives in research and innovation.


12


4. Quality of the consortium (threshold score 3 out of 5)The extent to which:

the participants collectively constitute a consortium of high quality. the participants are well-suited and committed to the tasks assigned to

them. there is good complementarity between participants. the profiles of the participants, including those to be included later, have been

clearly described. the real involvement of SMEs has been adequately addressed.

5. Quality of the management (threshold score 3 out of 5)The extent to which:

the organisational structure is well matched to the complexity of the project and to the degree of integration required.

the project management is demonstrably of high quality. there is a satisfactory plan for the management of knowledge, of intellectual

property and of other innovation-related activities. 6. Mobilisation of resources (threshold score 3 out of 5)The extent to which:

•the project mobilises the minimum critical mass of resources (personnel, equipment, finance…) necessary for success.

the resources are convincingly integrated to form a coherent project. the overall financial plan for the project is adequate.

Overall threshold score 24 out of 30.


13


1. Relevance (threshold score 3 out of 5) The extent to which the proposed project addresses the objectives of the work

programme. 2. Potential impact (threshold score 3 out of 5)The extent to which:

Europe has a strategic need to strengthen S&T excellence on the topic by means of a restructuring of the existing research capacities and the way research is carried out.

the goals of the network are, in that connection, suitably ambitious particularly, in terms of achieving European leadership and acting as a world force on this topic.


there is an effective plan for spreading excellence, exploiting results and disseminating knowledge, including to SMEs and to those outside the network.

the proposed approach is likely to have a durable structuring impact on European research.

3. Excellence of the participants (threshold score 3 out of 5)The extent to which:

the participants are currently conducting excellent research relevant to the topic of the network or are capable of important contributions to the joint programme of activities.

the participants are well suited to the tasks assigned to them. they collectively have the necessary critical mass of expertise and resources to

carry out the joint programme of activities successfully.


14


4. Degree of integration and the joint programme of activities (threshold score 4 out of 5)

The extent to which: the expected degree of integration justifies supporting the proposal as a

network of excellence. the joint programme of activities is sufficiently well designed to

achieve the expected degree of integration. the participating organisations have made a convincing commitment

towards a deep and durable integration continuing beyond the period of Community support.

5. Organisation and management (threshold score 3 out of 5)The extent to which:

the organisational structure of the network provides a secure framework for any necessary structural decisions to be taken

the management of the network is demonstrably of high quality. there is a well-considered plan for promoting gender equality in the

network.



15


1. Relevance (threshold score 3 out of 5)The extent to which the proposed project addresses the

objectives of the work programme. 2. S&T excellence (threshold score 4 out of 5)The extent to which:

the project has clearly defined and well focused objectives. the objectives represent clear progress beyond the current state-

of-the-art. the proposed S&T approach is likely to enable the project to

achieve its objectives in research and innovation 3. Potential impact (threshold score 3 out of 5)The extent to which:

the proposed project is likely to have an impact on reinforcing competitiveness or on solving societal problems.


exploitation and/or dissemination plans are adequate to ensure optimal use of the project results.


16


4. Quality of the consortium (threshold score 3 out of 5)The extent to which:

the participants collectively constitute a consortium of high quality.

the participants are well-suited and committed to the tasks assigned to them.

there is good complementarity between participants. the opportunity of involving SMEs has been adequately addressed.

5. Quality of the management (threshold score 3 out of 5)The extent to which:

the project management is demonstrably of high quality. there is a satisfactory plan for the management of knowledge, of

intellectual property and of other innovation-related activities. 6. Mobilisation of resources (threshold score 3 out of 5)The extent to which:

the project foresees the resources (personnel, equipment, financial…) necessary for success.

the resources are convincingly integrated to form a coherent project.

the overall financial plan for the project is adequate.



17

Il processo di selezione delle proposte

Quality. Projects selected for funding must demonstrate a high scientific, technical and managerial quality in the context of the objectives of the RTD programme in question.

Transparency. In order to provide a clear framework for researchers preparing proposals for funding and for evaluators evaluating proposals the process of reaching those funding decisions must be clearly described and available to any interested party. In addition, adequate feedback must be provided to proposers on the outcome of the evaluation of their proposals.

Equality of treatment. A fundamental principle of EU RTD support is that all proposals should be treated alike, irrespective of where they originate or the identity of the proposers.


18


Impartiality. All proposals are treated impartially on their merits.

Efficiency and speed. The procedures have been designed to be as rapid as possible, commensurate with maintaining the quality of the evaluation, appropriate use of public money and respecting the legal framework within which the specific programme is managed.

Ethical considerations. Any proposal which contravenes fundamental ethical principles may be excluded from being evaluated or selected at any time.


19


Appointment of independent experts Calls for applications from individuals published in the Official

Journal of the European Communities; or Calls addressed to research institutions with a view to

establishing lists of suitable candidates. The lists of individuals from which panels of experts

may be chosen, are drawn up by the Commission using the following selection criteria: an appropriate range of competencies; an appropriate balance between academic and industrial

expertise and users; a reasonable gender balance; a reasonable distribution of geographical origins of

independent experts; regular rotation of independent experts.

I valutatori sono retribuiti dalla CE !


20


Impegni sottoscritti dai valutatori Conflict of interest Confidentiality

(Independent observers)Eventualmente ammessi al processo di selezione


21


Step 1: Briefing of the independent experts Step 2: Individual evaluation of proposalsEach proposal is evaluated against the applicable criteria

independently by several experts who fill in individual evaluation forms giving marks and providing comments.

Proposal marking0 - the proposal fails to address the issue under examination or

can not be judged against the criterion due to missing or incomplete information

1 - poor2 - fair3 - good4 - very good5 – excellentWhere appropriate, half marks may be given.The outcome of this step is the Individual Assessment

Report


22


Step 3: Consensus meetingFor each proposal a consensus report is prepared. The

report faithfully reflects the views of the independent experts referred to in Step 2.

Nella (quasi) totalità dei casi il Consensus meeting termina con il consenso unanime dei valutatori. Il meeting è moderato da uno “scientific officer” della CE.

The experts attempt to agree on a consensus mark for each of the blocks of criteria. They justify their marks with comments suitable for feedback to the proposal coordinator and agree on an overall consensus report, which is signed by them.

The outcome of the consensus step is the consensus report


23


Step 4: Panel evaluationA panel discussion may be convened, if necessary, to

examine and compare the consensus reports and marks in a given area, to review the proposals with respect to each other and, in specific cases (e.g. equal scores) to make recommendations on a priority order and/or on possible clustering or combination of proposals. The panel discussion may include hearings with the proposers.

A Commission official acts as moderator of the Panel Cross readings Comparison of consensus reports grouping of proposalsThe outcome of the Panel meeting is An evaluation summary report for each proposal;A list of proposals passing thresholds, if any, along with a final

mark for each proposal passing the thresholds and the panel recommendations for priority order.


24


Hearings (Audizione)Per i nuovi strumenti del 6° PQ è prevista un’audizione

individuale per ogni proposta che ha superato la soglia. Il panel di cui al punto precedente si riunisce (eventualmente in una composizione ridotta) ancora una volta (dopo un tempo opportuno, cioè circa un mese) e, al termine delle audizioni, propone la graduatoria di merito finale.

Final stepsReporting on the evaluation processFeedback to the proposersNegotiationSelection of proposalsThese final steps do not involve the independent experts


25

Commenti sul processo di selezione

Il mio personale commento è che il processo di selezione raggiunge tutti gli scopi che si prefigge ed è il più efficiente ed obiettivo tra i vari processi a cui ho partecipato come soggetto attivo e/o passivo.

Alcuni feedback da valutatori IST:It was fascinating to see how such a complex problem, namely

how to rank such a large number of disparate proposals, could be solved, and it was my pleasure to participate in the process.

I'm glad that finally we made it! It was hard working, as you say, but I really enjoyed it: I enjoyed working with clever and skilled people as evaluators and very professionals people in the EU commission. I enjoyed the international involvement and the possibility of knowing interesting people in the field from all over Europe. I enjoyed the possibility of having an overview of top level research people is now working on.


26

Commenti sul processo di selezione

I learned a lot of very important things about science, present and future trends, and mainly how a good proposal should be prepared and how it should not

It was great experience to take part in EC Project evaluations for me. I have learned how to work hard, precisely and effectively and without any external pressures

I wish to say that I was very glad to participate in evaluations of IST proposals. Not only because of the scientific topic which is extremely interesting for me but also because of perfect organisation, competence of your team and friendly atmosphere during evaluation. Thank you very much

I have been already in Growth panels in the past and had the opportunity to realize that the commission do everything is possible to be fair and objective when evaluating proposals and distributing public money. I do believe that IST staff is the paradigm in that manner


27

Paragone con le procedure di valutazione di alcuni altri programmi

INTAS The International Association for the Promotion of Co-operation

with Scientists from the New Independent States (NIS) of the Former Soviet Union

COSTFounded in 1971, COST is an intergovernmental framework for

European CO-operation in the field of Scientific and Technical Research, allowing the co-ordination of nationally funded research on a European level. COST Actions cover basic and pre-competitive research as well as activities of public utility

Running Actions ~ 200 Total Budget (2005) € 13,500,000

MIUR - PRIN


28

Documents

Nuove frontiere e prospettive di finanziamento per la ricerca europea nel periodo 2007-2013