Bergen Conference Consensus Document

 

ORPHEUS2012, Bergen, 19-21 April 2012

Introduction

Conference affirmed that the ORPHEUS/AMSE/WFME PhD standards document was a practical tool for quality assurance of PhD programmes.

ORPHEUS warmly endorses the new PhD principles document: "Best practice based principles for innovative doctoral training", approved by the EU Council of Ministers November 2011. The PhD standards document is fully compatible with the PhD principles document, and ORPHEUS has expressed a wish to work closely with the European Commission on implementing this in the field of biomedicine and health sciences. ORPHEUS also looks forward to supporting the work of the EUA-CDE and the ESF in this field.

It was recognized that the PhD standards document could provide a basis for global conversations concerning the quality and content of PhD programmes and ORPHEUS will work closely with WFME on this.

To increase awareness of the PhD standards document, national workshops should be held, and all ORPHEUS members are encouraged to initiate these. For practical quality assurance, ORPHEUS members should consider establishing benchmarking exercises nationally and internationally.

ORPHEUS encourages that awards of PhD financing by funding bodies should be based on in-depth assessment of the quality of PhD training in addition to assessment of scientific quality.

ORPHEUS will introduce a labelling project based on initial self-evaluation that can be verified through information given via the institution website. This may be followed up by some form of site visit from academics with experience in PhD education.

Internal evaluation must be an essential part of an institution's quality assurance system for PhD education. The evaluation criteria have to be transparent and should be defined at the institutional level.

The PhD students should be represented in all committees and decision-making organs. To promote active involvement of students, their efforts must be acknowledged, the infrastructure provided and their influence reflected in decision-making. PhD students should have the opportunity to evaluate supervision and supervisors; this point should be added in any revision of the PhD standards document. Results of such evaluations should be communicated to the students.

Workshop reports

Participants were divided into four workshop groups

•·         Workshop 1: Internal evaluation of PhD programmes

•·         Workshop 2: External evaluation of PhD programmes according to Orpheus standards

•·         Workshop 3: International evaluation and ORPHEUS labelling of PhD programmes

•·         Workshop 4: The role of students in PhD training evaluation

The term ‘PhD programme' in the following text encompasses both the concept of an organized programme involving several individuals, as well as an individual PhD programme for one person. 

Workshop 1: Internal evaluation of PhD programmes

In the ORPHEUS/AMSE/WFME PhD standards document, Chapter 8, Structure, it is stated: "There should be approved procedures for regular review and updating of the structure, function and quality of PhD programmes". It was agreed that this should include both structural and ad-hoc feedback. Internal evaluation must be an essential part of the whole quality assurance system and regular analysis of internal evaluations is the optimal way to achieve quality improvement in PhD training. The following recommendations were made.

•·         Academic institutions should activate mechanisms to collect, analyse and respond to evaluations, as a part of quality assurance of PhD training.

•·         Each university should develop a formal framework for internal (intramural) evaluation by collecting and acting upon feedback from all involved in PhD education (students, supervisors, examiners, administrative staff, and others).

•·         Internal evaluation should include evaluation of the institution (university/faculty/programme faculty) and evaluation of performance indicators. The evaluation has to be carried out in different contexts (PhD programme / administration / supervision / training).  The evaluation criteria have to be transparent.

•·         The workshop strongly recommends the evaluation of the supervisors and other academics responsible for each student's PhD programme.

•·         Evaluation of the administration by the supervisors as well as by the PhD students (doctoral candidates) should also be performed

•·         Evaluation of all aspects of the PhD training programme (courses, time at other laboratory, etc.) has to be included in quality assurance system.

•·         Evaluation has to be carried out repeatedly over time. The workshop recommends evaluations at the time of the student's admission to the PhD programme, midway, and at the end of the programme.

Workshop 2: External evaluation of PhD programmes according to Orpheus standards

The workshop considered different ways in which the ORPHEUS PhD standards could be used for performing external evaluation of PhD programmes. There were several views.

1.      External evaluation is necessary and should be provided by an external body of auditors. It is, however, difficult to come up with a good practical means of implementing this. Neither the Commission nor the EUA can act as accrediting bodies.  Standards could serve as guidelines, not regulations. There could initially be internal evaluation by the institutions themselves, supplemented by institutional accreditation.

2.      Peer-review evaluation. Evaluation of applications by funding agencies for PhD stipends could be made not only in terms of scientific excellence (publications, received funding, etc.), but also on in-depth peer-review evaluation of PhD programmes. This approach might be of interest in assessing  Marie Curie applications.

3.      Benchmarking was seen as a practical means of external evaluation that could encourage compliance with the ORPHEUS standards at grass-roots level.

4.      No formal external evaluation needed. This was a view of industry which is concerned with outcomes. The outcomes of interest include academic excellence of the PhD thesis as well as "ability to work in a team". Thus industry places great emphasis on collaboration with universities to ensure development of competences in transferable skills. Academically trained employees who do not have practical skills are unlikely to be attractive for industry. It was recognized that the institutions following ORPHEUS PhD standards would provide the outcomes that industry needs.

Representatives for all views underlined that the standards that are established by ORPHEUS can form a basis for quality assurance.  However, they should be dynamic and the organisation should always aim for further quality improvements of the PhD programmes. University autonomy must be respected. Any evaluation of PhD programmes should be performed by academics with experience of PhD training. PhD programmes should be transparent. There should also be transparency concerning the employability of the institution's PhD graduates. Students are likely to form a driver to ensure the quality of PhD programmes.

These views were supplemented by comparison with the US system of accreditation of PhD training. This is university-driven and there is reluctance to link evaluation of PhD programmes with funding. The workshop representative from the US, Karen de Pauw, recommended that data should be collected to determine the extent to which graduate schools complied with ORPHEUS standards.

Workshop 3: International evaluation and ORPHEUS labelling of PhD programmes

International evaluation is usually part of the evaluation of research. Accordingly international evaluation of  PhD programmes is especially important for smaller scientific communities to ensure independent and competent evaluation of PhD programmes (and PhD Theses). Even for large and competitive scientific communities, international evaluation can be significant added value. Besides it can be expected that international evaluation will facilitate international collaboration of PhD programme providers.

ORPHEUS-AMSE-WFME standards, have been developed over 7 years by ORPHEUS, with participation of hundreds of health science schools, international associations such as AMSE and WFME, and positive assessment of the European Commission EU , and EUA-CDE. This PhD standards document is the only existing comprehensive document on PhD in biomedicine and health sciences and is suitable for use in international evaluation.

Participants of the workshop welcomed the idea of ORPHEUS labelling as a reward to those ORPHEUS members organizing PhD programmes in line with the PhD standards document. ORPHEUS labelling should be organised in a simple and not expensive way. Evaluation of compliance of PhD programmes with the PhD standards document should be the basis for evaluation and labelling. PhD programmes should be described on the website in English in sufficient detail to allow "virtual evaluation"of compliance of PhD programmes with the PhD standards document as the main requirements for ORPHEUS labelling. Participants had different views about the necessity of site visits. Most participants would only recommend site visits if they can be organised as short and not expensive procedures. Details of evaluation and labelling should be developed by the ORPHEUS Executive Committee.

Workshop 4: The role of students in PhD training evaluation

Participants were grouped according to nationality and student status ensuring that as many aspects as possible were discussed. Main focus of the discussions was student involvement in the topics PhD programme organisation, supervision and instruments for quality improvement.

PhD programme organisation. The following conclusions were drawn:                   

•·                    Active and formal involvement in the PhD programme organisation: The PhD students should be represented in all committees and decision-making organs. Student involvement should be hierarchically constructed with different levels of influence, ensuring that all students have a student representative for reporting issues. Student representatives bridge the students and the PhD organisation, and facilitate the communication between students and organisations.  Student representatives identify and communicate challenges and ideas to the programme organisation, and communicate the outcomes to the students. To promote active involvement of students, their efforts must be acknowledged, the infrastructure provided and their influence reflected in decision-making. 

•·                    Passive involvement: Mandatory evaluation of all courses taken, own project progress, supervision and the PhD programme should be completed every term (semester) by all PhD students. No approval of the term should be possible or accreditation of courses given before this is done.

•·                    Supervisors should be evaluated both by the PhD programme and the PhD student. The standards for supervision are stated by ORPHEUS in Chapter 5.  These standards should form a basis for objective, non-personal criteria for use by students in anonymous evaluation of supervision.  With basis in evaluation by the programme and the students, quality of supervision should be monitored continuously with regard to both individual supervisors and the general level of supervision in the programme. The results should be communicated back to the students, and should be publically available.  It was noted that several of the PhD students had no opportunity to evaluate supervision and supervisors. This aspect is not part of the ORPHEUS standards.

•·         At the beginning of the PhD, students and supervisors shall make a detailed expected progress plan.  The responsibility for supervision is shared between students asking for supervision and providing information on progress, and supervisors for monitoring progress and offering supervision where they see the necessity in following up the plan. The student is responsible for informing the faculty in case of insufficient supervision.

•·         PhD students shall have the opportunity for formal networking in settings outside the research programme. Association with a broad network increases student involvement and sense of belonging.