An evaluation of a standards based portfolio

Pharmacy Students' Perceptions of Reflective Portfolios and their Effect on Students' Deep Information-Processing Skills

Objective. To evaluate pharmacy students' perceptions of the educational value of reflective portfolio and to gain an understanding of the factors that might influence these perceptions. Methods. Bachelor of Pharmacy (BPharm) students' perceptions of using reflective portfolios were evaluated by administering the same questionnaire at the beginning of years 2, 3 and 4 of the curriculum. Statistical analysis was carried out to determine the differences among the perception scores of different academic years. Semi-structured interviews were completed with fourth-year students to further explore their experiences with the reflective portfolio. Students' deep information processing (DIP) skills were compared with those of students from another pharmacy cohort whose curriculum did not include a reflective portfolio. Results. The students' perceptions of the reflective portfolio improved significantly as they progressed from year 2 to year 4 of the curriculum. The factors that contributed to a positive experience were a clear understanding of objectives and guidelines for the reflective portfolio, useful mentor feedback, a positive learning attitude and motivation, and having a user-friendly technology platform for submission of the portfolio. The students' DIP skills after completing the reflective portfolio were higher than those of students who did not have a reflective portfolio assignment in their curriculum. Conclusion. Pharmacy students' appreciation of the educational value of a reflective portfolio increased as they progressed to their final year, and their DIP skills improved. These findings support the use of a reflective portfolio as a learning tool for BPharm students' personal and professional development.

The educational effects of portfolios on undergraduate student learning: a Best Evidence Medical Education (BEME) systematic review. BEME Guide No. 11

INTRODUCTION

In recent years, the use of portfolios as learning and assessment tools has become more widespread across the range of health professions. Whilst a growing body of literature has accompanied these trends, there is no clear collated summary of the evidence for the educational effects of the use of portfolios in undergraduate education. This systematic review is the result of our work to provide such a summary.

METHODS

We developed a protocol based on the recommendations of the Best Evidence Medical Education (BEME) collaboration. Citations retrieved by electronic searches of 10 databases were assessed against pre-defined inclusion/exclusion criteria by two independent reviewers and full texts of potentially relevant articles were obtained. Studies were identified for inclusion in the review by examination of full text articles by two independent reviewers. At all stages, discrepancies were resolved by consensus. Data relating to characteristics of the student population, intervention, outcome measures, student design and outcomes were collected using a piloted data extraction form. Each study was assessed against 11 quality indicators designed to provide information about how well it was designed and conducted; and against the Kirkpatrick hierarchy as modified for educational settings. Comparisons between different groups were carried out using the Kruskal-Wallis test (non-parametric ANOVA) or the Mann-Whitney U test as appropriate.

RESULTS

Electronic searches yielded 2,348 citations. A further 23 citations were obtained by hand searching of reference lists. About 554 full articles were retrieved and assessed against our inclusion criteria. Of the 69 studies included in our review, 18 were from medicine, 32 from nursing and 19 from other allied health professions, including dentistry, physiotherapy and radiography. In all professional groups, portfolios were used mainly in the clinical setting, completion was compulsory, reflection required and assessment (either formative, summative or a combination of both) the norm. Three studies used electronic portfolios. Whilst many studies used a combination of data collection methods, over half of all included studies used questionnaires, a third used focus group interviews and another third used direct assessment of portfolios. Most studies assessed student or tutor perceptions of the effect of the use of portfolios on their learning. Five studies used a comparative design, one of which was a randomized controlled trial. Studies were most likely to meet the quality indicators relating to appropriateness of study subjects, clarity of research question and completeness of data. However, in many studies, methods were not reported in sufficient detail to allow a judgement to be made. About 19 of the 69 included studies (27%) met seven or more quality indicators. Across all professions, such 'higher quality' studies were more likely to have been published recently. The median 'quality score' (number of indicators met) rose from two for studies published in 2000 or earlier to seven for studies published in 2005 or later. Significant differences were observed between the quality scores for studies published in or before 2000 and those published between 2001 and 2004 (p = 0.027), those published in or before 2000 and those published in 2005 or later (p = 0.002) and between all studies (p = 0.004). Similar trends were seen in all professional groups. About 59 (85%) of the included studies were assessed at level 1 of the modified Kirkpatrick hierarchy (i.e. 'participation' effects, including 'post hoc' evaluations of student perceptions of the effects of keeping a portfolio on their learning). About 9 (13%) of the studies reported direct measurement of changes in student skills or attitudes and one study reported a change in student behaviour. The main effects of portfolio use identified by the included studies were: Improvement in student knowledge and understanding (28 studies, six at Kirkpatrick level 2 or above), greater self-awareness and encouragement to reflection (44 studies, seven at Kirkpatrick level 2 or above) and the ability to learn independently (10 studies, one at Kirkpatrick level 2). The findings of higher quality studies also identified benefits in these areas. They reported improved student knowledge and understanding, particularly the ability to integrate theory with practice, although a correlation with improved scores in other assessments was not always apparent. Greater self-awareness and engagement in reflection were also noted, although some studies questioned the quality of the reflection undertaken. Higher quality studies also suggest that use of portfolios improves feedback to students and gives tutors a greater awareness of students' needs, may help students to cope with uncertain or emotionally demanding situations and prepares students for postgraduate settings in which reflective practice is required. Time commitment required to collate a portfolio was the major drawback identified. In two of the studies, this was found to detract from other clinical learning.

CONCLUSIONS

At present, the strength and extent of the evidence base for the educational effects of portfolios in the undergraduate setting is limited. However, there is evidence of an improving trend in the quality of reported studies. 'Higher quality' papers identify improvements in knowledge and understanding, increased self-awareness and engagement in reflection and improved student-tutor relationships as the main benefits of portfolio use. However, they also suggest that whilst portfolios encourage students to engage in reflection, the quality of those reflections cannot be assumed and that the time commitment required for portfolio completion may detract from other learning or deter students from engaging with the process unless required to do so by the demands of assessment. Further work is needed to strengthen the evidence base for portfolio use, particularly comparative studies which observe changes in student knowledge and abilities directly, rather than reporting on their perceptions once a portfolio has been completed.

Student perceptions of a portfolio assessment process

OBJECTIVES

The objectives of this study were to identify and analyse students' attitudes to the portfolio assessment process over time.

METHODS

A questionnaire containing statements and open questions was used to obtain feedback from students at the University of Dundee Medical School, Scotland. The responses to each statement were compared over 4 years (1999, 2000, 2002 and 2003).

RESULTS

Response rates were 83% in 1999, 70% in 2000, 89% in 2002 and 88% in 2003. A major finding is that students perceived that portfolio building heightened their understanding of the exit learning outcomes and enabled reflection on their work. Student reactions to the portfolio process were initially negative, although they appreciated that senior staff took time to become familiar with their work through reviewing their portfolios. Student attitudes became more positive over the 4 years as the process evolved. Although portfolio assessment was recognised as supporting student learning, portfolio building was perceived to interfere with clinical learning as a result of the excessive amounts of paper evidence required.

CONCLUSIONS

Paperwork should be kept within manageable limits. A student induction process that highlights the importance of providing evidence for achieving all learning outcomes, not just theoretical knowledge and skills, may be helpful in allaying student concern over portfolio building and assessment and support preparation for lifelong learning and reflective clinical practice.

Using portfolios for clinical practice learning and assessment: the pre-registration nursing student's perspective

Portfolios have been introduced to help to integrate theory and practice and thereby address the issue of the theory-practice divide. Although there has been much theoretical discussion about portfolio use in clinical placements, few studies have focused on the students' perceptions regarding their use. To obtain adult branch pre-registration nursing students' perspectives on using portfolios for their clinical practice learning and assessment, postal questionnaires were sent to 253 diploma of nursing students with a reminder to all students three weeks later. The response rate was 69% (174/253). This paper reports on the qualitative findings of the study, which employed both quantitative and qualitative methods. Although students stated that portfolios helped them in their development of self-awareness and independent learning, they indicated that portfolios do not sufficiently address the assessment of their clinical skills and the integration of theory and practice. They considered that portfolios could be greatly improved in three areas, namely in the conflict between using portfolios for both assessment and learning, the amount of support and guidance students feel they receive with their portfolio use and the portfolio design.

Portfolios in medical education: why do they meet with mixed success? A systematic review

CONTEXT

The move towards competence-based medical education has created a need for instruments that support and assess competence development. Portfolios seem suitable but mixed reports of their success are emerging.

METHODS

To examine the effectiveness of portfolios, we searched PubMed and EMBASE using the keyword 'portfol*', PsychInfo and ERIC using the keywords 'portfol*' and 'medical education' and references of retrieved papers for empirical studies on portfolios in all phases of medical education. Thirty of 1939 retrieved papers met the inclusion criteria and were analysed. Data were collated against the research question, number of subjects, design, setting, findings and limitations, purpose and content, mentoring and assessment. We analysed impact using a modified version of Kirkpatrick's hierarchy.

RESULTS

Because differences across studies precluded statistical meta-analysis, the data were analysed by context, goals and procedure. Positive effects were strongest in undergraduate education. Important factors for success were: clearly communicated goals and procedures; integration with curriculum and assessment; flexible structure; support through mentoring, and measures to heighten feasibility and reduce required time. Moderately good inter-rater reliability was reported and global criteria and discussions among raters were beneficial. Formative and summative assessment could be combined. Without assessment, portfolios were vulnerable to competition from other summative assessment instruments.

CONCLUSIONS

For portfolios to be effective in supporting and assessing competence development, robust integration into the curriculum and tutor support are essential. Further studies should focus on the effectiveness and user-friendliness of portfolios, the merits of holistic assessment procedures, and the competences of an effective portfolio mentor.

Use of reflective portfolios in health sciences education

Reflective portfolios are a collection of evidence that through critical reflection on its contents demonstrate achievement as well as personal and professional development. The purpose of this paper is to provide a review of the literature on the use of reflective portfolios and to develop a set of factors to guide schools and colleges of pharmacy as they implement reflective portfolios into their respective curricula as stated in Standards 2007.

Portfolios and the assessment of competence in nursing: A literature review

OBJECTIVES

The purpose of this paper is to explore the literature on the portfolio as a tool for the assessment of competence in nurse education.

DESIGN

Literature reviews are a valuable source of information; by locating, appraising and synthesising evidence from primary studies they can provide reliable answers to focused questions. They can also help to plan new research by identifying both what is known and not known in a given area. Literature reviews adhere to a scientific methodology which seeks to minimise bias and errors generating inferences based on the synthesis of best available evidence.

DATA SOURCES

The literature review was conducted utilising several databases, selected because of their relevance to the subject under review and including CINAHL and MEDLINE as well as a hand search of relevant journals and documents. The search terms included: nurses in education, portfolios and assessment and competence. Articles were included in the review if they focused on portfolios as a method of assessment in nurse education and if they were published after 1993 when portfolios first appeared in the nursing literature.

REVIEW METHODS

The review divides the literature into content themes allowing synthesis of the subject looking at consistencies and differences, followed by a summary and key arguments relating to the next theme.

RESULTS

Results highlight the importance of clear guidelines for portfolio construction and assessment, the importance of tri-partite support during portfolio development and guidelines for qualitative assessment. Where the portfolio process is well developed there are clear links to competence in practice.

CONCLUSIONS

The evidence on portfolios as a means of assessment continues to expand. If educators take on board the lessons learned from previous research and apply it to their assessment process, the difficulties found at present, in defining and measuring competence may be reduced.

Student nurses' knowledge, skills and attitudes towards the use of portfolios in a school of nursing

AIMS

The aims of this study are to examine the knowledge, skills and attitudes of student nurses about the value and purpose of portfolios and their relevance for professional development.

BACKGROUND

Portfolios have become an integral aspect of student centred learning within nursing. The literature is generally positive about the benefits of portfolios but questions remain about their effective implementation.

METHOD

A questionnaire was used to survey student nurses about the use of portfolios as a learning tool. The questionnaire was developed specifically for this study and was structured around a knowledge skills and attitude framework.

RESULTS

The sample was 413 preregistration student nurses in one school of nursing in the UK across course, year of study and branch of nursing. The main findings reveal that students are unclear but positive about the purpose and use of portfolios. Specifically, academic tutors do not provide sufficient direction in the development of their portfolios. There is also evidence that students found portfolios more useful for exploring the 'art' of nursing, rather than the 'technical' skills of nursing. Students also reported that they would welcome a dedicated portfolio module.

CONCLUSION

Students struggle with what is useful about portfolios and are unsure about its purpose. Nonetheless, they are positive towards portfolios as a learning tool and seem to acknowledge the usefulness of portfolios for professional development.

RELEVANCE TO CLINICAL PRACTICE

The importance of portfolios for professional development is now being emphasized but how academic tutors and clinical mentors support their development remains unclear. This study provides understanding about what support students would value.

Students’ perceptions on the use of portfolios in pre-registration nursing education: A questionnaire survey

BACKGROUND

Portfolios encourage both personal and professional development through the process of reflective practice and critical analysis. In clinical placements portfolios are used both as a learning tool and as an assessment tool.

OBJECTIVES

To determine nursing students' perceptions on the use and effectiveness of portfolios in their education as well as their perceptions on the use of the portfolio as both an assessment and learning tool in clinical placement.

DESIGN

Postal questionnaire survey.

SETTINGS

Higher Education Institute in the UK.

PARTICIPANTS

90 first and 84 third year pre-registration diploma of nursing students.

METHODS

Postal questionnaires were sent to 131 first year and 122 third year students, with a reminder 3 weeks later. The total response rate was 69% (174/253), with a response rate of 69% (90/131) for first year students and a response rate of 69% (84/122) for third year students.

RESULTS

Students felt that portfolios were very time-consuming, causing them a great deal of anxiety, and were not very effective in developing and assessing their learning and competence. Using portfolios for both assessment and learning creates a conflict. Any assessment reduces the honesty and learning value of reflective writing and of the portfolio. Additionally, students become increasingly demoralized with portfolio use over time with experience. The study suggested three main linked reasons for this, namely the conflict between using portfolios for both assessment and learning, the design of the portfolio and the amount of support and guidance students feel they receive with their portfolio use.

CONCLUSIONS

Portfolios can be very effective as an assessment and learning tool, but it is essential that both students and mentors receive clear guidelines on and comprehensive support with their use. They should be designed in such as way that they are relevant, clear and user-friendly for both students and mentors.

Learning portfolios - evidence of learning: an examination of students' perspectives

There is a lack of evaluative literature on the use of learning portfolios in nursing. Many students are still unclear on the benefit of using a learning portfolio, and fulfilling the criteria for the portfolios remains low priority. Over a 5-year period, tutors on the Specialist Practitioner Qualification in Critical Care found that there was a wide variety of practice in portfolio use. There is comparatively little known about how the students perceive the use of portfolios. Therefore, a small descriptive survey took place in 2001/2002 involving 22 previous students, to identify the value of portfolios from the student perspective. Recommendations from this study are given on how to improve portfolio use.

How the metaphor of a gap between theory and practice has influenced nursing education

The metaphor of a gap between theory and practice is firmly established in nursing education. Furthermore, it is embedded in conventional professional wisdom to the extent that when it used in professional reports, academic writing or in the conversations amongst nurses, it often requires no further explanation. The notion of a gap between theory and practice has proven to be a most fertile and an especially helpful concept which has enabled educators, clinicians and students to devise practical strategies, intended to resolve problems associated with the gap. However, that nursing has so enthusiastically embraced the notion of a gap between theory and practice may also have inhibited creativity in the design and delivery of programmes and thereby missing other opportunities for student nurses to learn from practical contexts.

Demonstration of portfolios to assess competency of residents

Residency educators are identifying approaches to measure resident competence. Portfolios are well suited since they require work already completed as part of patient care where competency must be demonstrated. This paper describes assessment of the reliability and validity of portfolios in a psychiatry residency program. This was a cross-sectional study across 4years of residency education. Using guidelines, 18 residents assembled portfolios containing five entries chosen from 13 skills. Trained raters scored the portfolios. Residents and faculty were interviewed about their perceptions. Generalizability results indicated five entries and two raters were sufficient for relative decisions. Six entries or a third rater would be sufficient for absolute decisions. Portfolio scores tended to improve with years of training and correlated with psychiatric knowledge but not clinical performance. Residents and faculty identified benefits to assembling a portfolio. Portfolios incorporate tasks embedded in the residency to provide evidence of resident competency. The results support that the score is reliable and valid.

From process to outcome: the effect of portfolio assessment on student learning

Literature suggests that assessment may influence student learning in a positive or negative way. Despite the support for the use of portfolio assessment, relatively little is known about its effect on student learning. The purpose of this study was to evaluate the effectiveness of portfolio assessment in enhancing student learning. Students undertaking the Bachelor of Nursing programme in a University in Hong Kong participated in the study. Twelve of the students who underwent portfolio assessment described, in individual semi-structured interviews, their experience and perceptions of this form of assessment. This paper reported on the qualitative findings of the study, which employed both qualitative and quantitative methods. Three themes emerged from the students' accounts: (1) the students favoured the use of portfolio assessment; (2) the process of preparing portfolios yielded positive academic and affective outcomes; and (3) unexpected findings in the form of spontaneous collaborative learning and apparent increased interest in learning during the process of preparing portfolios for those students who lacked motivation. The qualitative findings contribute to a better understanding of the use of portfolio assessment and have implications for teaching.

Portfolios and assessment of competence: a review of the literature

BACKGROUND

The literature review presented here was conducted as part of an English National Board for Nursing, Midwifery and Health Visiting funded project to evaluate the use of portfolios in the assessment of learning and competence. Initial consideration of the topic revealed the need to clarify the terminology and approaches used to assess competence using portfolios, and therefore the literature review was conducted to inform the study.

AIMS

To clarify definitions, theoretical bases and approaches to competence and the use of portfolios in the assessment of learning and competence in nursing education.

METHODS

A comprehensive literature review was conducted using the CINAHL and MEDLINE databases and the keywords competenc*, portfolios and nursing. Articles published in the period 1989-2001 in English were obtained and their reference lists scrutinized to identify additional references. Twenty articles were found using a combination of the keywords competenc* and portfolios, and 52 using the combination portfolios and nurse education. Articles were included in the review if they focused on the use of portfolios in nursing, and those concerned with professional or transitional portfolios were excluded. This article will analyse definitions of and approaches to competence and its measurement and to portfolios and their use as discussed in the articles identified.

RESULTS

Three approaches to competence were identified, each with its appropriate forms of assessment. With regard to portfolios, a number of definitions were again found, but there was a consensus that the theoretical basis of their use is theories of adult learning. A number of reasons for and advantages and disadvantages of their use were found, as well as varying ideas about what a portfolios should consist of and how it should be assessed.

CONCLUSION

A holistic approach to competence seems to be compatible with the use of portfolios to assess competence in nursing students, but the concept and its implementation is still evolving. A variety of assessment methods are needed for assessment and portfolios appear to have the potential to integrate these. Reflection is an essential component of a portfolio, as are the student-teacher relationship and explicit guidelines for constructing the portfolio. Issues of rigour in assessment of portfolios need to be addressed, but the assessor's professional judgement will inevitably enter into this assessment.