1
|
Brown T, Zakzanis KK. A review of the reliability of remote neuropsychological assessment. APPLIED NEUROPSYCHOLOGY. ADULT 2023:1-7. [PMID: 38000083 DOI: 10.1080/23279095.2023.2279208] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 11/26/2023]
Abstract
The provision of clinical neuropsychological services has predominately been undertaken by way of standardized administration in a face-to-face setting. Interpretation of psychometric findings in this context is dependent on the use of normative comparison. When the standardization in which such psychometric measures are employed deviates from how they were employed in the context of the development of its associated norms, one is left to question the reliability and hence, validity of any such findings and in turn, diagnostic decision making. In light of the current COVID-19 pandemic and resultant social distancing direction, face-to-face neuropsychological assessment has been challenging to undertake. As such, remote (i.e., virtual) neuropsychological assessment has become an obvious solution. Here, and before the results from remote neuropsychological assessment can be said to stand on firm scientific grounds, it is paramount to ensure that results garnered remotely are reliable and valid. To this end, we undertook a review of the literature and present an overview of the landscape. To date, the literature shows evidence for the reliability of remote administration and the clinical implications are paramount. When and where needed, neuropsychologists, psychometric technicians and examinees may no longer need to be in the same physical space to undergo an assessment. These findings are most relevant given the physical distancing practices because of COVID-19. And whilst remote assessment should never supplant face-to-face neuropsychological assessments, it does serve as a valid alternative when necessary.
Collapse
Affiliation(s)
- Tyler Brown
- Graduate Department of Psychological Clinical Science, University of Toronto, Toronto, ON, Canada
| | - Konstantine K Zakzanis
- Graduate Department of Psychological Clinical Science, University of Toronto, Toronto, ON, Canada
- Department of Psychology, University of Toronto, Scarborough, ON, Canada
| |
Collapse
|
2
|
Yung HY, Yeung WT, Law CW. The reliability of symptom assessment by telepsychiatry compared with face to face psychiatric interviews. Psychiatry Res 2022; 316:114728. [PMID: 35908348 PMCID: PMC9301901 DOI: 10.1016/j.psychres.2022.114728] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 03/22/2022] [Revised: 07/10/2022] [Accepted: 07/14/2022] [Indexed: 11/02/2022]
Abstract
INTRODUCTION With the start of the COVID-19 pandemic, the various social distancing policies imposed have mandated psychiatrists to consider the option of using telepsychiatry as an alternative to face-to-face interview in Hong Kong. Limitations over sample size, methodology and information technology were found in previous studies and the reliability of symptoms assessment remained a concern. AIM To evaluate the reliability of assessment of psychiatric symptoms by telepsychiatry comparing with face-to-face psychiatric interview. METHOD This study recruited a sample of adult psychiatric patients in psychiatric wards in Queen Mary Hospital. Semi-structural interviews with the use of standardized psychiatric assessment scales were carried out in telepsychiatry and face-to-face interview respectively by two clinicians and the reliability of psychiatric symptoms elicited were assessed. RESULTS 90 patients completed the assessments The inter-method reliability in Hamilton Depression Rating Scale, Hamilton Anxiety Rating Scale, Columbia Suicide Severity Rating Scale and Brief Psychiatric Rating Scale showed good agreement when compared with face-to-face interview. CONCLUSION Symptoms assessment by telepsychiatry is comparable to assessment conducted by face-to-face interview.
Collapse
Affiliation(s)
- Hiu Yan Yung
- Department of Psychiatry, Queen Mary Hospital, 102 Pok Fu Lam Road, Hong Kong.
| | | | | |
Collapse
|
3
|
Beishon LC, Elliott E, Hietamies TM, Mc Ardle R, O'Mahony A, Elliott AR, Quinn TJ. Diagnostic test accuracy of remote, multidomain cognitive assessment (telephone and video call) for dementia. Cochrane Database Syst Rev 2022; 4:CD013724. [PMID: 35395108 PMCID: PMC8992929 DOI: 10.1002/14651858.cd013724.pub2] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 11/07/2022]
Abstract
BACKGROUND Remote cognitive assessments are increasingly needed to assist in the detection of cognitive disorders, but the diagnostic accuracy of telephone- and video-based cognitive screening remains unclear. OBJECTIVES To assess the test accuracy of any multidomain cognitive test delivered remotely for the diagnosis of any form of dementia. To assess for potential differences in cognitive test scoring when using a remote platform, and where a remote screener was compared to the equivalent face-to-face test. SEARCH METHODS We searched ALOIS, the Cochrane Dementia and Cognitive Improvement Group Specialized Register, CENTRAL, MEDLINE, Embase, PsycINFO, CINAHL, Web of Science, LILACS, and ClinicalTrials.gov (www. CLINICALTRIALS gov/) databases on 2 June 2021. We performed forward and backward searching of included citations. SELECTION CRITERIA We included cross-sectional studies, where a remote, multidomain assessment was administered alongside a clinical diagnosis of dementia or equivalent face-to-face test. DATA COLLECTION AND ANALYSIS Two review authors independently assessed risk of bias and extracted data; a third review author moderated disagreements. Our primary analysis was the accuracy of remote assessments against a clinical diagnosis of dementia. Where data were available, we reported test accuracy as sensitivity and specificity. We did not perform quantitative meta-analysis as there were too few studies at individual test level. For those studies comparing remote versus in-person use of an equivalent screening test, if data allowed, we described correlations, reliability, differences in scores and the proportion classified as having cognitive impairment for each test. MAIN RESULTS The review contains 31 studies (19 differing tests, 3075 participants), of which seven studies (six telephone, one video call, 756 participants) were relevant to our primary objective of describing test accuracy against a clinical diagnosis of dementia. All studies were at unclear or high risk of bias in at least one domain, but were low risk in applicability to the review question. Overall, sensitivity of remote tools varied with values between 26% and 100%, and specificity between 65% and 100%, with no clearly superior test. Across the 24 papers comparing equivalent remote and in-person tests (14 telephone, 10 video call), agreement between tests was good, but rarely perfect (correlation coefficient range: 0.48 to 0.98). AUTHORS' CONCLUSIONS Despite the common and increasing use of remote cognitive assessment, supporting evidence on test accuracy is limited. Available data do not allow us to suggest a preferred test. Remote testing is complex, and this is reflected in the heterogeneity seen in tests used, their application, and their analysis. More research is needed to describe accuracy of contemporary approaches to remote cognitive assessment. While data comparing remote and in-person use of a test were reassuring, thresholds and scoring rules derived from in-person testing may not be applicable when the equivalent test is adapted for remote use.
Collapse
Affiliation(s)
- Lucy C Beishon
- Department of Cardiovascular Sciences, University of Leicester, Leicester, UK
| | - Emma Elliott
- Institute of Cardiovascular and Medical Sciences, University of Glasgow, Glasgow, UK
| | - Tuuli M Hietamies
- Institute of Cardiovascular and Medical Sciences, University of Glasgow, Glasgow, UK
- Anesthesiology, Perioperative and Pain Medicine, Stanford University, Stanford, California, USA
| | - Riona Mc Ardle
- Translational and Clinical Research Institute, Newcastle University, Newcastle, UK
| | - Aoife O'Mahony
- CUBRIC, School of Psychology, Cardiff University, Cardiff, UK
| | - Amy R Elliott
- Department of Cardiovascular Sciences, University of Leicester, Leicester, UK
- NIHR Academic Clinical Fellow, University Hospitals Leicester, Leicester, UK
| | - Terry J Quinn
- Institute of Cardiovascular and Medical Sciences, University of Glasgow, Glasgow, UK
| |
Collapse
|
4
|
Barnett P, Goulding L, Casetta C, Jordan H, Sheridan-Rains L, Steare T, Williams J, Wood L, Gaughran F, Johnson S. Implementation of Telemental Health Services Before COVID-19: Rapid Umbrella Review of Systematic Reviews. J Med Internet Res 2021; 23:e26492. [PMID: 34061758 PMCID: PMC8335619 DOI: 10.2196/26492] [Citation(s) in RCA: 61] [Impact Index Per Article: 20.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/14/2020] [Revised: 03/30/2021] [Accepted: 04/30/2021] [Indexed: 02/06/2023] Open
Abstract
BACKGROUND Telemental health care has been rapidly adopted for maintaining services during the COVID-19 pandemic, and a substantial interest is now being devoted in its future role. Service planning and policy making for recovery from the pandemic and beyond should draw on both COVID-19 experiences and the substantial research evidence accumulated before this pandemic. OBJECTIVE We aim to conduct an umbrella review of systematic reviews available on the literature and evidence-based guidance on telemental health, including both qualitative and quantitative literature. METHODS Three databases were searched between January 2010 and August 2020 for systematic reviews meeting the predefined criteria. The retrieved reviews were independently screened, and those meeting the inclusion criteria were synthesized and assessed for risk of bias. Narrative synthesis was used to report these findings. RESULTS In total, 19 systematic reviews met the inclusion criteria. A total of 15 reviews examined clinical effectiveness, 8 reported on the aspects of telemental health implementation, 10 reported on acceptability to service users and clinicians, 2 reported on cost-effectiveness, and 1 reported on guidance. Most reviews were assessed to be of low quality. The findings suggested that video-based communication could be as effective and acceptable as face-to-face formats, at least in the short term. Evidence on the extent of digital exclusion and how it can be overcome and that on some significant contexts, such as children and young people's services and inpatient settings, was found to be lacking. CONCLUSIONS This umbrella review suggests that telemental health has the potential to be an effective and acceptable form of service delivery. However, we found limited evidence on the impact of its large-scale implementation across catchment areas. Combining previous evidence and COVID-19 experiences may allow realistic planning for the future implementation of telemental health.
Collapse
Affiliation(s)
- Phoebe Barnett
- Centre for Outcomes Research and Effectiveness, Division of Psychology and Language Sciences, University College London, London, United Kingdom.,NIHR Mental Health Policy Research Unit, Division of Psychiatry, University College London, London, United Kingdom
| | - Lucy Goulding
- King's Improvement Science, Centre for Implementation Science, King's College London, London, United Kingdom
| | - Cecilia Casetta
- NIHR Applied Research Collaboration, King's College London, London, United Kingdom
| | - Harriet Jordan
- NIHR Applied Research Collaboration, King's College London, London, United Kingdom.,Institute of Psychiatry, Psychology and Neuroscience, King's College London and South London and Maudsley NHS Trust, London, United Kingdom
| | - Luke Sheridan-Rains
- NIHR Mental Health Policy Research Unit, Division of Psychiatry, University College London, London, United Kingdom
| | - Thomas Steare
- NIHR Mental Health Policy Research Unit, Division of Psychiatry, University College London, London, United Kingdom
| | - Julie Williams
- Centre for Implementation Science, Health Service and Population Research Department, King's College London, London, United Kingdom
| | - Lisa Wood
- Division of Psychiatry, University College London, London, United Kingdom
| | - Fiona Gaughran
- NIHR Applied Research Collaboration, King's College London, London, United Kingdom.,Institute of Psychiatry, Psychology and Neuroscience, King's College London and South London and Maudsley NHS Trust, London, United Kingdom
| | - Sonia Johnson
- NIHR Mental Health Policy Research Unit, Division of Psychiatry, University College London, London, United Kingdom.,Camden and Islington NHS Foundation Trust, London, United Kingdom
| |
Collapse
|
5
|
Liu J, Obioha T, Magpantay J, Kanovsky D, Niles L, Scholle SH. Inclusion of Telemedicine in Behavioral Health Quality Measures. Psychiatr Serv 2020; 71:1288-1291. [PMID: 33076791 DOI: 10.1176/appi.ps.201900449] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 11/30/2022]
Abstract
OBJECTIVE The authors evaluated available evidence to determine whether telemedicine services should be allowed in seven nationally reported behavioral health quality measures in the Healthcare Effectiveness Data and Information Set. METHODS The authors searched PubMed, Google Scholar, and the Cochrane Database of Systematic Reviews for articles that met several inclusion criteria (relevant diagnosis and age, as specified in the quality measures). They also consulted expert panels on whether, and how, to include telemedicine in behavioral health measure specifications. RESULTS Thirty-two studies met the inclusion criteria. Their findings suggested that video conferencing, telephone calls, and web-based telemedicine modalities are as effective as in-person visits for diagnosis and management of most mental health and substance use disorders. Expert panels supported including telemedicine modalities in specific behavioral health measures. CONCLUSIONS Specific telemedicine modalities are effective, convenient ways to deliver behavioral health care. Revising quality measures to allow telemedicine services will support new methods for providing care.
Collapse
Affiliation(s)
- Junqing Liu
- National Committee for Quality Assurance, Washington, D.C
| | | | | | | | - Lauren Niles
- National Committee for Quality Assurance, Washington, D.C
| | | |
Collapse
|
6
|
Ghosh A, Verma M, Lal S. A Review of Models and Efficacy of Telepsychiatry for Inpatient Service Delivery: Proposing a Model for Indian Settings. Indian J Psychol Med 2020; 42:34S-40S. [PMID: 33354061 PMCID: PMC7736746 DOI: 10.1177/0253717620958168] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Accepted: 08/20/2020] [Indexed: 11/15/2022] Open
Abstract
BACKGROUND The use of telepsychiatry (TP) for inpatient service delivery is still an emerging field and there is limited literature on its practice and evidence. This review was conducted with the objectives of (a) exploring the models of TP for inpatient service delivery, (b) qualitative synthesis of the efficacy of TP in inpatient settings, and (c) proposing a best-fit model of TP-based inpatient care for Indian settings. METHODS An electronic database search was conducted on July 22, 2020, in PubMed, Directory of Open Access Journals, and Google Scholar for relevant articles. Seventeen articles were included in the review. RESULTS The review revealed three models for TP-based inpatient care; direct care model, teleconsultation model, and the collaborative care model. Preliminary evidence suggests that TP is cost-effective and reliable, and that patients and service providers are highly satisfied with this approach. Evidence gaps were seen for some diagnostic categories such as psychosis and for extremes of age groups. Based on the existing models, we propose an Indian model for implementing TP in inpatient settings. CONCLUSION Promising initial results and the evidence gaps highlight the need for further research in this area.
Collapse
Affiliation(s)
- Abhishek Ghosh
- Dept. of Psychiatry, Postgraduate Institute of Medical Education and Research, Chandigarh, India
| | - Meha Verma
- Dept. of Psychiatry, Postgraduate Institute of Medical Education and Research, Chandigarh, India
| | - Shalini Lal
- School of Rehabilitation, University of Montreal, Montreal, QC, Canada.,Youth Mental Health and Technology Lab, University of Montreal Hospital Research Centre, Montreal, QC, Canada.,PEPP Montreal and ACCESS Open Minds, Douglas Mental Health University Institute, Montreal, QC, Canada
| |
Collapse
|
7
|
Marra DE, Hamlet KM, Bauer RM, Bowers D. Validity of teleneuropsychology for older adults in response to COVID-19: A systematic and critical review. Clin Neuropsychol 2020; 34:1411-1452. [DOI: 10.1080/13854046.2020.1769192] [Citation(s) in RCA: 72] [Impact Index Per Article: 18.0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/17/2023]
Affiliation(s)
- David E. Marra
- Department of Clinical and Health Psychology, University of Florida, Gainesville, FL, USA
- McKnight Brain Institute, University of Florida, Gainesville, FL, USA
| | - Kristin M. Hamlet
- Department of Clinical and Health Psychology, University of Florida, Gainesville, FL, USA
| | - Russell M. Bauer
- Department of Clinical and Health Psychology, University of Florida, Gainesville, FL, USA
- Department of Neurology, University of Florida, Gainesville, FL, USA
- Brain Rehabilitation Research Center, Malcolm Randall VAMC, Gainesville, FL, USA
| | - Dawn Bowers
- Department of Clinical and Health Psychology, University of Florida, Gainesville, FL, USA
- Department of Neurology, University of Florida, Gainesville, FL, USA
| |
Collapse
|
8
|
Christensen LF, Moller AM, Hansen JP, Nielsen CT, Gildberg FA. Patients' and providers' experiences with video consultations used in the treatment of older patients with unipolar depression: A systematic review. J Psychiatr Ment Health Nurs 2020; 27:258-271. [PMID: 31677331 DOI: 10.1111/jpm.12574] [Citation(s) in RCA: 28] [Impact Index Per Article: 7.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 04/10/2019] [Revised: 10/23/2019] [Accepted: 10/30/2019] [Indexed: 12/11/2022]
Abstract
WHAT IS KNOWN ON THE SUBJECT?: Depression is the most common cause of mental illness amongst older people. As a result of the growth of the older population, it is expected that an increasing number of older people will need treatment. Depression can be effectively treated-but fewer than half of those who are affected receive treatment. Barriers to receiving treatment are lack of resources and trained healthcare providers, social stigma, incorrect diagnoses and long distances to treatment facilities. There is an increased need for alternative ways of treating patients with depression. The use of video consultations has shown to be a viable option for delivering mental health care to older patients. Use of video consultations allows patients to receive treatment in their own homes. None of the existing reviews have focused on satisfaction with the use of video consultations amongst older people with depression. This paper provides knowledge regarding the use of video consultations, especially for older people with depression. WHAT THIS PAPER ADDS TO EXISTING KNOWLEDGE?: No previous review has sought to understand the use of video consultations in mental health care to older patients with depression. The results show that video consultations support mental health practice, especially as a useful alternative when face-to-face therapy is not possible. An initial scepticism from participants quickly disappeared when video consultations were experienced in action. The challenges associated with the use of video consultations seem to consist of technical problems and lack of support from staff. WHAT ARE THE IMPLICATIONS FOR PRACTICE?: Mental health practitioners should consider the use of video consultations because it can support mental health practice, especially as a useful alternative when face-to-face therapy is not possible. ABSTRACT: Introduction Depression is the leading cause of mental illness amongst an ageing population and fewer than half of those who are affected receive treatment. There is an increased need for alternative ways of treating patients; the use of video consultations has been shown to be a viable option for delivering mental health care. However, none of the existing reviews have focused on satisfaction with the use of video consultations amongst older people with depression and providers. Aim To conduct a systematic review of the existing literature focusing on patients' and providers' experiences of video consultations for depression. Method Eight scientific databases were searched. In all, 3,537 articles were identified and, of these, 21 peer-reviewed articles were included in this review. Results The results show that video consultations support mental health practice, especially as a useful alternative when face-to-face therapy is not possible. Any initial scepticism quickly disappeared when video consultations were experienced in action. The challenges seem to consist of technical problems and lack of support from staff. Discussion The experiences and satisfaction of older people with depression seem to be positive, although methodological limitations and deficiencies of the reviewed articles should be considered. More qualitative research is needed, and future studies should focus on specific diagnoses and providers' experiences. Implications for practice Video consultations support mental health practice, especially as a useful alternative when face-to-face therapy is not possible.
Collapse
Affiliation(s)
- Lone Fisker Christensen
- Department of Regional Health Research, Faculty of Health Sciences, University of Southern Denmark, Odense, Denmark.,Research Unit for Telepsychiatry and E-mental Health, University of Southern Denmark, Odense, Denmark.,Research Unit, Department of Mental Health, Esbjerg, Region of Southern Denmark, Esbjerg, Denmark
| | - Anne Marie Moller
- Research Unit for Telepsychiatry and E-mental Health, University of Southern Denmark, Odense, Denmark.,Health Promotion Research, Department of Public Health, University of Southern Denmark, Odense, Denmark
| | - Jens Peter Hansen
- Research Unit, Department of Mental Health, Esbjerg, Region of Southern Denmark, Esbjerg, Denmark.,Center for Psychiatric Nursing and Health Research, Department of Regional Health Research, Faculty of Health Sciences, University of Southern Denmark, Odense, Denmark
| | - Connie Thuroe Nielsen
- Department of Regional Health Research, Faculty of Health Sciences, University of Southern Denmark, Odense, Denmark.,Department of Mental Health, Vejle, Region of Southern Denmark, Vejle, Denmark
| | - Frederik Alkier Gildberg
- Center for Psychiatric Nursing and Health Research, Department of Regional Health Research, Faculty of Health Sciences, University of Southern Denmark, Odense, Denmark
| |
Collapse
|
9
|
Gentry MT, Lapid MI, Rummans TA. Geriatric Telepsychiatry: Systematic Review and Policy Considerations. Am J Geriatr Psychiatry 2019; 27:109-127. [PMID: 30416025 DOI: 10.1016/j.jagp.2018.10.009] [Citation(s) in RCA: 93] [Impact Index Per Article: 18.6] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 06/13/2018] [Revised: 10/12/2018] [Accepted: 10/12/2018] [Indexed: 01/02/2023]
Abstract
Telemental health (TMH) for older patients has the potential to increase access to geriatric specialists, reduce travel times for patients and providers, and reduce ever growing healthcare costs. This systematic review article examines the literature regarding psychiatric assessment and treatment via telemedicine for geriatric patients. English language literature was searched using Ovid Medline, PubMed, and PsycINFO with search terms including telemedicine, telemental health, aging, and dementia. Abstracts were reviewed for relevance based on inclusion criteria. Multiple study types were reviewed, including open label, qualitative and randomized controlled trial study designs. Data was compiled regarding participants, study intervention, and outcomes. 76 articles were included. TMH was shown to be feasible and well accepted in the areas of inpatient and nursing home consultation, cognitive testing, dementia diagnosis and treatment, depression in integrated and collaborative care models, and psychotherapy. There is limited data on cost-effectiveness of TMH in the elderly. This article will discuss the current barriers to broader implementation of telemedicine for geriatric patients including reimbursement from the Medicare program. Medicare reimbursement for telemedicine is limited to rural areas, which does not allow for the widespread development of telemedicine programs. All Medicare beneficiaries would benefit from increased access to telemedicine services, not only those living in rural areas. As many elderly and disabled individuals have mobility problems, home-based telemedicine services should also be made available. There are efforts in Congress to expand the coverage of these services under Medicare, but strong advocacy will be needed to ensure these efforts are successful.
Collapse
Affiliation(s)
- Melanie T Gentry
- Department of Psychiatry and Psychology, Mayo Clinic, Rochester, Minnesota.
| | - Maria I Lapid
- Department of Psychiatry and Psychology, Mayo Clinic, Rochester, Minnesota
| | - Teresa A Rummans
- Department of Psychiatry and Psychology, Mayo Clinic, Rochester, Minnesota
| |
Collapse
|
10
|
Brearly TW, Shura RD, Martindale SL, Lazowski RA, Luxton DD, Shenal BV, Rowland JA. Neuropsychological Test Administration by Videoconference: A Systematic Review and Meta-Analysis. Neuropsychol Rev 2017. [DOI: 10.1007/s11065-017-9349-1] [Citation(s) in RCA: 172] [Impact Index Per Article: 24.6] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/17/2023]
|
11
|
Castanho TC, Amorim L, Moreira PS, Mariz J, Palha JA, Sousa N, Santos NC. Assessing Cognitive Function in Older Adults Using a Videoconference Approach. EBioMedicine 2016; 11:278-284. [PMID: 27515687 PMCID: PMC5049917 DOI: 10.1016/j.ebiom.2016.08.001] [Citation(s) in RCA: 20] [Impact Index Per Article: 2.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/22/2016] [Revised: 07/22/2016] [Accepted: 08/01/2016] [Indexed: 01/17/2023] Open
Abstract
BACKGROUND The use of communication technologies is an emerging trend in healthcare and research. Despite efficient, reliable and accurate neuropsychological batteries to evaluate cognitive performance in-person, more diverse and less expensive and time consuming solutions are needed. Here we conducted a pilot study to determine the applicability of a videoconference (VC, Skype®) approach to assess cognitive function in older adults, using The Telephone Interview for Cognitive Status-Modified - Portuguese version (TICSM-PT). METHODS After inclusion and exclusion criteria, 69 individuals (mean age=74.90±9.46years), selected from registries of local health centers and assisted-living facilities, were assessed on cognitive performance using videoconference, telephone and in-person approaches. FINDINGS The videoconference administration method yielded comparable results to the traditional application. Correlation analyses showed high associations between the testing modalities: TICSM-PT VC and TICSM-PT telephone (r=0.885), TICSM-PT VC and MMSE face-to-face (r=0.801). Using the previously validated threshold for cognitive impairment on the TICSM-PT telephone, TICSM-PT VC administration presented a sensitivity of 87.8% and a specificity of 84.6%. INTERPRETATION Findings indicate for the range of settings where videoconference approaches can be used, and for their applicability and acceptability, providing an alternative to current cognitive assessment methods. Continued validation studies and adaptation of neuropsychological instruments is warranted.
Collapse
Affiliation(s)
- Teresa Costa Castanho
- Life and Health Sciences Research Institute (ICVS), School of Health Sciences, University of Minho, Braga, Portugal; ICVS/3B's, PT Government Associate Laboratory, Braga, Guimarães, Portugal; Clinical Academic Center-Braga (CCAB), Braga, Portugal
| | - Liliana Amorim
- Life and Health Sciences Research Institute (ICVS), School of Health Sciences, University of Minho, Braga, Portugal; ICVS/3B's, PT Government Associate Laboratory, Braga, Guimarães, Portugal; Clinical Academic Center-Braga (CCAB), Braga, Portugal
| | - Pedro Silva Moreira
- Life and Health Sciences Research Institute (ICVS), School of Health Sciences, University of Minho, Braga, Portugal; ICVS/3B's, PT Government Associate Laboratory, Braga, Guimarães, Portugal; Clinical Academic Center-Braga (CCAB), Braga, Portugal
| | - José Mariz
- Life and Health Sciences Research Institute (ICVS), School of Health Sciences, University of Minho, Braga, Portugal; ICVS/3B's, PT Government Associate Laboratory, Braga, Guimarães, Portugal; Clinical Academic Center-Braga (CCAB), Braga, Portugal; Emergency Department, Intermediate Care Unit (EDIMCU), Hospital de Braga, Braga, Portugal
| | - Joana Almeida Palha
- Life and Health Sciences Research Institute (ICVS), School of Health Sciences, University of Minho, Braga, Portugal; ICVS/3B's, PT Government Associate Laboratory, Braga, Guimarães, Portugal; Clinical Academic Center-Braga (CCAB), Braga, Portugal
| | - Nuno Sousa
- Life and Health Sciences Research Institute (ICVS), School of Health Sciences, University of Minho, Braga, Portugal; ICVS/3B's, PT Government Associate Laboratory, Braga, Guimarães, Portugal; Clinical Academic Center-Braga (CCAB), Braga, Portugal
| | - Nadine Correia Santos
- Life and Health Sciences Research Institute (ICVS), School of Health Sciences, University of Minho, Braga, Portugal; ICVS/3B's, PT Government Associate Laboratory, Braga, Guimarães, Portugal; Clinical Academic Center-Braga (CCAB), Braga, Portugal.
| |
Collapse
|
12
|
Abstract
The use of videoconference technology to deliver health care diagnostics and treatment continues to grow at a rapid pace. Telepsychiatry and telepsychology applications are well-accepted by patients and providers, and both diagnostic and treatment outcomes have generally been similar to traditional face-to-face interactions. Preliminary applications of videoconference-based neuropsychological assessment (teleneuropsychology) have yielded promising results in the feasibility and reliability of several standard tests, although large-scale studies are lacking. This investigation was conducted to determine the reliability of video teleconference (VTC) - based neuropsychological assessment using a brief battery of standard neuropsychological tests commonly used in the evaluation of known or suspected dementia. Tests included the Mini-Mental State Examination (MMSE), Hopkins Verbal Learning Test-Revised, Digit Span forward and backward, short form Boston Naming Test, Letter and Category Fluency, and Clock Drawing. Tests were administered via VTC and in-person to subjects, counterbalanced using alternate test forms and standard instructions. Two hundred two adult subjects were tested in both rural and urban settings, including 83 with cognitive impairment and 119 healthy controls. We found highly similar results across VTC and in-person conditions, with significant intraclass correlations (mean=.74; range: 0.55-0.91) between test scores. Findings remained consistent in subjects with or without cognitive impairment and in persons with MMSE scores as low as 15. VTC-based neuropsychological testing is a valid and reliable alternative to traditional face-to-face assessment using selected measures. More VTC-based studies using additional tests in different populations are needed to fully explore the utility of this new testing medium.
Collapse
|
13
|
Teleneuropsychology: evidence for video teleconference-based neuropsychological assessment. J Int Neuropsychol Soc 2014. [PMID: 25343269 DOI: 10.1017/s135561771400873] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.1] [Reference Citation Analysis] [Abstract] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 12/11/2022]
Abstract
The use of videoconference technology to deliver health care diagnostics and treatment continues to grow at a rapid pace. Telepsychiatry and telepsychology applications are well-accepted by patients and providers, and both diagnostic and treatment outcomes have generally been similar to traditional face-to-face interactions. Preliminary applications of videoconference-based neuropsychological assessment (teleneuropsychology) have yielded promising results in the feasibility and reliability of several standard tests, although large-scale studies are lacking. This investigation was conducted to determine the reliability of video teleconference (VTC) - based neuropsychological assessment using a brief battery of standard neuropsychological tests commonly used in the evaluation of known or suspected dementia. Tests included the Mini-Mental State Examination (MMSE), Hopkins Verbal Learning Test-Revised, Digit Span forward and backward, short form Boston Naming Test, Letter and Category Fluency, and Clock Drawing. Tests were administered via VTC and in-person to subjects, counterbalanced using alternate test forms and standard instructions. Two hundred two adult subjects were tested in both rural and urban settings, including 83 with cognitive impairment and 119 healthy controls. We found highly similar results across VTC and in-person conditions, with significant intraclass correlations (mean=.74; range: 0.55-0.91) between test scores. Findings remained consistent in subjects with or without cognitive impairment and in persons with MMSE scores as low as 15. VTC-based neuropsychological testing is a valid and reliable alternative to traditional face-to-face assessment using selected measures. More VTC-based studies using additional tests in different populations are needed to fully explore the utility of this new testing medium.
Collapse
|
14
|
Vedel I, Akhlaghpour S, Vaghefi I, Bergman H, Lapointe L. Health information technologies in geriatrics and gerontology: a mixed systematic review. J Am Med Inform Assoc 2013; 20:1109-19. [PMID: 23666776 PMCID: PMC3822120 DOI: 10.1136/amiajnl-2013-001705] [Citation(s) in RCA: 42] [Impact Index Per Article: 3.8] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/06/2013] [Revised: 04/08/2013] [Accepted: 04/13/2013] [Indexed: 01/18/2023] Open
Abstract
OBJECTIVE To review, categorize, and synthesize findings from the literature about the application of health information technologies in geriatrics and gerontology (GGHIT). MATERIALS AND METHODS This mixed-method systematic review is based on a comprehensive search of Medline, Embase, PsychInfo and ABI/Inform Global. Study selection and coding were performed independently by two researchers and were followed by a narrative synthesis. To move beyond a simple description of the technologies, we employed and adapted the diffusion of innovation theory (DOI). RESULTS 112 papers were included. Analysis revealed five main types of GGHIT: (1) telecare technologies (representing half of the studies); (2) electronic health records; (3) decision support systems; (4) web-based packages for patients and/or family caregivers; and (5) assistive information technologies. On aggregate, the most consistent finding proves to be the positive outcomes of GGHIT in terms of clinical processes. Although less frequently studied, positive impacts were found on patients' health, productivity, efficiency and costs, clinicians' satisfaction, patients' satisfaction and patients' empowerment. DISCUSSION Further efforts should focus on improving the characteristics of such technologies in terms of compatibility and simplicity. Implementation strategies also should be improved as trialability and observability are insufficient. CONCLUSIONS Our results will help organizations in making decisions regarding the choice, planning and diffusion of GGHIT implemented for the care of older adults.
Collapse
Affiliation(s)
- Isabelle Vedel
- Department of Family Medicine, McGill University, Solidage Research Group on Frailty and Aging, Lady Davis Institute, Jewish General Hospital, Montreal, Quebec, Canada
- Division of Geriatric Medicine, McGill University, Montreal, Quebec, Canada
- Desautels Faculty of Management, McGill University, Montreal, Quebec, Canada
| | - Saeed Akhlaghpour
- Desautels Faculty of Management, McGill University, Montreal, Quebec, Canada
- Middlesex University Business School, Middlesex University, London, UK
| | - Isaac Vaghefi
- Desautels Faculty of Management, McGill University, Montreal, Quebec, Canada
| | - Howard Bergman
- Department of Family Medicine, McGill University, Solidage Research Group on Frailty and Aging, Lady Davis Institute, Jewish General Hospital, Montreal, Quebec, Canada
- Division of Geriatric Medicine, McGill University, Montreal, Quebec, Canada
| | - Liette Lapointe
- Department of Family Medicine, McGill University, Solidage Research Group on Frailty and Aging, Lady Davis Institute, Jewish General Hospital, Montreal, Quebec, Canada
- Desautels Faculty of Management, McGill University, Montreal, Quebec, Canada
| |
Collapse
|
15
|
Abstract
BACKGROUND The use of telemedicine for the care of mental health problems has developed significantly over the last decade thanks to the emergence of a number of stable telepsychiatry programs in many countries. Parallel to this development, this care modality has also targeted specific populations with higher difficulty in gaining access to mental health services such as the elderly. Telepsychogeriatrics is expected to have an increasing role in providing care to geographically isolated rural communities, with a particular focus on long-term care facilities, in light of the high prevalence of psychiatric disorders in these centers and the lack of available specialized care. METHODS A thorough search of the literature was conducted using Medline, Web of Science, and PsychINFO databases in order to gather available evidence on the applicability of telepsychiatry, specifically the use of videoconferencing for remote consultation, in the elderly population with mental disorders. A succinct description of the selected studies is given along with a general reflection on the state-of-the-art in the field of psychogeriatric clinical practice and research. RESULTS Research on the use of telemedicine in this age group has taken into account their special characteristics, and has focused on demonstrating its applicability, the acceptance and satisfaction of elderly users and their healthcare providers, the possibility of carrying out cognitive and diagnostic assessments, and the efficiency of these programs. CONCLUSIONS Despite limited experience, telepsychogeriatrics appears to be a viable option, well accepted by patients, including those having dementia. More systematized studies are needed in this new field based on larger sample sizes, including comparison with traditional consultations and assessment of the clinical outcomes.
Collapse
|
16
|
Nieves JE, Godleski LS, Stack KM, Zinanni T. Videophones for intensive case management of psychiatric outpatients. J Telemed Telecare 2009; 15:51-4. [PMID: 19139221 DOI: 10.1258/jtt.2008.080706] [Citation(s) in RCA: 8] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/18/2023]
Abstract
When visiting patients with serious mental illness at their homes, case managers carried a portable videophone. This was used to access the hospital clinical team via the home telephone line, when an acute clinical need arose in addressing questions related to medication management and treatment planning; travel was therefore avoided. In an acceptability study lasting 12 months, 24 patients received the supplemental videophone mental health service and 19 of them completed a satisfaction survey. Only one patient was not satisfied with the videophone treatment, while 74% of them were very satisfied. Specific areas of satisfaction concerned the savings of time and travel, assistance with medication questions and increased involvement in treatment. No patient reported any difficulty in using the equipment. Staff members were enthusiastic about the decrease in travel time and there were no complaints about the videophone process or encounters. During the study, 135 h of the case manager's time was saved by using the videophone for urgent access visits, equivalent to a saving of $4000 in salary costs. In addition, the patients saved 135 h in travel time. We believe that there are other potential uses of videophones for seriously mentally ill patients, including discharge planning, intensive post-discharge monitoring and transition to community life.
Collapse
Affiliation(s)
- J Edwin Nieves
- VA Hampton Medical Center, Hampton, Virginia 23667, USA.
| | | | | | | |
Collapse
|
17
|
Godleski L, Nieves JE, Darkins A, Lehmann L. VA telemental health: suicide assessment. BEHAVIORAL SCIENCES & THE LAW 2008; 26:271-286. [PMID: 18548515 DOI: 10.1002/bsl.811] [Citation(s) in RCA: 47] [Impact Index Per Article: 2.9] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/26/2023]
Abstract
The Department of Veterans Affairs (VA) encompasses one of the largest telemental health networks in the world, with over 45,000 videoconferencing and over 5,000 home telemental health encounters annually. Recently, the VA designated suicide prevention as a major priority, with telehealth modalities providing opportunities for remote interventions. Suicide risk assessments, using videoconferencing, are now documented in the literature, as are current studies that find telemental health to be equivalent to face-to-face treatment. Remote assessment of suicidality, however, involves complex legal issues: licensing requirements for remote delivery of care, legal procedures for involuntary detainment and commitment of potentially harmful patients, and liability questions related to the remote nature of the mental health service. VA best practices for remote suicide risk assessment include paradigms for establishing procedures in the context of legal challenges (licensing and involuntary detainment/commitment), for utilizing clinical assessment and triage decision protocols, and for contingency planning to optimize patient care and reduce liability.
Collapse
Affiliation(s)
- Linda Godleski
- Yale School of Medicine, VA Connecticut Healthcare System, CT 06516, USA.
| | | | | | | |
Collapse
|
18
|
Williams JBW, Ellis A, Middleton A, Kobak KA. Primary care patients in psychiatric clinical trials: a pilot study using videoconferencing. Ann Gen Psychiatry 2007; 6:24. [PMID: 17916254 PMCID: PMC2093932 DOI: 10.1186/1744-859x-6-24] [Citation(s) in RCA: 6] [Impact Index Per Article: 0.4] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 07/26/2007] [Accepted: 10/04/2007] [Indexed: 01/18/2023] Open
Abstract
BACKGROUND While primary care physicians play a pivotal role in the treatment of depression, collaboration between primary care and psychiatry in clinical research has been limited. Primary care settings provide unique opportunities to improve the methodology of psychiatric clinical trials, by providing more generalizable and less treatment-resistant patients. We examined the feasibility of identifying, recruiting, screening and assessing primary care patients for psychiatric clinical trials using high-quality videoconferencing in a mock clinical trial. METHODS 1329 patients at two primary care clinics completed a self-report questionnaire. Those screening positive for major depression, panic, or generalized anxiety were given a diagnostic interview via videoconference. Those eligible were provided treatment as usual by their primary care physician, and had 6 weekly assessments by the off-site clinician via videoconferencing. RESULTS 45 patients were enrolled over 22 weeks, with 36 (80%) completing the six-week study with no more than two missed appointments. All diagnostic groups improved significantly; 94% reported they would participate again, 87% would recommend participation to others, 96% felt comfortable communicating via videoconference, and 94% were able to satisfactorily communicate their feelings via video. CONCLUSION Results showed that primary care patients will enroll, participate in and complete psychiatric research protocols using remote interviews conducted via videoconference.
Collapse
Affiliation(s)
- Janet BW Williams
- MedAvante, Inc., MedAvante Research Institute, Hamilton, NJ, USA
- Columbia University, Dept. of Psychiatry, New York, NY, USA
| | - Amy Ellis
- MedAvante, Inc., MedAvante Research Institute, Hamilton, NJ, USA
| | | | - Kenneth A Kobak
- MedAvante, Inc., MedAvante Research Institute, Hamilton, NJ, USA
| |
Collapse
|
19
|
Arnaert A, Klooster J, Chow V. Attitudes Toward Videotelephones: An Exploratory Study of Older Adults with Depression. J Gerontol Nurs 2007; 33:5-13. [PMID: 17899995 DOI: 10.3928/00989134-20070901-02] [Citation(s) in RCA: 17] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/18/2023]
Abstract
Although technology holds promising benefits to support older adults at home, attitudes can be a hindrance to the acceptance and use of new technologies. The aim of this study was to explore attitudes of older adults with depressive symptoms using videotelephones (VTs) in their homes. A descriptive design involving 4 participants who received problem solving interventions from a clinical psychologist via VTs was used. Data analysis revealed that the participants' preattitudes were dependent on their active or passive role in the learning process of the new technology. Their postattitudes were classified as ambivalent and positive. Two participants who had a positive attitude toward the VT expressed a positive behavior use.
Collapse
Affiliation(s)
- Antonia Arnaert
- School of Nursing, McGill University, Montreal, Quebec, Canada.
| | | | | |
Collapse
|
20
|
Abstract
Videoconferencing (VC) technology has been used successfully to provide psychiatric services to patients in rural and otherwise underserved settings. VC-based diagnostic interviewing has shown good agreement with conventional face-to-face diagnosis of dementia in several investigations, but extension of this technology to neurocognitive assessment has received little attention. To this end, the authors administered a brief battery of common neuropsychological tests via VC technology (telecognitive) and traditional face-to-face methods to 14 older persons with mild cognitive impairment (MCI) and 19 persons with mild to moderate Alzheimer's disease (AD). Highly similar test scores were obtained when participants were tested in-person or via VC. Telecognitive assessment appears to be a valid means to conduct neuropsychological evaluation of older adults with cognitive impairment. Furthermore, continued development of VC technology has implications for expanding neuropsychological assessment options in under-served populations.
Collapse
Affiliation(s)
- C Munro Cullum
- University of Texas Southwestern Medical Center at Dallas, Dallas, TX, USA.
| | | | | | | |
Collapse
|
21
|
Hersh WR, Hickam DH, Severance SM, Dana TL, Pyle Krages K, Helfand M. Diagnosis, access and outcomes: Update of a systematic review of telemedicine services. J Telemed Telecare 2007; 12 Suppl 2:S3-31. [PMID: 16989671 DOI: 10.1258/135763306778393117] [Citation(s) in RCA: 96] [Impact Index Per Article: 5.6] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/17/2022]
Abstract
Telemedicine services are being increasingly used. Although insurers and other payers are covering some services in the USA, the rationale for these coverage decisions is not always evidence-based. We reviewed the literature for telemedicine services that substitute for face-to-face medical diagnosis and treatment. We focused on three types of telemedicine services: store-and-forward, home-based and office/hospital-based services. Studies were included if they were relevant to at least one of the three study areas, addressed at least one key question and contained reported results. We excluded articles that did not study a service requiring face-to-face encounters (i.e. teleradiology was excluded). Our search initially identified 4083 citations. After review, 597 were judged to be potentially relevant at the title/abstract level. Following a full-text review, 106 studies were included. Store-and-forward services have been studied in many specialties, the most common being dermatology, wound care and ophthalmology. The evidence for their efficacy is mixed. Several limited studies showed the benefits of home-based telemedicine interventions in chronic diseases. Studies of office/hospital-based telemedicine suggest that telemedicine is most effective for verbal interactions, e.g. videoconferencing for diagnosis and treatment in specialties like neurology and psychiatry. There are still significant gaps in the evidence base between where telemedicine is used and where its use is supported by high-quality evidence. Further well-designed research is necessary to understand how best to deploy telemedicine services in health care.
Collapse
Affiliation(s)
- William R Hersh
- Department of Medical Informatics and Clinical Epidemiology, Oregon Health and Science University, Portland, Oregon 97239, USA.
| | | | | | | | | | | |
Collapse
|
22
|
Kobak KA, Kane JM, Thase ME, Nierenberg AA. Why do clinical trials fail? The problem of measurement error in clinical trials: time to test new paradigms? J Clin Psychopharmacol 2007; 27:1-5. [PMID: 17224705 DOI: 10.1097/jcp.0b013e31802eb4b7] [Citation(s) in RCA: 90] [Impact Index Per Article: 5.3] [Reference Citation Analysis] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 02/07/2023]
|
23
|
Oliver DP, Demiris G, Hensel B. A Promising Technology to Reduce Social Isolation of Nursing Home Residents. J Nurs Care Qual 2006; 21:302-5. [PMID: 16985398 DOI: 10.1097/00001786-200610000-00005] [Citation(s) in RCA: 13] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/26/2022]
|
24
|
Williams TL, May CR, Esmail A. Limitations of patient satisfaction studies in telehealthcare: a systematic review of the literature. Telemed J E Health 2002; 7:293-316. [PMID: 11886667 DOI: 10.1089/15305620152814700] [Citation(s) in RCA: 151] [Impact Index Per Article: 6.9] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/16/2022] Open
Abstract
The objective of this study is to provide a systematic review of studies on patient satisfaction with telemedicine. The review included empirical studies that investigated patient satisfaction with that telemedicine service. The search strategy involved matching at least one of 11 'telemedicine' terms with one of 5 'satisfaction' terms. The following databases were searched: Telemedicine Information Exchange (TIE) database, MEDLINE, Science Citation Index (SCI), Social Science Citation Index (SSCI), Psycinfo, and Citation Index of Nursing and Allied Health (CINAHL). A highly structured instrument was used for data extraction. The review included 93 studies. Telepsychiatry represents the largest portion of these studies (25%), followed by multispecialty care (14%), nursing (11%), and dermatology (8%). Real-time videoconferencing was used in 88% of these studies. Only 19 (20%) included an independent control group, including 9 (10%) randomized control trial (RCT) studies. One third of studies were based on samples of less than 20 patients, and only 21% had samples of over 100 patients. Aspects of patient satisfaction most commonly assessed were: professional-patient interaction, the patient's feeling about the consultation, and technical aspects of the consultation. Only 33% of the studies included a measure of preference between telemedicine and face-to-face consultation. Almost half the studies measured only 1 or 2 dimensions of satisfaction. Reported levels of satisfaction with telemedicine are consistently greater than 80%, and frequently reported at 100%. Progression of telemedicine services from "trial" status to routine health service must be supported by improved research into patients' satisfaction with telemedicine. Further investigation of factors that influence patient acceptance of telemedicine is indicated.
Collapse
Affiliation(s)
- T L Williams
- School of Primary Care, University of Manchester, United Kingdom.
| | | | | |
Collapse
|
25
|
Current awareness in geriatric psychiatry. Int J Geriatr Psychiatry 2001. [PMID: 11607950 DOI: 10.1002/gps.378] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 11/07/2022]
|