Applying Website Rankings to Digital Health Centers in the United States to Assess Public Engagement: Website Usability Study

Digital Impact Factor: A Quality Index for Educational Blogs and Podcasts in Emergency Medicine and Critical Care

STUDY OBJECTIVE

Given the popularity of educational blogs and podcasts in medicine, learners and educators need tools to identify trusted and impactful sites. The Social Media Index was a multi-sourced formula to rank the effect of emergency medicine and critical care blogs. In 2022, a key data point for the Social Media Index became unavailable. This bibliometric study aimed to develop a new measure, the Digital Impact Factor, as a replacement.

METHODS

The Digital Impact Factor incorporated modern measures of website authority and reach. This formula was applied to a cross-sectional study of active emergency medicine and critical care blogs and podcasts. For each website, we generated a Digital Impact Factor score based on Ahrefs Domain Rating and the follower count of the websites' pages from 8 social media platforms. A series of Spearman correlations provided evidence of association by comparing a rank-ordered list to rank lists derived from the Social Media Index over the last 5 years. The Bland-Altman analysis assessed for agreement.

RESULTS

The authors identified 88 relevant websites with a median Ahrefs Domain Rating of 28 (range 0 to 71, maximum 100) and total social media followership count across 8 platforms of 1,828,557. The Domain Rating and individual social media followership scores were normalized based on the highest recorded values to yield the Digital Impact Factor (median 4.57; range 0.02 to 9.50, maximum 10). The correlation between the 2022 Digital Impact Factor and the 2021 Social Media Index was 0.94 (95% confidence interval 0.89 to 0.97; p<.001; n=41 rankings correlated), suggesting that they measure similar constructs. The Bland-Altman plot also demonstrated fair agreement between the 2 scores.

CONCLUSION

The Digital Impact Factor is a measure of the relative effect of educational blogs and podcasts within emergency medicine and critical care.

Website Usability Analysis of U.S. Military Residency Programs

INTRODUCTION

The Military Match is the residency matching system for medical students attending the Uniformed Services University of Health Sciences, and the students were funded by the Health Professions Scholarship Program through the U.S. Army, Air Force, and Navy. To evaluate and compare military residency programs, students use residency program websites. Often, the residency program's website serves as a key source, or the only point of reference, when considering residency options, especially during times when face-to-face interactions are limited.This report aims to provide a systematic evaluation of military residency programs and their websites.

MATERIALS AND METHODS

Utilizing a previously published website usability scoring system, military residency programs were categorized to objectively and quantitatively analyze their websites. Usability was divided into four categories for quantifiable analysis: accessibility, marketing, content quality, and technology. The methodology for this analysis was replicated from published reports that have examined healthcare website usability. Each website was analyzed and scored in four categories: accessibility, content quality, marketing, and technology. A "General Usability" score was calculated for each website using a composite of the key factors within the four categories. An overall score was generated utilizing the weighted percentage across all four categories. To address deficiencies of the original methodology, a secondary analysis was performed on the listed websites utilizing an automated methodology for website usability.

RESULTS

A comprehensive list of 125 Accreditation Council for Graduate Medical Education U.S. Military residency program websites was compiled. Of these, 96 programs and 106 websites were evaluated. The primary analysis employing usability methodology identified technology as the highest ranked category with a score of 0.749 (SD ± 0.039) (SE 0.005) (P < .05). Marketing and content quality were the lowest scoring categories with mean scores of 0.414 (SD ± 0.054) (SE 0.006) and 0.428 (SD ± 0.229) (SE 0.027), respectively (P < .05). There was no significant difference in overall usability rankings or scores among the 96 residency program websites across the three branches (P < .05).Secondary analysis with the new usability methodology demonstrated military residency websites to exhibit more external backlinking compared to internal backlinking (P < 0.05) and no social media backlinking to any of the 106 analyzed websites. When comparing the three services, the Army had significantly lower external backlinking ranking 43.4 (P < .05) and overall backlinking ranking 56.4 (P < 0.05) when compared to the Navy (mean 48.8 and 71.7, and 43.4). There were no other differences in backlinking rankings across the three branches.

CONCLUSIONS

Residency websites have become a primary way to communicate information to applicants. By assessing the overall usability of the various military residency websites, we determined the effectiveness of these websites to relay information to prospective students interested in applying for military residency. We predict that by improving website accessibility, residency programs increase their effectiveness at communicating information to potential applicants and increase interest in military residency programs.

Methodological Guidelines for Systematic Assessments of Health Care Websites Using Web Analytics: Tutorial

With the growing importance of communicating with the public via the web, many industries have used web analytics to provide information that organizations can use to better achieve their goals. Although the importance of health care websites has also grown, the health care industry has been slower to adopt the use of web analytics. Web analytics are the measurement, collection, analysis, and reporting of internet data used to measure direct user interaction. Our objective is to provide generalized methods for using web analytics as key performance metrics to evaluate websites and outline actionable recommendations for improvement. By deconstructing web analytic categories such as engagement, users, acquisition, content, and platform, we describe how web analytics are used to evaluate websites and how improvements can be made using this information. Engagement is how a user interacts with a website. It can be evaluated using the daily active users to monthly active users (DAU/MAU) ratio, bounce rate, pages viewed, and time on site. Poor engagement indicates potential problems with website usability. Users pertains to demographic information regarding the users interacting with a website. This data can help administrators understand who is engaging with their website. Acquisition refers to the overall website traffic and the method of traffic, which allows administrators to see how people are accessing their website. This information helps websites expand their methods of attracting users. Content refers to the overall relevancy, accuracy, and trustworthiness of a website’s content. If a website has poor content, it will likely experience difficulty with user engagement. Finally, platform refers to the technical aspects of how people access a website. It includes both the internet browsers and devices used. By providing detailed descriptions of these categories, we have identified how web administrators can use web analytics to systematically assess their websites. We have also provided generalized recommendations for actionable improvements. By introducing the potential of web analytics to augment usability and the conversion rate, we hope to assist health care organizations in better communicating with the public and therefore accomplishing the goals of their websites.

Website usability analysis of American anesthesiology residency programs: Original Paper (Preprint)

Background

The Association of American Medical Colleges has recently issued recommendations for the upcoming 2022-2023 application cycle that residency programs should conduct all interviews for this upcoming application cycle over the web. In light of these recommendations, many students will have limited exposure to anesthesiology programs and will rely on information gleaned digitally. This change means that the aspects of program websites used to provide information, such as size, structure, location, requirements, and contact information, will be crucial in helping prospective residents decide where and how to apply in the future. An evaluation of website usability, which includes initial appearance along with factors that influence its ease of navigation and convenience of use, can thus be applied to anesthesiology residency websites. Areas of need can be targeted to increase web presence and provide effective pathways to exhibit the different attributes of their programs to future applicants.

Objective

This study aimed to compile a list of US anesthesiology residency programs and their websites while objectively analyzing the websites using a formally published usability scoring system, as well as to identify positive and negative trends to offer areas of improvement among anesthesiology residency websites.

Methods

We included only 114 US anesthesiology residency program websites in our sample set, since some websites we analyzed showed errors or inconclusive. Website usability was separated into 4 distinct categories for analysis based on methodology outlined in previous literature on both health care website usability and residency website usability. The 4 categories were Accessibility, Marketing, Content Quality, and Technology. Each website was then analyzed and scored based on key components highlighted within the 4 categories. The multiple factors were then graded using a percentage system to create a comprehensive score for each program.

Results

The highest scoring category was Content Quality (mean 4.7, SD 2.48, SE 0.23). The lowest scoring category was Technology (mean 0.9, SD 0.38, SE 0.04).

Conclusions

Through the application of a health care website usability framework, multiple anesthesiology residency programs were analyzed and scored in the areas of Accessibility, Marketing, Content Quality, and Technology, which allowed us to determine the effectiveness of the usability of these websites to convey information to their end user. Websites must communicate vital information, with usability at the forefront, to continue to grow, especially as the United States faces challenges due to the COVID-19 pandemic. Our recommendation is that anesthesiology programs should strive to improve website usability to increase the ease by which applicants can collect vital information about anesthesiology programs. A few proposed solutions include making changes such as decreasing error pages on websites, migrating away from using in-line cascading style sheets, and improving web page loading speeds to improve the Technology category.

Analyzing opioid-use disorder websites in the United States: An optimized website usability study

Background

As the United States continues to tackle the opioid epidemic, it is imperative for digital healthcare organizations to provide Internet users with accurate and accessible online resources so that they can make informed decisions with regards to their health.

Objective

The primary objectives were to adapt and modify a previously established usability methodology from literature, apply this modified methodology in order to perform usability analysis of opioid-use-disorder (OUD)-related websites, and make important recommendations that OUD-related digital health organizations may utilize to improve their online presence.

Methods

A list of 208 websites (later refined) was generated for usability testing using a modified Google Search methodology. Four keywords were chosen and used in the search: “DEA-X Waiver Training”, “opioid-use-disorder (OUD) Initiatives”, “Buprenorphine Assisted Treatment”, and “Opioid-Use Disorder Websites”. Usability analysis was performed concurrently with optimization of the methodology. OUD websites were analyzed and scored on several usability categories established by previous literature.

Results

“DEA-X Waiver Training” yielded websites that scored the highest average in “Accessibility” (0.84), while “Opioid-Use Disorder Websites” yielded websites that scored the highest average in “Content Quality” (0.67). “Buprenorphine Assisted Treatment” yielded websites that scored the highest average across “Marketing” (0.52), “Technology” (0.89), “General Usability” (0.69), and “Overall Usability” (0.68). “Technology” and “Marketing” were the highest and lowest scoring usability categories, respectively. T-test analysis revealed that each usability, except “Marketing” had a pair of one or more keywords that were significantly different with a p-value that was equal to or less than 0.05.

Conclusions

Based on the study findings, we recommend that digital organizations in the OUD space should improve their “General Usability” score by making their websites easier to find online. Doing so, may allow users, especially individuals in the OUD space, to discover accurate information that they are seeking. Based on the study findings, we also made important recommendations that OUD-related digital organizations may utilize in order to improve website usability as well as overall reach.

An Analysis of US Academic Medical Center Websites: Usability Study

Background

Health care organizations are tasked with providing web-based health resources and information. Usability refers to the ease of user experience on a website. In this study, we conducted a usability analysis of academic medical centers in the United States, which, to the best of our knowledge, has not been previously carried out.

Objective

The primary aims of the study were to the following: (1) adapt a preexisting usability scoring methodology to academic medical centers; (2) apply and test this methodology on a sample set of academic medical center websites; and (3) make recommendations from these results on potential areas of improvements for our sample of academic medical center websites.

Methods

All website usability testing took place from June 1, 2020, to December 15, 2020. We replicated a methodology developed in previous literature and applied it to academic medical centers. Our sample included 73 US academic medical centers. Usability was split into four broad categories: accessibility (the ability of those with low levels of computer literacy to access and navigate the hospital’s website); marketing (the ability of websites to be found through search engines and the relevance of descriptions to the links provided); content quality (grammar, frequency of information updates, material relevancy, and readability); and technology (download speed, quality of the programming code, and website infrastructure). Using these tools, we scored each website in each category. The composite of key factors in each category contributed to an overall “general usability” score for each website. An overall score was then calculated by applying a weighted percentage across all factors and was used for the final “overall usability” ranking.

Results

The category with the highest average score was technology, with a 0.82 (SD 0.068, SE 0.008). The lowest-performing category was content quality, with an average of 0.22 (SD 0.069, SE 0.008). As these numbers reflect weighted percentages as an integer, the higher the score, the greater the overall usability in that category.

Conclusions

Our data suggest that technology, on average, was the highest-scored variable among academic medical center websites. Because website functionality is essential to a user’s experience, it is justified that academic medical centers invest in optimal website performance. The overall lowest-scored variable was content quality. A potential reason for this may be that academic medical center websites are usually larger in size, making it difficult to monitor the increased quantity of content. An easy way to improve this variable is to conduct more frequent website audits to assess readability, grammar, and relevance. Marketing is another area in which these organizations have potential for improvement. Our recommendation is that organizations utilize search engine optimization techniques to improve their online visibility and discoverability.