Evaluation in design science: A framework to support project studies in the context of University Research Centres

This paper proposes a discussion of the evaluation of an artefact developed under the Design Science paradigm using the Delphi method. It evaluates the Ecosystem framework of University Research Centres in Project Studies, considering a set of criteria pre-established in the literature. The Delphi method is an evaluation implemented in an electronic platform involving twenty-one participants, among whom were academics, practitioners, and PhD candidates in the field of project management. It reached consensus and stability in two rounds: the results indicate a consensus among the participants in the applicability, novelty, simplicity, completeness, fidelity to modelled phenomena, consistency and internal coherence, scalability, flexibility, interest, elegance, and reusability criteria. Usability was the only criterion that did not attain the predefined percentage of consensus among the participants (70%). Given the framework's characteristics, Delphi participants indicated the need to produce complementary guidelines for its implementation.

A case-based reasoning tool to recommend drinking water source protection actions

In response to a need expressed by water actors to support them in identifying appropriate actions for protecting drinking water sources, we modelled, designed, implemented, tested, and validated a prototype case-based reasoning (CBR) system. This required an intensive knowledge acquisition and structuring process which we conducted in collaboration with 102 water management and governance actors in Quebec, Canada. Knowledge was organized in a case base containing nearly 200 past experiences implementing water protection actions at different decision-making scales (local, regional, provincial), by various organizations (government, municipalities, water-related associations), over the last two decades. This paper describes our case engineering process to design case attributes based on content analysis. It presents the case edition, the case retrieval process, and the prototype's implementation and validation, conducted through a rigorous and transparent participatory procedure. This is, to our knowledge, the first successful attempt to apply a CBR approach to support water actors in protecting drinking water sources. It provides empirical evidence of the positive potential of such an approach for knowledge sharing and transfer and can be generalized to other similar contexts.

Participatory design research for the development of real-time simulation models in healthcare

The implementation challenges for modelling and simulation in health and social care are well-known and understood. Yet increasing availability of data and a better understanding of the value of Operational Research (OR) applications are strengthening opportunities to support healthcare delivery. Participative approaches in healthcare modelling have shown value through stakeholder engagement and commitment towards co-creation of models and knowledge but are limited in focus on model design and development. For simulation modelling, a participative design research methodology can support development for sustained use, emphasising model usefulness and usability using iterative cycles of development and evaluation. Within a structured methodology, measures of success are built into the design process, focusing on factors which contribute to success, with implicit goals of implementation and improvement. We illustrate this through a participative case study which demonstrates development of the component parts of a real-time simulation model aimed at reducing emergency department crowding.

An investigation on trust in AI-enabled collaboration: Application of AI-Driven chatbot in accommodation-based sharing economy

Several measures taken to control the spread of the COVID-19 pandemic have severely disrupted the accommodation sharing sector. This study attempts to find solutions to aid the recovery of the accommodation sharing sector via team efforts. Accordingly, we focus on the integration of artificial intelligence (AI) and collaboration. Despite the significant developments in AI technologies, there exists no research considering the application of AI in team collaboration. Utilizing the design science research method and collaboration engineering, we developed an AI-driven prototype system, AI-Driven, for collaboration process recommendation. Qualitative results show that the newly developed tool for collaboration process recommendation has achieved satisfactory performance. Furthermore, we investigated the antecedents and outcomes of trust in the AI-driven collaboration context. From a practical perspective, we propose several solutions to the challenges looming over the accommodation sharing sector according to collaboration deliverables. Furthermore, a system prototype was developed to facilitate collaboration process recommendation and provide procedural guidance.

Developing and validating high-value patient digital follow-up services: a pilot study in cardiac surgery

Background

The existing digital healthcare solutions demand a service development approach that assesses needs, experience, and outcomes, to develop high-value digital healthcare services. The objective of this study was to develop a digital transformation of the patients’ follow-up service after cardiac surgery, based on a remote patient monitoring service that would respond to the real context challenges.

Methods

The study followed the Design Science Research methodology framework and incorporated concepts from the Lean startup method to start designing a minimal viable product (MVP) from the available resources. The service was implemented in a pilot study with 29 patients in 4 iterative develop-test-learn cycles, with the engagement of developers, researchers, clinical teams, and patients.

Results

Patients reported outcomes daily for 30 days after surgery through Internet-of-Things (IoT) devices and a mobile app. The service’s evaluation considered experience, feasibility, and effectiveness. It generated high satisfaction and high adherence among users, fewer readmissions, with an average of 7 ± 4.5 clinical actions per patient, primarily due to abnormal systolic blood pressure or wound-related issues.

Conclusions

We propose a 6-step methodology to design and validate a high-value digital health care service based on collaborative learning, real-time development, iterative testing, and value assessment.

Using design science research to propose an IT governance model for higher education institutions

The increasing use of, and dependence on, Information Technology (IT) to support operational teaching, research, and management activities in Higher Education Institutions (HEI)-mainly due to their multi-unit organizational structure-have evidenced the need of encouraging managers to focus more on IT Governance (ITG) effectiveness, which has been an issue for many of these institutions. In this sense, we aim to develop a specific ITG model for Higher Education Institutions, by following the methodological principles of Design Science Research (DSR). The model was developed based on a robust theoretical basis that took into account different approaches substantiated by both the alignment of good practices and ITG focus areas as well as addressing the current ITG context of the Brazilian higher education institutions. The validation of the ITG model was based on its presentation to members of the IT Steering Committee of a Brazilian HEI, who confirmed its usefulness and applicability.

Retailer's Dual Role in Digital Marketplaces: Reference Architectures for Retail Information Systems

Digital marketplaces have entered the retail sector and have proven to be a successful business model compared to traditional retailing. Established retailers are increasingly launching digital marketplaces as well as participating in marketplaces of pure online companies. Retailers transforming to digital marketplaces orchestrate formerly independent markets and enable retail transactions between participants while simultaneously selling articles from their own assortment to customers in the digital marketplace (dual role). A retailer's dual role must be supported by retail information systems. However, this support is not explicitly represented in existing reference architectures for retail information systems. Thus, we propose to develop a reference architecture for retail information systems that facilitates the orchestration of supply- and demand-side participants, selling their own articles, and providing innovation platform services. We apply a design science research approach and present nine architectural requirements that a reference architecture for a multi-sided market business model in retail needs to fulfill (dual role, additional participants, affiliation, matchmaking, variety of services, innovation services, smart services, aggregated assortment, and boundary resources) from the rigor cycle. From the first design iteration, we propose four exemplary, conceptual architectural patterns as a solution for the requirements (matchmaking for participants, innovation platform services, boundary resources, and aggregated assortment). These patterns can form a conceptual reference architecture that guides the design and implementation of information systems.

Designing Personality-Adaptive Conversational Agents for Mental Health Care

Millions of people experience mental health issues each year, increasing the necessity for health-related services. One emerging technology with the potential to help address the resulting shortage in health care providers and other barriers to treatment access are conversational agents (CAs). CAs are software-based systems designed to interact with humans through natural language. However, CAs do not live up to their full potential yet because they are unable to capture dynamic human behavior to an adequate extent to provide responses tailored to users' personalities. To address this problem, we conducted a design science research (DSR) project to design personality-adaptive conversational agents (PACAs). Following an iterative and multi-step approach, we derive and formulate six design principles for PACAs for the domain of mental health care. The results of our evaluation with psychologists and psychiatrists suggest that PACAs can be a promising source of mental health support. With our design principles, we contribute to the body of design knowledge for CAs and provide guidance for practitioners who intend to design PACAs. Instantiating the principles may improve interaction with users who seek support for mental health issues.

The joint use of resilience engineering and lean production for work system design: A study in healthcare

Although lean production (LP) has been increasingly adopted in healthcare systems, its benefits often fall short of expectations. This might be partially due to the failure of lean to account for the complexity of healthcare. This paper discusses the joint use of principles of LP and resilience engineering (RE), which is an approach for system design inspired by complexity science. Thus, a framework for supporting the design of socio-technical systems, which combines insights from LP and RE, was developed and tested in a system involving a patient flow from an emergency department to an intensive care unit. Based on this empirical study, as well as on extant theory, eight design propositions that support the framework application were developed. Both the framework and its corresponding propositions can contribute to the design of socio-technical systems that are at the same time safe and efficient.

Developing an emergency department crowding dashboard: A design science approach

BACKGROUND

As an emergency department (ED) is a complex adaptive system, the analysis of continuously gathered data is valuable to gain insight in the real-time patient flow. To support the analysis and management of ED operations, relevant data should be provided in an intuitive way.

AIM

Within this context, this paper outlines the development of a dashboard which provides real-time information regarding ED crowding.

METHODS

The research project underlying this paper follows the principles of design science research, which involves the development and study of artifacts which aim to solve a generic problem. To determine the crowding indicators that are desired in the dashboard, a modified Delphi study is used. The dashboard is implemented using the open source Shinydashboard package in R.

RESULTS

A dashboard is developed containing the desired crowding indicators, together with general patient flow characteristics. It is demonstrated using a dataset of a Flemish ED and fulfills the requirements which are defined a priori.

CONCLUSIONS

The developed dashboard provides real-time information on ED crowding. This information enables ED staff to judge whether corrective actions are required in an effort to avoid the adverse effects of ED crowding.

Implementing an online pharmaceutical service using design science research

BACKGROUND

The rising prevalence of chronic diseases is pressing health systems to introduce reforms. Primary healthcare and multidisciplinary models have been suggested as approaches to deal with this challenge, with new roles for nurses and pharmacists being advocated. More recently, implementing healthcare based on information systems and technologies (e.g. eHealth) has been proposed as a way to improve health services. However, implementing online pharmaceutical services, including their adoption by pharmacists and patients, is still an open research question. In this paper we present ePharmacare, a new online pharmaceutical service implemented using Design Science Research.

METHODS

The Design Science Research Methodology (DSRM) was chosen to implement this online service for chronic diseases management. In the paper, DSRM's different activities are explained, from the definition of the problem to the evaluation of the artifact. During the design and development activities, surveys, observations, focus groups, and eye-tracking glasses were used to validate pharmacists' and patients' requirements. During the demonstration and evaluation activities the new service was used with real-world pharmacists and patients.

RESULTS

The results show the contribution of DSRM in the implementation of online services for pharmacies. We found that pharmacists spend only 50% of their time interacting with patients, uncovering a clear opportunity to implement online pharmaceutical care services. On the other hand, patients that regularly visit the same pharmacy recognize the value in patient follow-up demanding to use channels such as the Internet for their pharmacy interactions. Limitations were identified regarding the high workload of pharmacists, but particularly their lack of know-how and experience in dealing with information systems (IST) for the provision of pharmaceutical services.

CONCLUSIONS

This paper summarizes a research project in which an online pharmaceutical service was proposed, designed, developed, demonstrated and evaluated using DSRM. The main barriers for pharmacists' adoption of online pharmaceutical services provision were the lack of time, time management and information systems usage skills, as well as a precise role definition within pharmacies. These problems can be addressed with proper training and services reorganization, two proposals to be investigated in future works.

How can information systems provide support to nurses' hand hygiene performance? Using gamification and indoor location to improve hand hygiene awareness and reduce hospital infections

Background

Hospital-acquired infections are still amongst the major problems health systems are facing. Their occurrence can lead to higher morbidity and mortality rates, increased length of hospital stay, and higher costs for both hospital and patients. Performing hand hygiene is a simple and inexpensive prevention measure, but healthcare workers’ compliance with it is often far from ideal. To raise awareness regarding hand hygiene compliance, individual behaviour change and performance optimization, we aimed to develop a gamification solution that collects data and provides real-time feedback accurately in a fun and engaging way.

Methods

A Design Science Research Methodology (DSRM) was used to conduct this work. DSRM is useful to study the link between research and professional practices by designing, implementing and evaluating artifacts that address a specific need. It follows a development cycle (or iteration) composed by six activities. Two work iterations were performed applying gamification components, each using a different indoor location technology. Preliminary experiments, simulations and field studies were performed in an Intensive Care Unit (ICU) of a Portuguese tertiary hospital. Nurses working on this ICU were in a focus group during the research, participating in several sessions across the implementation process.

Results

Nurses enjoyed the concept and considered that it allows for a unique opportunity to receive feedback regarding their performance. Tests performed on the indoor location technology applied in the first iteration regarding distances estimation presented an unacceptable lack of accuracy. Using a proximity-based technique, it was possible to identify the sequence of positions, but beacons presented an unstable behaviour. In the second work iteration, a different indoor location technology was explored but it did not work properly, so there was no chance of testing the solution as a whole (gamification application included).

Conclusions

Combining automated monitoring systems with gamification seems to be an innovative and promising approach, based on the already achieved results. Involving nurses in the project since the beginning allowed to align the solution with their needs. Despite strong evolution through recent years, indoor location technologies are still not ready to be applied in the healthcare field with nursing wards.

Electronic supplementary material

The online version of this article (doi:10.1186/s12911-017-0410-z) contains supplementary material, which is available to authorized users.

The design of scenario-based training from the resilience engineering perspective: a study with grid electricians

Although scenario-based training (SBT) can be an effective means to help workers develop resilience skills, it has not yet been analyzed from the resilience engineering (RE) perspective. This study introduces a five-stage method for designing SBT from the RE view: (a) identification of resilience skills, work constraints and actions for re-designing the socio-technical system; (b) design of template scenarios, allowing the simulation of the work constraints and the use of resilience skills; (c) design of the simulation protocol, which includes briefing, simulation and debriefing; (d) implementation of both scenarios and simulation protocol; and (e) evaluation of the scenarios and simulation protocol. It is reported how the method was applied in an electricity distribution company, in order to train grid electricians. The study was framed as an application of design science research, and five research outputs are discussed: method, constructs, model of the relationships among constructs, instantiations of the method, and theory building. Concerning the last output, the operationalization of the RE perspective on three elements of SBT is presented: identification of training objectives; scenario design; and debriefing.

The development of a new methodology for knowledge sharing in the interface between university and society--an example from the meat sector

Design science research was used for the generation, use and evaluation of a model for knowledge sharing in the user community through open educational resources (OER). The focus of interest was on the development process of a model for knowledge sharing that emphasizes the characteristics and the needs of the user community; the empowerment and democratic issues of openness; the collaboration between institutions and dialog with society; and the consideration of quality and sustainability issues. Initially, the community needs were analyzed through surveys and workshops, and the findings used, through negotiations, to formulate the development process. An open-training platform served as an infrastructure and included a repository with OER, a wiki and a discussion forum. The purpose of this article is an attempt to provide universities with a plan and template for integrated knowledge sharing that responds to societal needs. Usability and usefulness has not been evaluated.