An empirical evaluation of a novel domain-specific language - modelling vehicle routing problems with Athos

Domain-specific languages (DSLs) are a popular approach among software engineers who demand for a tailored development interface. A DSL-based approach allows to encapsulate the intricacies of the target platform in transformations that turn DSL models into executable software code. Often, DSLs are even claimed to reduce development complexity to a level that allows them to be successfully applied by domain-experts with limited programming knowledge. Recent research has produced some scientifically backed insights on the benefits and limitations of DSLs. Further empirical studies are required to build a sufficient body of knowledge from which support for different claims related to DSLs can be derived. In this research study, we adopt current DSL evaluation approaches to investigate potential gains in terms of effectiveness and efficiency, through the application of our DSL Athos, a language developed for the domain of traffic and transportation simulation and optimisation. We compare Athos to the alternative of using an application library defined within a general-purpose language (GPL). We specified two sets of structurally identical tasks from the domain of vehicle routing problems and asked study groups with differing levels of programming knowledge to solve the tasks with the two approaches. The results show that inexperienced participants achieved considerable gains in effectiveness and efficiency with the usage of Athos DSL. Though hinting at Athos being the more efficient approach, the results were less distinct for more experienced programmers. The vast majority of participants stated to prefer working with Athos over the usage of the presented GPL's API.

SUPPLEMENTARY INFORMATION

The online version contains supplementary material available at 10.1007/s10664-022-10210-whttps://doi.org/10.1007/s10664-022-10210-w.

Empirical evaluation of an architectural technical debt index in the context of the Apache and ONAP ecosystems

BACKGROUND

Architectural Technical Debt (ATD) in a software-intensive system denotes architectural design choices which, while being suitable or even optimal when adopted, lower the maintainability and evolvability of the system in the long term, hindering future development activities. Despite the growing research interest in ATD, how to gain an informative and encompassing viewpoint of the ATD present in a software-intensive system is still an open problem.

OBJECTIVE

In this study, we evaluate ATDx, a data-driven approach providing an overview of the ATD present in a software-intensive system. The approach, based on the analysis of a software portfolio, calculates severity levels of architectural rule violations via a clustering algorithm, and aggregates results into different ATD dimensions.

METHOD

To evaluate ATDx, we implement an instance of the approach based on SonarQube, and run the analysis on the Apache and ONAP ecosystems. The analysis results are then shared with the portfolio contributors, who are invited to participate in an online survey designed to evaluate the representativeness and actionability of the approach.

RESULTS

The survey results confirm the representativeness of the ATDx, in terms of both the ATDx analysis results and the used architectural technical debt dimensions. Results also showed the actionability of the approach, although to a lower extent when compared to the ATDx representativeness, with usage scenarios including refactoring, code review, communication, and ATD evolution analysis.

CONCLUSIONS

With ATDx, we strive for the establishment of a sound, comprehensive, and intuitive architectural view of the ATD identifiable via source code analysis. The collected results are promising, and display both the representativeness and actionability of the approach. As future work, we plan to consolidate the approach via further empirical experimentation, by considering other development contexts (e.g., proprietary portfolios and other source code analysis tools), and enhancing the ATDx report capabilities.

A roadmap to neural automatic post-editing: an empirical approach

In a translation workflow, machine translation (MT) is almost always followed by a human post-editing step, where the raw MT output is corrected to meet required quality standards. To reduce the number of errors human translators need to correct, automatic post-editing (APE) methods have been developed and deployed in such workflows. With the advances in deep learning, neural APE (NPE) systems have outranked more traditional, statistical, ones. However, the plethora of options, variables and settings, as well as the relation between NPE performance and train/test data makes it difficult to select the most suitable approach for a given use case. In this article, we systematically analyse these different parameters with respect to NPE performance. We build an NPE "roadmap" to trace the different decision points and train a set of systems selecting different options through the roadmap. We also propose a novel approach for APE with data augmentation. We then analyse the performance of 15 of these systems and identify the best ones. In fact, the best systems are the ones that follow the newly-proposed method. The work presented in this article follows from a collaborative project between Microsoft and the ADAPT centre. The data provided by Microsoft originates from phrase-based statistical MT (PBSMT) systems employed in production. All tested NPE systems significantly increase the translation quality, proving the effectiveness of neural post-editing in the context of a commercial translation workflow that leverages PBSMT.

Placebo run-in periods in anticholinergic trials are not associated with treatment effect size or risk of attrition: an empirical evaluation

OBJECTIVES

We aimed to explore the impact of run-in periods on the magnitude of treatment effect and the risk of attrition in a sample of randomized trials.

STUDY DESIGN AND SETTING

We identified randomized trials from a published systematic review examining the effects of anticholinergics for the treatment of overactive bladders. We fitted meta-analytic mixed-effects models to assess whether the type of run-in (placebo run-in vs. no run-in) was associated with the magnitude of the effect estimates for the following outcomes: the number of voids per day, number of leakages per day, presence of dry mouth, cure/improvement, patient withdrawal from the trial, compliance with the trial protocol, and/or adherence to study drug. We adjusted for potential confounders.

RESULTS

A total of 96 trials met the eligibility criteria; 59 trials had no run-in (which included those with a screening or withdrawal period), 37 trials had a placebo run-in, and no trials had a drug run-in. The magnitude of the effect estimates for all outcomes did not importantly differ between trials with a placebo run-in and trials with no run-in. Adjustment for the confounding variables did not materially change the estimates.

CONCLUSION

The hypothesized benefits of placebo run-in periods were not observed in our sample of anticholinergic randomized trials for the treatment of overactive bladders. Designing future trials of anticholinergics with more pragmatic intentions is likely to result in evidence that more directly informs clinical decision-making.

An investigation of the impact of using different methods for network meta-analysis: a protocol for an empirical evaluation

BACKGROUND

Network meta-analysis, a method to synthesise evidence from multiple treatments, has increased in popularity in the past decade. Two broad approaches are available to synthesise data across networks, namely, arm- and contrast-synthesis models, with a range of models that can be fitted within each. There has been recent debate about the validity of the arm-synthesis models, but to date, there has been limited empirical evaluation comparing results using the methods applied to a large number of networks. We aim to address this gap through the re-analysis of a large cohort of published networks of interventions using a range of network meta-analysis methods.

METHODS

We will include a subset of networks from a database of network meta-analyses of randomised trials that have been identified and curated from the published literature. The subset of networks will include those where the primary outcome is binary, the number of events and participants are reported for each direct comparison, and there is no evidence of inconsistency in the network. We will re-analyse the networks using three contrast-synthesis methods and two arm-synthesis methods. We will compare the estimated treatment effects, their standard errors, treatment hierarchy based on the surface under the cumulative ranking (SUCRA) curve, the SUCRA value, and the between-trial heterogeneity variance across the network meta-analysis methods. We will investigate whether differences in the results are affected by network characteristics and baseline risk.

DISCUSSION

The results of this study will inform whether, in practice, the choice of network meta-analysis method matters, and if it does, in what situations differences in the results between methods might arise. The results from this research might also inform future simulation studies.

Rasmussen and the boundaries of empirical evaluation

In this special issue, many of the papers focus on Rasmussen's analytic contributions to the understanding of work in complex sociotechnical systems. Work is analysed for the purpose of developing new designs that can improve the nature of that work. The evaluation of such designs was a key part of Rasmussen's program, yet he was often sceptical of the claims made for the generalizability of empirical studies. To tackle this problem, he extended his work analysis framework to provide a way of thinking about empirical evaluation. As authors of this paper, we come from two different backgrounds-systems engineering in the case of Burns, and engineering psychology in the case of Sanderson-and over the decades of our respective research programs, we have both performed many empirical investigations: field investigations, simulation studies, and behavioural laboratory experiments. Rasmussen's scepticism-and his writings on the issue-have stimulated and shaped our own research. In this brief paper we present our interpretation of Rasmussen's perspective, we provide examples how our research sits within Rasmussen's framework of constraints defining boundary conditions for experiments, and we draw conclusions for the future.

"Almost-stable" matchings in the Hospitals / Residents problem with Couples

The Hospitals / Residents problem with Couples (hrc) models the allocation of intending junior doctors to hospitals where couples are allowed to submit joint preference lists over pairs of (typically geographically close) hospitals. It is known that a stable matching need not exist, so we consider min bp hrc, the problem of finding a matching that admits the minimum number of blocking pairs (i.e., is "as stable as possible"). We show that this problem is NP-hard and difficult to approximate even in the highly restricted case that each couple finds only one hospital pair acceptable. However if we further assume that the preference list of each single resident and hospital is of length at most 2, we give a polynomial-time algorithm for this case. We then present the first Integer Programming (IP) and Constraint Programming (CP) models for min bp hrc. Finally, we discuss an empirical evaluation of these models applied to randomly-generated instances of min bp hrc. We find that on average, the CP model is about 1.15 times faster than the IP model, and when presolving is applied to the CP model, it is on average 8.14 times faster. We further observe that the number of blocking pairs admitted by a solution is very small, i.e., usually at most 1, and never more than 2, for the (28,000) instances considered.