Metrics reloaded: recommendations for image analysis validation

Increasing evidence shows that flaws in machine learning (ML) algorithm validation are an underestimated global problem. In biomedical image analysis, chosen performance metrics often do not reflect the domain interest, and thus fail to adequately measure scientific progress and hinder translation of ML techniques into practice. To overcome this, we created Metrics Reloaded, a comprehensive framework guiding researchers in the problem-aware selection of metrics. Developed by a large international consortium in a multistage Delphi process, it is based on the novel concept of a problem fingerprint-a structured representation of the given problem that captures all aspects that are relevant for metric selection, from the domain interest to the properties of the target structure(s), dataset and algorithm output. On the basis of the problem fingerprint, users are guided through the process of choosing and applying appropriate validation metrics while being made aware of potential pitfalls. Metrics Reloaded targets image analysis problems that can be interpreted as classification tasks at image, object or pixel level, namely image-level classification, object detection, semantic segmentation and instance segmentation tasks. To improve the user experience, we implemented the framework in the Metrics Reloaded online tool. Following the convergence of ML methodology across application domains, Metrics Reloaded fosters the convergence of validation methodology. Its applicability is demonstrated for various biomedical use cases.

Federated benchmarking of medical artificial intelligence with MedPerf

Medical artificial intelligence (AI) has tremendous potential to advance healthcare by supporting and contributing to the evidence-based practice of medicine, personalizing patient treatment, reducing costs, and improving both healthcare provider and patient experience. Unlocking this potential requires systematic, quantitative evaluation of the performance of medical AI models on large-scale, heterogeneous data capturing diverse patient populations. Here, to meet this need, we introduce MedPerf, an open platform for benchmarking AI models in the medical domain. MedPerf focuses on enabling federated evaluation of AI models, by securely distributing them to different facilities, such as healthcare organizations. This process of bringing the model to the data empowers each facility to assess and verify the performance of AI models in an efficient and human-supervised process, while prioritizing privacy. We describe the current challenges healthcare and AI communities face, the need for an open platform, the design philosophy of MedPerf, its current implementation status and real-world deployment, our roadmap and, importantly, the use of MedPerf with multiple international institutions within cloud-based technology and on-premises scenarios. Finally, we welcome new contributions by researchers and organizations to further strengthen MedPerf as an open benchmarking platform.

Musculoskeletal Injuries and Outcomes Pre- and Post- Emergency Medicine Training Program

Introduction

Musculoskeletal injuries (MSI) comprise a large portion of the trauma burden in low- and middle-income countries (LMIC). Rwanda recently launched its first emergency medicine training program (EMTP) at the University Teaching Hospital-Kigali (UTH-K), which may help to treat such injuries; yet no current epidemiological data is available on MSI in Rwanda.

Methods

We conducted this pre-post study during two data collection periods at the UTH-K from November 2012 to July 2016. Data collection for MSI is limited and thus is specific to fractures. We included all patients with open, closed, or mixed fractures, hereafter referred to as MSI. Gathered information included demographics and outcomes including death, traumatic complications, and length of hospital stay, before and after the implementation of the EMTP.

Results

We collected data from 3609 patients. Of those records, 691 patients were treated for fractures, and 674 of them had sufficient EMTP data measured for inclusion in the analysis of results (279 from pre-EMTP and 375 from post-EMTP). Patient demographics demonstrate that a majority of MSI cases are male (71.6% male vs 28.4% female) and young (64.3% below 35 years of age). Among mechanisms of injury, major causes included road traffic accidents (48.1%), falls (34.2%), and assault (6.0%). There was also an observed association between EMTP and trends of the three primary outcomes: a reduction of death in the emergency department (ED) from those with MSI by 89.9%, from 2.51% to 0.25% (p = 0.0077); a reduction in traumatic complications for MSI patients by 71.7%, from 3.58% to 1.01% (p = 0.0211); and a reduction in duration of stay in the ED among those with MSI by 52.7% or 2.81 days on average, from 5.33 to 2.52 days (p = 0.0437).

Conclusion

This study reveals the current epidemiology of MSI morbidity and mortality for a major Rwandan teaching hospital and the potential impacts of EM training implementation among those with MSI. Residency training programs such as EMTP appear capable of reducing mortality, complications, and ED length of stay among those with MSI caused by fractures. Such findings underscore the efficacy and importance of investments in educating the next generation of health professionals to combat prevalent MSI within their communities.

Abstract P2-09-08: Imprime PGG, a novel innate immune modulator, combined with pembrolizumab in a phase 2 multicenter, open label study in chemotherapy-resistant metastatic triple negative breast cancer (TNBC)

Background: CPI monotherapy provides substantial clinical benefit to patients (pts) in multiple cancers, yet response rates are limited (˜15-30%) and fails to benefit the majority. In these pts there is limited or no ongoing T cell-based immune response. Imprime PGG (Imprime), a novel beta glucan derived from Saccharomyces, may expand the clinical benefit of CPI therapy by stimulating an anti-cancer immune response. Acting as a pathogen-associated molecularpattern (PAMP), Imprime enlists innate immune functions including cytotoxic effector mechanisms, reversal of immunosuppression and cross-talk with the adaptive immune system.Imprime-mediated innate immune activation requires formation of an immune complex with naturally-occurring anti-beta glucan antibodies (ABA); sufficient ABA levels is required for complex formation. Imprime is now being studied in combination with pembrolizumab (KEYTRUDA®,Pembro), a humanized mAb against PD-1 which has been previously studied in TNBC pts.

Methods: In this study of patients who previously failed chemotherapy for metastatic TNBC, Imprime is being used in combination with Pembro in a Simon 2 stage design. Asample size of 12 evaluable pts in Stage 1 was planned.Evaluable pts received at least one dose of study treatment (tx), had measurable disease at baseline per RECIST v1.1, had at least one post-baseline scan or discontinued tx as a result of progressive disease, death, or a tx-related adverse event before the first post-baseline scan.Pts received Imprime (4 mg/kg IV days 1, 8, 15 of each 3-week cycle) + Pembro 200 mg on D1 of each cycle. Criteria to advance to Stage 2 were ≤4 grade 3/4 AEs during the first tx cycle (other than infusion reactions) and ≥1 objective response. Study primary endpoints are ORR and safety; secondary endpoints are TTR, CRR, DoR, PFS, and OS. Exploratory endpoints include ORR and PFS per irRECIST. Biopsies and blood samples are being collected to assess tx impact on immune activating events at the tumor site and in the periphery.

Results: A review of efficacy and safety data was conducted at the end of Stage 1. Thirteen pts (12 evaluable) were enrolled into Stage 1. Safety review noted 2 grade 3 adverse events that met protocol definition of Stage 1 events (1 pt: cellulitis and 1 pt: pleural infusion; both unrelated to treatment). Two events lead to 2 pts discontinuing treatment (infusion reaction and pancreatitis) and only 1 autoimmune event was observed (pancreatitis). Observed efficacy responses in the evaluable pts included 1 complete response (CR; ongoing) and 2 partial responses (PR; ongoing). Secondary efficacy endpoints have not been assessed. Early translational results support proposed MOA and analysis of Stage 1 translational data is ongoing.

Conclusion: The use of Imprime with Pembro was well tolerated and met both safety and efficacy requirements to move forward with Stage 2 of the study. No significant safety concerns were identified in Stage 1. Further investigation is thus warranted and enrollment into Stage 2 is ongoing. Updated data will be presented.

Citation Format: O'Day S, Borges V, Chmielowski B, Rao R, Abu-Khalaf M, Stopeck A, Lowe J, Mattson P, Breuer K, Gargano M, Bose N, Uhlik M, Graff J, Chisamore M, Cox J, Osterwalder B. Imprime PGG, a novel innate immune modulator, combined with pembrolizumab in a phase 2 multicenter, open label study in chemotherapy-resistant metastatic triple negative breast cancer (TNBC) [abstract]. In: Proceedings of the 2018 San Antonio Breast Cancer Symposium; 2018 Dec 4-8; San Antonio, TX. Philadelphia (PA): AACR; Cancer Res 2019;79(4 Suppl):Abstract nr P2-09-08.

A randomized, open-label, multicenter, phase II study evaluating the efficacy and safety of BTH1677 (1,3-1,6 beta glucan; Imprime PGG) in combination with cetuximab and chemotherapy in patients with advanced non-small cell lung cancer

Introduction BTH1677, a 1,3–1,6 beta-glucan immunomodulator, stimulates a coordinated anti-cancer immune response in combination with anti-tumor antibody therapies. This phase II study explored the efficacy, pharmacokinetics (PK), and safety of BTH1677 combined with cetuximab/carboplatin/paclitaxel in untreated stage IIIB/IV non-small cell lung cancer (NSCLC) patients. Methods Patients were randomized 2:1 to the BTH1677 arm (N=60; BTH1677, 4 mg/kg, weekly; cetuximab, initial dose 400 mg/m² and subsequent doses 250 mg/m², weekly; carboplatin, 6 mg/mL/min AUC (area-under-the-curve) by Calvert formula, once each 3-week cycle [Q3W]); and paclitaxel, 200 mg/m², Q3W) or Control arm (N=30; cetuximab/carboplatin/paclitaxel as above). Carboplatin/paclitaxel was discontinued after 4–6 cycles; patients who responded or remained stable received maintenance therapy with BTH1677/cetuximab (BTH1677 arm) or cetuximab (Control arm). Investigator and blinded central radiology reviews were conducted. Efficacy assessments included objective response rate (ORR; primary endpoint), disease control rate, duration of objective response, time-to-progression and overall survival (OS); safety was assessed by adverse events (AEs). Potential biomarker analysis for BTH1677 response was also conducted. Results Compared to control treatment, the addition of BTH1677 numerically increased ORR by both investigator (47.8% vs 23.1%; p=0.0468) and central (36.6% vs 23.1%; p=0.2895) reviews. No other endpoints differed between arms. PK was consistent with previous studies. BTH1677 was well tolerated, with AEs expected of the backbone therapy predominating. Biomarker-positive patients displayed better ORR and OS than negative patients. Conclusions BTH1677 combined with cetuximab/carboplatin/paclitaxel was well tolerated and improved ORR as first-line treatment in patients with advanced NSCLC. Future patient selection by biomarker status may further improve efficacy

ClinicalTrials.gov Identifier: NCT00874848

Memory access scheduling

The bandwidth and latency of a memory system are strongly dependent on the manner in which accesses interact with the “3-D” structure of banks, rows, and columns characteristic of contemporary DRAM chips. There is nearly an order of magnitude difference in bandwidth between successive references to different columns within a row and different rows within a bank. This paper introduces memory access scheduling, a technique that improves the performance of a memory system by reordering memory references to exploit locality within the 3-D memory structure. Conservative reordering, in which the first ready reference in a sequence is performed, improves bandwidth by 40% for traces from five media benchmarks. Aggressive reordering, in which operations are scheduled to optimize memory bandwidth, improves bandwidth by 93% for the same set of applications. Memory access scheduling is particularly important for media processors where it enables the processor to make the most efficient use of scarce memory bandwidth.

Microtubules, organelle transport, and steroidogenesis in cultured adrenocortical tumor cells. 2. Reversibility of taxol's inhibition of basal and ACTH-induced steroidogenesis is unaccompanied by reversibility of taxol-induced changes in cell ultrastructure

Taxol inhibits the basal and ACTH-stimulated steroidogenesis of cultured mouse adrenocortical tumor cells, presumably by preventing the arrival of cholesterol in mitochondria. In these cells, taxol polymerizes and rearranges microtubules, disperses SER masses, disrupts the Golgi, and impedes the formation of cholesterol-containing lysosomes. However, taxol's alterations in ultrastructure appear likely to permit both a microtubule-based organelle transport proposed to bring mitochondria of unstimulated cells close to alternate sources of cholesterol--the SER and lipid droplets--and postulated ACTH-caused increases in these encounters. Conceivably, taxol may prevent the transfer of cholesterol from the SER and lipid droplets to mitochondria, once the meetings are achieved. To investigate this possibility, we determined the reversibility of taxol's ultrastructural effects and inhibition of steroidogenesis. Primary cultured adrenal tumor cells were incubated for 4 hr with and without ACTH (10 mU/ml). with taxol (50 micrograms/ml), and with ACTH and taxol 50 simultaneously. Some cultures from each set were washed with fresh medium and re-incubated for 1.5 hr. with and without ACTH. Media taken from cultures at the ends of pre- and post-washout incubations were analyzed for the presence of secreted steroids. Sample cultures were fixed for electron microscopy at the ends of both incubations. Data derived from pre-washout incubations confirmed previous reports of taxol's ultrastructural changes and inhibition of steroidogenesis. When cells recovered from taxol in the absence of ACTH, the inhibition of steroidogenesis was completely reversed. In the presence of ACTH, ex-taxol-treated cells demonstrated a "rounding up' and an increased steroid production that are characteristic responses to the hormone. However, in all cases, there was a persistence of taxol's alterations in organelle numbers and arrangements. Our findings establish that the ultrastructural effects of taxol which we recorded cannot prevent mitochondria of unstimulated and ACTH-stimulated adrenal tumor cells from gaining cholesterol. They strengthened the possibility that in pre-washout incubations, taxol allowed organelle motility to bring mitochondria adjacent to cholesterol-containing SER tubules and lipid droplets, but inhibited steroidogenesis by preventing the cholesterol transfer. Taxol might limit the availability of a protein required for the transfer, an effect not visible in our electron micrographs.

Microtubules, organelle transport, and steroidogenesis in cultured adrenocortical tumor cells. 1. An ultrastructural analysis of cells in which basal and ACTH-induced steroidogenesis was inhibited by taxol

In adrenocortical cells, the first step in the enzymatic processing of cholesterol to steroid end products occurs in the mitochondria. ACTH increases mitochondrial cholesterol and steroidogenesis. In cultured mouse adrenocortical tumor cells, microtubule-based organelle motility may increase the proximity of mitochondria to the SER, lipid droplets and endoscome-derived lysosomes, thereby facilitating the transfer of cholesterol from these organelles to the mitochondrial outer membrane. ACTH may increase opportunities for the transfer by promoting organelle motility and by increasing the number of lysosomes. Taxol, a microtubule polymerizer, inhibits basal and ACTH-induced steroidogenesis in these cells, presumably at the step where mitochondria obtain cholesterol. We examined the ultrastructure of taxol-treated, unstimulated and ACTH-stimulated cells, seeking alterations which conceivably could interefer with the proposed organelle transport and encounters, and thus correlate with taxol's inhibition of steroidogenesis. Primary cultured cells were incubated in serum-containing medium for 4 hr with and without ACTH (10 mU/ml), with 10 micrograms/ml and 50 micrograms/ml of taxol, and with ACTH and taxol 10 or taxol 50 simultaneously. Culture media were analyzed for the presence of secreted steroids at the end of 1, 2, and 4 hr of incubation. At the end of the fourth hour, unstimulated cells and cells treated with ACTH, taxol 50, and both agents simultaneously, were fixed and processed for EM. Taxol inhibited basal and ACTH-induced steroidogenesis in a dose-dependent fashion. In both unstimulated and ACTH-stimulated cells, taxol 50 formed numerous microtubule bundles, but did not markedly change the distribution of mitochondria and lipid droplets. SER tubules, and clusters of Golgi fragments, endosomes, and lysosomes appeared to be translocated towards the cell periphery along some of the microtubules. Taxol permitted an ACTH-induced cell rounding and microfilament rearrangement considered to facilitate organelle motility. Our data indicate that taxol disrupts the formation of lysosomes by these adrenal cells, but it seemed unlikely that taxol's ultrastructural effects could prevent organelle transport proposed to cause meetings between mitochondria and the SER or lipid droplets, or prevent ACTH-caused increases in these encounters. Taxol may instead prevent the transfer of lipid droplet or SER-contained cholesterol to adjacent mitochondria, by a means not detectable in our electron micrographs.

Cinemicrographic observations of cultured adrenocortical tumor cells. Dynamic responses to ACTH and cytochalasin B

ACTH increases the basal steroidogenic activity of cultured adrenocortical tumor cells, whereas moderate-high doses of cytochalasin B (CB) inhibit both basal and ACTH-induced steroidogenesis. Previous ultrastructural studies have revealed that ACTH rearranges microfilaments in these adrenal cells, whereas CB causes microfilaments to aggregate into felt-like masses. It has been postulated that the ACTH effects may facilitate organelle motility and increase organelle interactions that are required for steroid biosynthesis, and that the CB-created "foci" may impede or prevent the organelle meetings. To shed light on these possibilities, we have employed 16 mm cinemicrography of unstimulated adrenal tumor cells and cells incubated for 1-2 h with ACTH (10 mU/ml), or low (10 micrograms/ml), or high (50 micrograms/ml) doses of CB. ACTH caused initial increases in membrane ruffling and a "flurry" of particle (organelle) activity above that seen in unstimulated cells. The stimulated cells then retracted from each other and began their characteristic "rounding up" in response to the hormone. Particles appeared to move towards the nucleus, and in fully-rounded cells were extremely congested. Steroid production rose several fold above basal levels. CB10 produced slight-marked cell convexities, nearly stopped particle motility and inhibited steroid production moderately. CB50 produced an asymmetrical, spidery cell form, stopped membrane ruffling and particle motility and abolished steroidogenesis. After a washout of CB50, particle motility resumed nearly immediately. Our CB data indicate that associations between particles, presumably between mitochondria and various sources of cholesterol, are prerequisite for basal steroidogenesis in the adrenal tumor cells. In ACTH-stimulated cells, increases in steroid output correspond with increased opportunities for particle associations. These opportunities appear to arise directly or indirectly from ACTH effects on microfilaments. The responses of microfilaments to the hormone may be particularly intense in tumorous forms. By these means, the cells may express their differentiated function, although their cytoplasm has a distinctly unspecialized appearance.

The effects of ACTH on acid phosphatase activity in endosomes, GERL and lysosomes of cultured adrenal tumor cells

Cultured cells derived from a mouse adrenocortical tumor transplant are unspecialized in appearance, but produce basal levels of steroids and demonstrate a near-immediate steroidogenic response to ACTH. There is biochemical evidence that ACTH induces increases in the uptake of serum lipoproteins by these cells and that this material is hydrolyzed in lysosomes to free cholesterol, a precursor for steroid end products. To investigate morphologically the role of lysosomes in the steroidogenic activity of these cells, cultures were incubated for 4 h with and without ACTH, then processed for the ultrastructural localization of acid phosphatase (ACPase), a marker enzyme for lysosomes, and for GERL, the lysosome-forming subcompartment of the Golgi, and examined by TEM and HVEM. Steroid output was determined by a fluorometric technique. Unstimulated cells secreted basal levels of steroids. By TEM, large endosomes, some containing semi-compact material and ACPase reaction product, were occasionally seen at the cell periphery and in the Golgi region. The Golgi and GERL were poorly developed. Residual bodies, a few of them ACPase+, appeared in the Golgi region and in microtubule-associated clusters near the cell membrane. ACTH-stimulated cells secreted steroids at 8-10 fold basal values. In TEM records, they displayed numerous ACPase+ endosomes between the cell periphery and the Golgi. The Golgi and GERL regions appeared to be hypertrophied and many large, inclusion-containing, strongly ACPase+ residual bodies appeared here and in elongated microtubule-containing cell processes. HVEM micrographs showed more definitively that ACTH produced distinct increases in the size of GERL and in the number of ACPase+ organelles. Our results suggest that in unstimulated cells, endosomes, presumably containing media-derived material, gain lysosomal enzymes in or near GERL, are transformed to residual bodies as their contents are hydrolyzed, and are subsequently translocated by microtubules to the cell periphery for exocytosis. ACTH appears to intensify all of these effects. The "giant" lysosomes seen in stimulated cells may result from a fusion of smaller lysosomes. Their amorphous contents may reflect an inefficient hydrolysis of LDL to free cholesterol.

The reversibility of the effects of ACTH and cytochalasin B on the ultrastructure and steroidogenic activity of adrenocortical tumor cells in vitro

We have demonstrated previously that the steroidogenic activity of ACTH on cultured adrenal tumor cells is associated with cell rounding and a rearrangement of microfilaments. Cytochalasin B (CB) also induces cell rounding, but changes the conformation of microfilaments and severely inhibits steroidogenesis. ACTH and CB may have different modes of action on the contractile machinery which are related to their opposing actions on steroidogenesis. To investigate this possibility further, we have examined the reversibility of the morphological and functional effects of these agents. Cultures were incubated for 1 hr, with and without ACTH (10 microU/ml of media), or with CB (50 micrograms/ml), or with both agents simultaneously. After a media wash, the cultures were incubated for 1 hr, with and without ACTH. The steroid production of the cells during pre- and post-washout incubations was determined, and some cultures were fixed for electron microscopy at the end of both incubation periods. The three- to ten-fold increases in steroidogenic activity of ACTH-stimulated cells declined during recovery incubations, but remained well above basal values. These cells nearly reflattened and began to regain stress fibers which had been 'pulled apart'. The 'washed out' ACTH-stimulated cells were often refractory to restimulation. Cells recovering from CB also reflattened. Masses of filamentous felt induced by the drug disappeared from the cytoplasm, lost microvilli reappeared and stress fibers reformed. The 20-50% inhibition of basal steroidogenesis by CB was completely reversed. When ex-CB-treated cells were incubated with ACTH, their morphology and steroid production were typical of acutely stimulated cells. The recovery behavior of cells incubated with ACTH and CB simultaneously reflected the observation that there were cell-specific responses to one agent or the other during initial incubations. The persistence of heightened steroidogenic activity following a washout of ACTH and the rapid reversal of the effects of CB strongly support the concept that regulated actomyosin interactions are an integral part of the steroidogenic process.

Acute steroidogenic stimulation of cultured adrenocortical tumor cells: an electron microscopic analysis

The addition of ACTH to primary cultures of functional mouse adrenocortical tumor cells results in a marked increase in the production and release of steroids by these cells. The steroidogenic response to ACTH is detectable within minutes and is associated with a 'rounding-up' of the cells. The effects of ACTH were analyzed by scanning, conventional transmission and stereoscopic high voltage electron microscopy. Cell morphology was studied in unstimulated cells, in cells stimulated for 10-15 min in which only partial rounding had occurred, and after 2 hr of continuous stimulation when most of the cells had fully rounded. In unstimulated cultures, bundles of microfilaments (stress fibers) were prominent in cell extensions, subjacent to coated regions of the plasma membrane and frequently in proximity to microtubules and clusters of lysosome-like organelles. As the rounding process commenced, stress fibers disappeared from the cell edges. In 2 hr stimulated cells, Golgi complexes were hypertrophied. There were more microprojections from the plasma membrane than in control cells, as well as evidence of increased pinocytotic activity. There was an apparent polymerization of microtubules in elongated processes extending outward from the bases of the rounded cells and a concentration of lysosome-like organelles in these formations. The possible significance of these changes with respect to the differentiated function of adrenal cells is under continuing investigation in our laboratory.

The ultrastructure of functional mouse adrenal cortical tumor cells in vitro

Cells derived from a transplantable mouse adrenal cortical tumor maintain their differentiated function in vitro and secrete steroids in response to ACTH and other stimulatory agents. The cell line has been widely employed for various biochemical investigations but there have been few attempts to correlate this work with morphologic data. This communication describes the electron microscopic appearance of the tumor transplant in vivo and primary cultures derived from it at various intervals after the cells are placed in culture. Tumor cells in vivo bear considerable resemblance to normal adult mouse adrenal cortical cells. Organelles generally considered to be directly involved in steroid biosynthesis (mitochondria, smooth endoplasmic reticulum and lipid droplets) are not drastically altered. Certain modifications of the vasculature and cell membrane, seemingly related to steroidogenesis, are present in both the tumor and normal adrenal cortex. Within 2 days after the tumor cells are introduced to culture, their cytoplasm assumes a more simplified appearance. Smooth endoplasmic reticulum is less conspicuous and free ribosomes and polysomes are very abundant. Mitchondrial inner membranes are reorganized from a saccular arrangement in the cells in vivo into distinct lamellar cristae. The tumor cells now resemble undifferentiated embryonic adrenal cells, or cultured adrenal cells from various mammalian sources which have dedifferentiated in the absence of ACTH. In their morphologically unspecialized state, the normal cells are incapable of functional responses to ACTH. In contrast, the cultured, dedifferentiated tumor cells respond within minutes to this hormone and can demonstrate 5-20 fold increases in their basal steroid output. These data suggest that substantial steroidogenic activity can occur although the characteristic appearance of adrenal mitochondria is absent.