Filter by type:

Sort by year:

A user-friendly tool for cloud-based whole slide image segmentation, with examples from renal histopathology

B. Lutnick, D. Manthey, J. U. Becker, B. Ginley, K. Moos, J. E. Zuckerman, L. Rodrigues, A. J. Gallan, L. Barisoni, C. E. Alpers, X. X. Wang, K. Myakala, B. A. Jones, M. Levi, J. B. Kopp, T. Yoshida, S. S. Han, S. Jain, A. Z. Rosenberg, K. Y. Jen, and P. Sarder, for the Kidney Precision Medicine Project
Journal Paper [34] Communications Medicine - Nature. In revision.

A self-attention mechanism for COVID-19 detection using chest x-ray images

U. Muhammad, Md. Z. Hoque, M. Oussalah, A. Keskinarkaus, T. Seppänena, and P. Sarder
Journal Paper [33] Knowledge-Based Systems - Elsevier. In revision.

PodoCount: A robust, fully automated whole-slide podocyte quantification tool

B. A. Santo, D. Govind, P. Daneshpajouhnejad, X. Yang, X. X. Wang, K. Myakala, B. A. Jones, M. Levi, J. B. Kopp, L. J. Niedernhofer, D. Manthey, K. C. Moon, S. S. Han, J. Zee, A. Z. Rosenberg, and P. Sarder
Journal Paper [32] Kidney International Reports. In revision.

From what to why, the growing need for a focus shift towards explainability of AI in digital pathology

S. P. Border and P. Sarder
Journal Paper [31] Frontiers in Physiology. To appear.

Histo-fetch – On-the-fly processing of gigapixel whole slide images simplifies and speeds neural network training

B. Lutnick, L. K. Murali, B. Ginley, A. Z. Rosenberg, and P. Sarder
Journal Paper [30] Journal of Pathology Informatics. To appear.


Background Training convolutional neural networks using pathology whole slide images (WSIs) is traditionally prefaced by the extraction of a training dataset of image patches.  While effective, for large datasets of WSIs, this dataset preparation is inefficient.

Methods We created a custom pipeline (histo-fetch) to efficiently extract random patches and labels from pathology WSIs for input to a neural network on-the-fly.  We prefetch these patches as needed during network training, avoiding the need for WSI preparation such as chopping/tiling.

Results & Conclusion We demonstrate the utility of this pipeline to perform artificial stain transfer and image generation using the popular networks CycleGAN and ProGAN, respectively.  For a large WSI dataset, histo-fetch is 98.6% faster to start training and used 7535X less disk space.

A cloud-based tool for federated segmentation of whole slide images

B. Lutnick, D. Manthey, J. U. Becker, J. Zuckerman, L. Rodrigues, K. Y. Jen, and P. Sarder
Conference Paper [35] Proc. of SPIE–Medical Imaging 2022: Digital & Computational Pathology, vol. 12039, pp. 1203951:X–X, San Diego, California, USA, Feb. 2022.

Automated tubular morphometric visualization for whole kidney biopsy

N. Kavthekar, B. Ginley, S. P. Border, N. Lucarelli, K. Y. Jen, and P. Sarder
Conference Paper [34] Proc. of SPIE–Medical Imaging 2022: Digital & Computational Pathology, vol. 12039, pp. 1203948:X–X, San Diego, California, USA, Feb. 2022.

Computational integration of renal histology and urinary proteomics using deep learning regression

N. Lucarelli, D. Yun, D. Han, B. Ginley, K. C. Moon, A. Rosenberg, J. E. Tomaszewski, S. S. Han, and P. Sarder
Conference Paper [33] Proc. of SPIE–Medical Imaging 2022: Digital & Computational Pathology, vol. 12039, pp. 1203931:X–X, San Diego, California, USA, Feb. 2022.

HistoLens: A stand-alone tool for quantitative feature visualization of glomerular histology images

S. P. Border, B. Ginley, J. E. Tomaszewski, and P. Sarder
Conference Paper [32] Proc. of SPIE–Medical Imaging 2022: Digital & Computational Pathology, vol. 12039, pp. 1203929:X–X, San Diego, California, USA, Feb. 2022.

Integrating image analysis with single cell RNA-seq data to study podocyte-specific changes in diabetic kidney disease

D. Govind, S. Meamardoost, R. Yacoub, R. Gunawan, J. E. Tomaszewski, and P. Sarder
Conference Paper [31] Proc. of SPIE–Medical Imaging 2022: Digital & Computational Pathology, vol. 12039, pp. 1203927:X–X, San Diego, California, USA, Feb. 2022.

PodoSighter: A Cloud-Based Tool for Label-Free Podocyte Detection in Kidney Whole-Slide Images

D. Govind, J. U. Becker, J. Miecznikowski, A. Z. Rosenberg, J. Dang, P. L. Tharaux, R. Yacoub, F. Thaiss, P. F. Hoyer, D. Manthey, B. Lutnick, A. M. Worral, I. Mohammad, V. Walavalkar, J. E. Tomaszewski, K. Y. Jen, and P. Sarder
Journal Paper [29] Journal of the American Society of Nephrology, vol. 32, no. 11, pp. 2795–2813, Nov. 2021.


Background Podocyte depletion precedes progressive glomerular damage in several kidney diseases.  However, the current standard of visual detection and quantification of podocyte nuclei from brightfield microscopy images is laborious and imprecise.

Methods We have developed PodoSighter, an online cloud-based tool, to automatically identify and quantify podocyte nuclei from giga-pixel brightfield whole-slide images (WSIs) using deep learning.  Ground-truth to train the tool used immunohistochemically or immunofluorescence-labeled images from a multi-institutional cohort of 122 histologic sections from mouse, rat, and human kidneys.  To demonstrate the generalizability of our tool in investigating podocyte loss in clinically relevant samples, we tested it in rodent models of glomerular diseases, including diabetic kidney disease, crescentic GN, and dose-dependent direct podocyte toxicity and depletion, and in human biopsies from steroid-resistant nephrotic syndrome and from human autopsy tissues.

Results The optimal model yielded high sensitivity/specificity of 0.80/0.80, 0.81/0.86, and 0.80/0.91, in mouse, rat, and human images, respectively, from periodic acid–Schiff-stained WSIs.  Furthermore, the podocyte nuclear morphometrics extracted using PodoSighter were informative in identifying diseased glomeruli.  We have made PodoSighter freely available to the general public as turnkey plugins in a cloud-based web application for end users.

Conclusions Our study demonstrates an automated computational approach to detect and quantify podocyte nuclei in standard histologically stained WSIs, facilitating podocyte research, and enabling possible future clinical applications.

Automated computational detection of interstitial fibrosis, tubular atrophy, and glomerulosclerosis

B. Ginley, K. Y. Jen, S. S. Han, L. Rodrigues, S. Jain, A. Fogo, J. Zuckerman, V. Walavalkar, J. Miecznikowski, Y. Wen, F. Yen, D. Yun, K. C. Moon, A. Rosenberg, C. Parikh, and P. Sarder
Journal Paper [28] Journal of the American Society of Nephrology, vol. 32, no. 4, pp. 837–50, Apr. 2021.


Background Interstitial fibrosis, tubular atrophy (IFTA), and glomerulosclerosis are indicators of irrecoverable kidney injury.  Modern machine learning (ML) tools have enabled robust, automated identification of image structures that can be comparable with analysis by human experts.  ML algorithms were developed and tested for the ability to replicate the detection and quantification of IFTA and glomerulosclerosis that renal pathologists perform.

Methods A renal pathologist annotated renal biopsy specimens from 116 whole-slide images (WSIs) for IFTA and glomerulosclerosis.  A total of 79 WSIs were used for training different configurations of a convolutional neural network (CNN), and 17 and 20 WSIs were used as internal and external testing cases, respectively.  The best model was compared against the input of four renal pathologists on 20 new testing slides.  Further, for 87 testing biopsy specimens, IFTA and glomerulosclerosis measurements made by pathologists and the CNN were correlated to patient outcome using classic statistical tools.

Results The best average performance across all image classes came from aDeepLab version 2 network trained at 40X magnification.  IFTA and glomerulosclerosis percentages derived from this CNN achieved high levels of agreement with four renal pathologists.  The pathologist- and CNN-based analyses of IFTA and glomerulosclerosis showed statistically significant and equivalent correlation with all patient-outcome variables.

Conclusions ML algorithms can be trained to replicate the IFTA and glomerulosclerosis assessment performed by renal pathologists.  This suggests computational methods may be able to provide a standardized approach to evaluate the extent of chronic kidney injury in situations in which renal-pathologist time is restricted or unavailable.

In silico multi-compartment detection based on multiplex immunohistochemical staining in renal pathology

K. Y. Jen, L. K. Murali, B. Lutnick, B. Ginley, D. Govind, H. Mori, G. Gao, and P. Sarder
Conference Paper [30] Proc. of SPIE–Medical Imaging 2021: Digital & Computational Pathology, vol. 11603, pp. 1160337:1–8, San Diego, California, USA, Feb. 2021.


With the rapid advancement in multiplex tissue staining, computer hardware, and machine learning, computationally-based tools are becoming indispensable for the evaluation of digital histopathology.  Historically, standard histochemical staining methods such as hematoxylin and eosin, periodic acid-Schiff, and trichrome have been the gold standard for microscopic tissue evaluation by pathologists, and therefore brightfield microscopy images derived from such stains are primarily used for developing computational pathology tools.  However, these histochemical stains are nonspecific in terms of highlighting structures and cell types.  In contrast, immunohistochemical stains use antibodies to specifically detect and quantify proteins, which can be used to specifically highlight structures and cell types of interest.  Traditionally, such immunofluorescence-based methods are only able to simultaneously stain a limited number of target proteins/antigens, typically up to three channels.  Fluorescence-based multiplex immunohistochemistry (mIHC) is a new technology that enables simultaneous localization and quantification of numerous proteins/antigens, allowing for the possibility to detect a wide range of histologic structures and cell types within tissue.  However, this method is limited by cost, specialized equipment, technical expertise, and time. In this study, we implemented a deep learning-based pipeline to synthetically generate in silico mIHC images from brightfield images of tissue slides-stained with routinely used histochemical stains, in particular PAS. Our tool was trained using fluorescence-based mIHC images as the ground-truth.  The proposed pipeline offers high contrast detection of structures in brightfield imaged tissue sections stained with standard histochemical stains.  We demonstrate the performance of our pipeline by computationally detecting multiple compartments in kidney biopsies, including cell nuclei, collagen/fibrosis, distal tubules, proximal tubules, endothelial cells, and leukocytes, from PAS-stained tissue sections.  Our work can be extended for other histologic structures and tissue types and can be used as a basis for future automated annotation of histologic structures and cell types without the added cost of actually generating mIHC slides.

A distributed system improves inter-observer and AI concordance in annotating interstitial fibrosis and tubular atrophy

A. K. Shashiprakash, B. Lutnick, B. Ginley, D. Govind, N. Lucarelli, K. Y. Jen, A. Z. Rosenberg, A. Urisman, V. Walavalkar, J. E. Zuckerman, M. Delsante, M. L. Z. Bissonnette, J. E. Tomaszewski, D. Manthey, and P. Sarder
Conference Paper [29] Proc. of SPIE–Medical Imaging 2021: Digital & Computational Pathology, vol. 11603, pp. 1160328:1–6, San Diego, California, USA, Feb. 2021.


Histologic examination of interstitial fibrosis and tubular atrophy (IFTA) is critical to determine the extent of irreversible kidney injury in renal disease.  The current clinical standard involves pathologist's visual assessment of IFTA, which is prone to inter-observer variability.  To address this diagnostic variability, we designed two case studies (CSs), including seven pathologists, using HistomicsTK- a distributed system developed by Kitware Inc. (Clifton Park, NY).  Twenty-five whole slide images (WSIs) were classified into a training set of 21 and a validation set of four.  The training set was composed of seven unique subsets, each provided to an individual pathologist along with four common WSIs from the validation set.  In CS 1, all pathologists individually annotated IFTA in their respective slides.  These annotations were then used to train a deep learning algorithm to computationally segment IFTA.  In CS 2, manual and computational annotations from CS 1 were first reviewed by the annotators to improve concordance of IFTA annotation.  Both the manual and computational annotation processes were then repeated as in CS1.  The inter-observer concordance in the validation set was measured by Krippendorff's alpha (KA).  The KA for the seven pathologists in CS1 was 0.62 with CI [0.57, 0.67], and after reviewing each other’s annotations in CS2, 0.66 with CI [0.60, 0.72].  The respective CS1 and CS2 KA were 0.58 with CI [0.52, 0.64] and 0.63 with CI [0.56, 0.69] when including the deep learner as an eighth annotator.  These results suggest that our designed annotation framework refines agreement of spatial annotation of IFTA and demonstrates a human-AI approach to significantly improve the development of computational models.

User friendly, cloud based, whole slide image segmentation

B. Lutnick, A. K. Shashiprakash, D. Manthey, and P. Sarder
Conference Paper [28] Proc. of SPIE–Medical Imaging 2021: Digital & Computational Pathology, vol. 11603, pp. 1160316:1–6, San Diego, California, USA, Feb. 2021.


Convolutional neural networks, the state of the art for image segmentation, have been successfully applied to histology images by many computational researchers.  However, the translatability of this technology to clinicians and biological researchers is limited due to the complex and undeveloped user interface of the code, as well as the extensive computer setup required.  We have developed a plugin for segmentation of whole slide images (WSIs) with an easy to use graphical user interface.  This plugin runs a state-of-the-art convolutional neural network for segmentation of WSIs in the cloud. Our plugin is built on the open source tool HistomicsTK by Kitware Inc. (Clifton Park, NY), which provides remote data management and viewing abilities for WSI datasets.  The ability to access this tool over the internet will facilitate widespread use by computational non-experts.  Users can easily upload slides to a server where our plugin is installed and perform the segmentation analysis remotely.  This plugin is open source and once trained, has the ability to be applied to the segmentation of any pathological structure.  For a proof of concept, we have trained it to segment glomeruli from renal tissue images, demonstrating it on holdout tissue slides.

Automated detection and quantification of Wilms’ Tumor 1-positive cells in murine diabetic kidney disease

D. Govind, B. A. Santo, B. Ginley, R. Yacoub, A. Z. Rosenberg, K. Y. Jen, V. Walavalkar, G. E. Wilding, A. M. Worral, I. Mohammad, and P. Sarder
Conference Paper [27] Proc. of SPIE–Medical Imaging 2021: Digital & Computational Pathology, vol. 11603, pp. 1160312:1–7, San Diego, California, USA, Feb. 2021.


In diabetic kidney disease (DKD), podocyte depletion, and the subsequent migration of parietal epithelial cells (PECs) to the tuft, is a precursor to progressive glomerular damage, but the limitations of brightfield microscopy currently preclude direct pathological quantitation of these cells.  Here we present an automated approach to podocyte and PEC detection developed using kidney sections from mouse model emulating DKD, stained first for Wilms' Tumor 1 (WT1) (podocyte and PEC marker) by immunofluorescence, then post-stained with periodic acid-Schiff (PAS).  A generative adversarial network (GAN)-based pipeline was used to translate these PAS-stained sections into WT1-labeled IF images, enabling in silico label-free podocyte and PEC identification in brightfield images.  Our method detected WT1-positive cells with high sensitivity/specificity (0.87/0.92).  Additionally, our algorithm performed with a higher Cohen's kappa (0.85) than the average manual identification by three renal pathologists (0.78).  We propose that this pipeline will enable accurate detection of WT1- positive cells in research applications.

The rediscovery of nephropathology with artificial intelligence

L. Rodrigues, P. Sarder, K. Y. Jen, B. Ginley, J. Pratas, V. Sousa, A. Figueiredo, and R. Alves
Journal Paper [27] Portuguese Journal of Nephrology and Hypertension, vol. 34, no. 4, pp. 195–97, Jan. 2021.

Improving the accuracy of gastrointestinal neuroendocrine tumor grading with deep learning

D. Govind, K. Y. Jen, K. Matsukuma, G. Gao, K. A. Olson, D. Gui, G. Wilding, S. P. Border, and P. Sarder
Journal Paper [26] Scientific Reports - Nature, vol. 10, pp. 11064: 1–11, Jul. 2020.


The Ki-67 index is an established prognostic factor in gastrointestinal neuroendocrine tumors (GI-NETs) and defines tumor grade. It is currently estimated by microscopically examining tumor tissue single-immunostained (SS) for Ki-67 and counting the number of Ki-67-positive and Ki-67-negative tumor cells within a subjectively picked hot-spot. Intraobserver variability in this procedure as well as difficulty in distinguishing tumor from non-tumor cells can lead to inaccurate Ki-67 indices and possibly incorrect tumor grades. We introduce two computational tools that utilize Ki-67 and synaptophysin double-immunostained (DS) slides to improve the accuracy of Ki-67 index quantitation in GI-NETs: (1) Synaptophysin-KI-Estimator (SKIE), a pipeline automating Ki-67 index quantitation via whole-slide image (WSI) analysis and (2) deep-SKIE, a deep learner-based approach where a Ki-67 index heatmap is generated throughout the tumor. Ki-67 indices for 50 GI-NETs were quantitated using SKIE and compared with DS slide assessments by three pathologists using a microscope and a fourth pathologist via manually ticking off each cell, the latter of which was deemed the gold standard (GS). Compared to the GS, SKIE achieved a grading accuracy of 90% and substantial agreement (linear-weighted Cohen's kappa 0.62). Using DS WSIs, deep-SKIE displayed a training, validation, and testing accuracy of 98.4%, 90.9%, and 91.0%, respectively, significantly higher than using SS WSIs. Since DS slides are not standard clinical practice, we also integrated a cycle generative adversarial network into our pipeline to transform SS into DS WSIs. The proposed methods can improve accuracy and potentially save a significant amount of time if implemented into clinical practice.

Artificial intelligence driven next-generation renal histomorphometry

B. Santo, A. Rosenberg, and P. Sarder
Journal Paper [25] Current Opinion in Nephrology and Hypertension, vol. 29, no. 3, pp. 265–72, May 2020.


Purpose of review Successful integration of artificial intelligence into extant clinical workflows is contingent upon a number of factors including clinician comprehension and interpretation of computer vision. This article discusses how image analysis and machine learning have enabled comprehensive characterization of kidney morphology for development of automated diagnostic and prognostic renal pathology applications.

Recent findings The primordial digital pathology informatics work employed classical image analysis and machine learning to prognosticate renal disease. Although this classical approach demonstrated tremendous potential, subsequent advancements in hardware technology rendered artificial neural networks (ANNs) the method of choice for machine vision in computational pathology. Offering rapid and reproducible detection, characterization and classification of kidney morphology, ANNs have facilitated the development of diagnostic and prognostic applications. In addition, modern machine learning with ANNs has revealed novel biomarkers in kidney disease, demonstrating the potential for machine vision to elucidate novel pathologic mechanisms beyond extant clinical knowledge.

Summary Despite the revolutionary developments potentiated by modern machine learning, several challenges remain, including data quality control and curation, image annotation and ontology, integration of multimodal data and interpretation of machine vision or 'opening the black box'. Resolution of these challenges will not only revolutionize diagnostic pathology but also pave the way for precision medicine and integration of artificial intelligence in the process of care.

Deep learning-based automated hot-spot detection and tumor grading in human gastrointestinal neuroendocrine tumor

D. Govind, K. Y. Jen, and P. Sarder
Conference Paper [26] Proc. of SPIE–Medical Imaging 2020: Digital Pathology, vol. 11320, pp. 1132009:1–7, Houston, Texas, USA, Feb. 2020.


Ki-67 index is an important diagnostic factor in gastrointestinal neuroendocrine tumor (GI-NET). The current gold standard for grading GI-NETs involves the visual screening of histopathologically stained tissues, for hot-spots containing high amounts of proliferating tumor cells (stained with Ki-67 antibody). Subsequently, the Ki-67 index, i.e. the percentage of proliferating tumor cells within the hot-spot is manually obtained. To automate this subjective and time consuming process, we have developed an integrated pipeline, termed SKIE (synaptophysin-Ki-67 index estimator), combining double-immunohistochemical (IHC) staining for synaptophysin (stains tumor) and Ki-67, with whole slide image (WSI) analysis. The Ki-67 index for 50 human GI-NET WSIs were estimated by SKIE and compared with three pathologists' assessment, and the gold standard (exhaustive counting by a fourth pathologist) based on the double-stained image. All four pathologists unanimously graded 38 WSIs, among which, SKIE achieved 94.74% accuracy. One discrepant case was attributed to staining inconsistencies and the other to SKIE selecting a better hot-spot. The remaining 12 WSIs had discrepant grades among pathologists, and hence, the gold standard was chosen for comparison, wherein, 10 WSI grades matched with that of the gold standard, and SKIE assigned a lower and higher grade to two cases. Overall, SKIE agreed with the gold standard with a substantial linear weighted Cohen's kappa k = 0.622 with CI [0.286, 0.958]. We further expanded our method to deep-SKIE, wherein, a deep convolutional neural network (DCNN) was trained and validated using 13,736 hotspot-sized tiles from 40 WSIs, each categorized into one of four classes (background, non-tumor, tumor grade 1, tumor grade 2) by SKIE and tested on 9 WSIs. Deep-SKIE achieved an accuracy of 91.63% with near-perfect agreement (k = 0.88 with CI [0.87, 0.89]) with the gold standard.

Generative modeling for renal microanatomy

L. K. Murali, B. Lutnick, B. Ginley, J. E. Tomaszewski, and P. Sarder
Conference Paper [25] Proc. of SPIE–Medical Imaging 2020: Digital Pathology, vol. 11320, pp. 1132040:1–10, Houston, Texas, USA, Feb. 2020.


Generative adversarial networks (GANs) have received immense attention in the field of machine learning for their potential to learn high-dimensional and real data distribution. These methods do not rely on any assumptions about the data distribution of the input sample and can generate real-like samples from latent vector space based on unsupervised learning. In the medical field, particularly, in digital pathology expert annotation and availability of a large set of training data is costly and the study of manifestations of various diseases is based on visual examination of stained slides. In clinical practice, various staining information is required to improve the pathological diagnosis process. But when the sampled tissue to be examined is limited, the final diagnosis made by the pathologist is based on limited stain styles. These limitations can be overcome by studying the usability and reliability of generative models in the field of digital pathology. To understand the usability of the generative models, we synthesize in an unsupervised manner, high resolution renal microanatomical structures like renal glomerulus in thin tissue histology images using state-of-art architectures like Deep Convolutional Generative Adversarial Network (DCGAN) and Enhanced Super- Resolution Generative Adversarial Network (ESRGAN). Successful generation of such structures will lead to obtaining a large set of labeled data for further developing supervised algorithms for disease classification and understanding progression. Our study suggests while GAN is able to attain formalin fixed and paraffin embedded tissue image quality, GAN requires further prior knowledge as input to model intrinsic micro-anatomical details, such as capillary wall, urinary pole, nuclei placement, suggesting developing semi-supervised architectures by using these above details as prior information. Also, the generative models can be used to create an artificial effect of staining without physically tampering the histopathological slide. To demonstrate this, we use a CycleGAN network to transform Hematoxylin and eosin (H&E) stain to Periodic acid-Schiff (PAS) stain and Jones methenamine silver (JMS) stain to PAS stain. In this way GAN can be employed to translate different renal pathology stain styles when the relevant staining information is not available in the clinical settings.

The presence and location of podocytes in glomeruli as affected by diabetes mellitus

K. E. Maraszek, B. Santo, R. Yacoub, J. E. Tomaszewski, I. Mohammad, A. Worral, and P. Sarder
Conference Paper [24] Proc. of SPIE–Medical Imaging 2020: Digital Pathology, vol. 11320, pp. 1132012:1–10, Houston, Texas, USA, Feb. 2020.


The primary purpose of the kidney, specifically the glomerulus, is filtration. Filtration is accomplished through the glomerular filtration barrier, which consists of the fenestrated endothelium, glomerular basement membrane, and specialized epithelial cells called podocytes. In pathologic states, such as Diabetes Mellitus (DM) and diabetic kidney disease (DKD), variable glomerular conditions result in podocyte injury and depletion, followed by progressive glomerular injury and DKD progression. In this work, we quantified glomerulus and podocyte structural changes in histopathology image data derived from a murine model of DM. Using a variety of image processing techniques, we studied changes in podocyte morphology and intra-glomerular distribution across healthy, mild DM, and DM glomeruli. Our feature analysis provided feature trends which we believe are reflective of DKD pathology; while glomerular area peaked in mild DM, average podocyte number and distance from the urinary pole continued to increase throughout DM. Ultimately, this study aims to augment the set of quantifiable image biomarkers used for evaluation of DKD progression in digital pathology, as well as underscore the importance of engineering biologically inspired image features.

Probabilistic modelling of diabetic nephropathy progression

S. P. Border, K. Y. Jen, W. L. C. dos-Santos, J. E. Tomaszewski, and P. Sarder
Conference Paper [23] Proc. of SPIE–Medical Imaging 2020: Digital Pathology, vol. 11320, pp. 1132018:1–10, Houston, Texas, USA, Feb. 2020.


Diabetic Nephropathy (DN) progression is stratified into several stages with different levels of proteinuria, albuminuria, and physical characteristics as observed by pathologists. These physical changes are primarily visible within a patient's glomeruli which function as filtration units for blood returning for oxygenation. As DN stage increases, it is possible to observe the thickening of the glomerular basement membrane, expansion of the mesangium, and development of nodular sclerosis. Classification of different stages of DN by pathologists is based on semiqualitative assessments of these characteristics on an individual glomerulus basis. Being able to probabilistically infer stage membership of individual glomeruli based on a combination of easily observable and hidden image features would be an invaluable tool for furthering our understanding of the drivers of DN progression. Markov Particle filters, included in the bnlearn package in R, were used to query a Bayesian Network (BN) constructed using the structural Hill-Climbing algorithm on a set of glomerular features. These features included both traditional characteristics such as glomerular area and number of mesangial nuclei as well as more abstract features derived from Minimum Spanning Trees (MST) to quantify spatial distribution of mesangial nuclei. Our results using images from multiple institutions suggest that these abstract features exercise a variable influence on DN stage membership over the course of disease progression. Further research incorporating clinical data will give nephrologists a "white box" visual of quantitative factors present in DN patients.

Neutrophil Extracellular Traps (NETs): an unexplored territory in renal pathobiology, a pilot computational study

B. Santo, B. Segal, J. E. Tomaszewski, I. Mohammad, A. Worral, S. Jain, M. Visser, and P. Sarder
Conference Paper [22] Proc. of SPIE–Medical Imaging 2020: Digital Pathology, vol. 11320, pp. 1132022:1–7, Houston, Texas, USA, Feb. 2020.


In the age of modern medicine and artificial intelligence, image analysis and machine learning have revolutionized diagnostic pathology, facilitating the development of computer aided diagnostics (CADs) which circumvent prevalent diagnostic challenges. Although CADs will expedite and improve the precision of clinical workflow, their prognostic potential, when paired with clinical outcome data, remains indeterminate. In high impact renal diseases, such as diabetic nephropathy and lupus nephritis (LN), progression often occurs rapidly and without immediate detection, due to the subtlety of structural changes in transient disease states. In such states, exploration of quantifiable image biomarkers, such as Neutrophil Extracellular Traps (NETs), may reveal alternative progression measures which correlate with clinical data. NETs have been implicated in LN as immunogenic cellular structures, whose occurrence and dysregulation results in excessive tissue damage and lesion manifestation. We propose that renal biopsy NET distribution will function as a discriminate, predictive biomarker in LN, and will supplement existing classification schemes. We have developed a computational pipeline for segmenting NET-like structures in LN biopsies. NET-like structures segmented from our biopsies warrant further study as they appear pathologically distinct, and resemble nonlytic, vital NETs. Examination of corresponding H&E regions predominantly placed NET-like structures in glomeruli, including globally and segmentally sclerosed glomeruli, and tubule lumina. Our work continues to explore NET-like structures in LN biopsies by: 1.) revising detection and analytical methods based on evolving NETs definitions, and 2.) cataloguing NET morphology in order to implement supervised classification of NET-like structures in histopathology images.

Fully automated classification of glomerular lesions in lupus nephritis

B. Ginley, K. Y. Jen, A. Rosenberg, G. M. Rossi, S. Jain, and P. Sarder
Conference Paper [21] Proc. of SPIE–Medical Imaging 2020: Digital Pathology, vol. 11320, pp. 1132005:1–7, Houston, Texas, USA, Feb. 2020.


Systemic lupus erythematosus is a disease in which the immune system attacks host tissues. One organ commonly attacked is the kidney, in which case the resultant acute and chronic damages are called lupus nephritis. The accumulated damage can result in renal failure. The percutaneous renal biopsy is invaluable to the assessment of the disease and its therapeutic response. A large portion of the pathological assessment is done by histological analysis of the biopsied tissue with light microscopy. Computational models can alleviate a portion of expert disagreement by providing unified, reproducible quantifications of digitized image structures. In this work, we perform fully automated whole slide segmentation of glomeruli from Periodic Acid- Schiff (PAS), hematoxylin and eosin, silver, and trichrome stained lupus nephritis biopsies. The automatically extracted PAS glomeruli are quantified by a set of 285 hand-crafted features designed specifically to target glomerular lesions in lupus nephritis. These features are fed in sequence to a recurrent neural network architecture which views multiple glomerular features from a single biopsy, and outputs a continuous diagnostic value representative of classes II-V of the scheme by Weening et al1. On 82 whole slide images taken from 65 patients, compared to renal pathologist annotations and using only the PAS stain, the network achieved a Cohen's kappa of 0.42 with 95% confidence interval [0.32, 0.52] to render the correct class chosen from II-V, and 0.56, 95% CI [0.43, 0.69] to render an additional class V diagnosis when required.

Generative modeling for label-free glomerular modeling and classification

B. Lutnick, B. Ginley, K. Y. Jen, W. Dong, and P. Sarder
Conference Paper [20] Proc. of SPIE–Medical Imaging 2020: Digital Pathology, vol. 11320, pp. 1132008:1–6, Houston, Texas, USA, Feb. 2020.


Generative modeling using GANs has gained traction in machine learning literature, as training does not require labeled datasets. This is perfect for applications in biological datasets, where large labeled datasets are often difficult and expensive to acquire. However, generative models offer no easy way to encode real images into feature-sets, something that is desirable for network explainability and may yield potentially informative image features. For this reason, we test a VAE-GAN architecture for label-free modeling of glomerular structural features. We show that this network can generate realistic looking synthetic images, and be used to interpolate between images. To prove the biological relevance of the network encodings, we classify small-labeled sets of encoded glomeruli by biopsy Tervaert class and for the presence of sclerosis, obtaining a Cohen's kappa values of 0.87 and 0.78 respectfully.

Computational segmentation and classification of diabetic glomerulosclerosis

B. Ginley, B. Lutnick, K. Y. Jen, A. Fogo, S. Jain, A. Rosenberg, V. Walavalkar, G. Wilding, J. E. Tomaszewski, R. Yacoub, G. M. Rossi, and P. Sarder
Journal Paper [24] Journal of the American Society of Nephrology, vol. 30, no. 10, pp. 1953–67, Oct. 2019.


Background Pathologists use visual classification of glomerular lesions to assess samples from patients with diabetic nephropathy (DN).  The results may vary among pathologists.  Digital algorithms may reduce this variability and provide more consistent image structure interpretation.

Methods We developed a digital pipeline to classify renal biopsies from patients with DN.  We combined traditional image analysis with modern machine learning to efficiently capture important structures, minimize manual effort and supervision, and enforce biologic prior information onto our model.  To computationally quantify glomerular structure despite its complexity, we simplified it to three components consisting of nuclei, capillary lumina and Bowman spaces; and Periodic Acid-Schiff positive structures.  We detected glomerular boundaries and nuclei from whole slide images using convolutional neural networks, and the remaining glomerular structures using an unsupervised technique developed expressly for this purpose.  We defined a set of digital features which quantify the structural progression of DN, and a recurrent network architecture which processes these features into a classification.

Results Our digital classification agreed with a senior pathologist whose classifications were used as ground truth with moderate Cohen’s kappa κ = 0.55 and 95% confidence interval [0.50, 0.60].  Two other renal pathologists agreed with the digital classification with κ1 = 0.68, 95% interval [0.50, 0.86] and κ2 = 0.48, 95% interval [0.32, 0.64].  Our results suggest computational approaches are comparable to human visual classification methods, and can offer improved precision in clinical decision workflows.  We detected glomerular boundaries from whole slide images with 0.93±0.04 balanced accuracy, glomerular nuclei with 0.94 sensitivity and 0.93 specificity, and glomerular structural components with 0.95 sensitivity and 0.99 specificity.

Conclusions Computationally derived, histologic image features hold significant diagnostic information that may augment clinical diagnostics.

Computational analysis of cerebrovascular structures imaged using two-photon microscopy

S. Dhiman, I. Singh, and P. Sarder
Conference Paper [19] Proc. of SPIE–Medical Imaging 2019: Digital Pathology, vol. 10956, pp. 1095617:1–6, San Diego, California, USA, Feb. 2019.


Neurodegenerative diseases including Alzheimer's affect millions around the world, and this number is projected to increase over the years unless a breakthrough is made.  There are several theories on the pathogenesis of neurodegenerative diseases, with the amyloid cascade and tau theory being the most prominent ones.  The formation of amyloid plaques and tau tangles collapses capillaries in the brain, thereby inducing hypoxia and destruction of neurons from loss of nourishment.  While we do understand some of the changes that occur in the brain's vasculature form the pathogenesis of these diseases, they have not yet been mathematically characterized with precision.  A computational pipeline is presented here to analyze optically sectioned mice brain sections imaged via two-photon microscopy and characterize various vasculature parameters which are known to deteriorate from neurogenerative diseases.  Our proposed pipeline aims to quantify various brain vasculature parameters, such as, vessel tortuosity, diameter, volume and length, as well the degree of difference to understand disease pathogenesis with the eventual hope of providing drug intervention to regress or minimize these changes.

Examining structural patterns and causality in diabetic nephropathy using inter-glomerular distance and Bayesian graphical models

A. Majumdar, K. Y. Jen, S. Jain, J. E. Tomaszewski, and P. Sarder
Conference Paper [18] Proc. of SPIE–Medical Imaging 2019: Digital Pathology, vol. 10956, pp. 1095608:1–6, San Diego, California, USA, Feb. 2019.


In diabetic nephropathy (DN), hyperglycemia drives a progressive thickening of glomerular filtration surfaces, increased cell proliferation as well as mesangial expansion and a constriction of capillary lumens.  This leads to progressive structural changes inside the glomeruli.  In this work, we make a study of structural glomerular changes in DN from a graph-theoretic standpoint, using features extracted from minimal spanning trees (MSTs) constructed over intercellular distances in order to classify the "packing signatures" of different DN stages.  We further investigate the significance of the competing effects of volume change measured here in 2-dimensional pixel span area on one hand and increased cell proliferation on the other in determining the packing patterns.  Towards that we formulate the problem as dynamic Bayesian network (DBN).  From our preliminary results we do postulate that volume expansion caused by internal pressure as capillary lumens constriction has perhaps has a greater effect in the early stages.

An integrated iterative annotation technique for easing neural network training in medical image analysis

B. Lutnick, B. Ginley, D. Govind, S. D. McGarry, P. S. LaViolette, R. Yacoub, S. Jain, J. E. Tomaszewski, K. Y. Jen, and P. Sarder
Journal Paper [23] Nature Machine Intelligence, vol. 1, no. 2, pp. 112–19, Feb. 2019.


Neural networks promise to bring robust, quantitative analysis to medical fields. However, their adoption is limited by the technicalities of training these networks and the required volume and quality of human-generated annotations. To address this gap in the field of pathology, we have created an intuitive interface for data annotation and the display of neural network predictions within a commonly used digital pathology whole-slide viewer.  This strategy used a 'human-in-the-loop' to reduce the annotation burden.  We demonstrate that segmentation of human and mouse renal micro compartments is repeatedly improved when humans interact with automatically generated annotations throughout the training process.  Finally, to show the adaptability of this technique to other medical imaging fields, we demonstrate its ability to iteratively segment human prostate glands from radiology imaging data.

Induced pluripotent stem cells reveal common developmental genome deprograming in schizophrenia

S. T. Narla, B. Decker, P. Sarder, E. Stachowiak, and M. Stachowiak
Book Chapter [2] Human Neural Stem Cells | pp. 137–62 | L. Buzanska, Ed. | Results and Problems in Cell Differentiation, vol. 66 | Springer, Cham | 2018.


Schizophrenia is a neurodevelopmental disorder characterized by complex aberrations in the structure, wiring, and chemistry of multiple neuronal systems.  The abnormal developmental trajectory of the brain is established during gestation, long before clinical manifestation of the disease.  Over 200 genes and even greater numbers of single nucleotide polymorphisms and copy number variations have been linked with schizophrenia.  How does altered function of such a variety of genes lead to schizophrenia?  We propose that the protein products of these altered genes converge on a common neurodevelopmental pathway responsible for the development of brain neural circuit and neurotransmitter systems.  The results of a multichanneled investigation using induced pluripotent stem cell (iPSCs)- and embryonic stem cell (ESCs)-derived neuronal committed cells (NCCs) indicate an early (preneuronal) developmental-genomic etiology of schizophrenia and that the dysregulated developmental gene networks are common to genetically unrelated cases of schizophrenia.  The results support a "watershed" mechanism in which mutations within diverse signaling pathways affect the common pan-ontogenic mechanism, integrative nuclear (n)FGFR1 signaling (INFS).  Dysregulation of INFS in schizophrenia NCCs deconstructs coordinated gene networks and leads to formation of new networks by the dysregulated genes.  This genome deprograming affects critical gene programs and pathways for neural development and functions.  Studies show that the genomic deprograming reflect an altered nFGFR1—genome interactions and deregulation of miRNA genes by nFGFR1.  In addition, changes in chromatin topology imposed by nFGFR1 may play a role in coordinate gene dysregulation in schizophrenia.

Cellular trafficking of Sn-2 phosphatidylcholine prodrugs studied with fluorescence lifetime imaging and super-resolution microscopy

D. Maji, J. Lu, P. Sarder, A. H. Schmieder, G. Cui, X. Yang, D. Pan, M. D. Lew, S. Achilefu, and G. M. Lanza
Journal Paper [22] Prec. Nanomed., vol. 1, no. 2, pp. 127–45, Jul. 2018.


While the in vivo efficacy of Sn-2 phosphatidylcholine prodrugs incorporated into targeted, non-pegylated lipid-encapsulated nanoparticles was demonstrated in prior preclinical studies, the microscopic details of cell prodrug internalization and trafficking events are unknown.  Classic fluorescence microscopy, fluorescence lifetime imaging microscopy, and single-molecule super-resolution microscopy were used to investigate the cellular handling of doxorubicin-prodrug and AlexaFluor-488-prodrug.  Sn-2 phosphatidylcholine prodrugs delivered by hemifusion of nanoparticle and cell phospholipid membranes functioned as phosphatidylcholine mimics, circumventing the challenges of endosome sequestration and release.  Phosphatidylcholine prodrugs in the outer cell membrane leaflet translocated to the inner membrane leaflet by ATP-dependent and ATP-independent mechanisms and distributed broadly within the cytosolic membranes over the next 12 h.  A portion of the phosphatidylcholine prodrug populated vesicle membranes trafficked to the perinuclear Golgi/ER region, where the drug was enzymatically liberated and activated.  Native doxorubicin entered the cells, passed rapidly to the nucleus, and bound to dsDNA, whereas DOX was first enzymatically liberated from DOX-prodrug within the cytosol, particularly in the perinuclear region, before binding nuclear dsDNA.  Much of DOX-prodrug was initially retained within intracellular membranes.  In vitro anti-proliferation effectiveness of the two drug delivery approaches was equivalent at 48 h, suggesting that residual intracellular DOX-prodrug may constitute a slow-release drug reservoir that enhances effectiveness.  We have demonstrated that Sn-2 phosphatidylcholine prodrugs function as phosphatidylcholine mimics following reported pathways of phosphatidylcholine distribution and metabolism.  Drug complexed to the Sn-2 fatty acid is enzymatically liberated and reactivated over many hours, which may enhance efficacy over time. 

Automated erythrocyte detection and classification from whole slide images

D. Govind, B. Lutnick, J. E. Tomaszewski, and P. Sarder
Journal Paper [21] Journal of Medical Imaging - SPIE, vol. 5, no. 2, pp. 027501:1–11, Apr. 2018.


Blood smear is a crucial diagnostic aid.  Quantification of both solitary and overlapping erythrocytes within these smears, directly from their whole slide images (WSIs), remains a challenge.  Existing software designed to accomplish the computationally extensive task of hematological whole slide image analysis are too expensive and are widely unavailable.  We have thereby developed a fully automated software targeted for erythrocyte detection and quantification from WSIs.  In this paper, we define an optimal region within the smear which contain cells which are neither too scarce or damaged nor too crowded.  We detect the optimal regions within the smear and subsequently extract all the cells from these regions, both solitary and overlapped, the latter of which undergoes a clump splitting before extraction.  The performance was systematically tested on 28 WSIs of blood smears obtained from 13 different species from three classes of the subphylum vertebrata including birds, mammals, and reptiles.  This data poses as an immensely variant erythrocyte database with diversity in size, shape, intensity and textural features.  Our method detected ~3.02 times more cells than that detected from the traditional monolayer and resulted in a testing accuracy of 99.14% for the classification into their respective class (bird, mammal, or reptile), and a testing accuracy of 84.73% for the classification into their respective species.  The results suggest the potential employment of this software for the diagnosis of hematological disorders like sickle cell anemia. 

Glomerular detection and segmentation from multimodal microscopy images using a Butterworth band-pass filter

D. Govind, B. Ginley, B. Lutnick, J. E. Tomaszewski, and P. Sarder
Conference Paper [17] Proc. of SPIE–Medical Imaging 2018: Digital Pathology, vol. 10581, pp. 10158114:1–7, Houston, Texas, USA, Feb. 2018.


We present a rapid, scalable, and high throughput computational pipeline to accurately detect and segment the glomerulus from renal histopathology images with high precision and accuracy.  Our proposed method integrates information from fluorescence and bright-field microscopy imaging of renal tissues.  For computation, we exploit the simplicity, yet extreme robustness of Butterworth bandpass filter to extract the glomeruli by utilizing the information inherent in the renal tissue stained with immunofluorescence marker sensitive at blue emission wavelength as well as tissue auto-fluorescence.  The resulting output is in-turn used to detect and segment multiple glomeruli within the field-of-view in the same tissue section post-stained with histopathological stains.  Our approach, optimized over 40 images, produced a sensitivity/specificity of 0.95/0.84 on n = 66 test images, each containing one or more glomeruli.  The work not only has implications in renal histopathology involving diseases with glomerular structural damages, which is vital to track the progression of the disease, but also aids in the development of a tool to rapidly generate a database of glomeruli from whole slide images, essential for training neural networks.  The current practice to detect glomerular structural damage is by the manual examination of biopsied renal tissues, which is laborious, time intensive and tedious.  Existing automated pipelines employ complex neural networks which are computationally extensive, demand expensive high-performance hardware and require large expert-annotated datasets for training.  Our automated method to detect glomerular boundary will aid in rapid extraction of glomerular compartmental features from large renal histopathological images.

Deep variational auto-encoders for unsupervised glomerular classification

B. Lutnick, R. Yacoub, K. Y. Jen, J. E. Tomaszewski, S. Jain, and P. Sarder
Conference Paper [16] Proc. of SPIE–Medical Imaging 2018: Digital Pathology, vol. 10581, pp. 1015810C:1–7, Houston, Texas, USA, Feb. 2018.


The adoption of deep learning techniques in medical applications has thus far been limited by the availability of the large labeled datasets required to robustly train neural networks, as well as difficulty interpreting these networks.  However, recent techniques for unsupervised training of neural networks promise to address these issues, leveraging only structure to model input data.  We propose the use of a variational autoencoder (VAE) which utilizes data from an animal model to augment the training set and non-linear dimensionality reduction to map this data to human sets.  This architecture utilizes variational inference, performed on latent parameters, to statistically model the probability distribution of training data in a latent feature space.  We show the feasibility of VAEs, using images of mouse and human renal glomeruli from various pathological stages of diabetic nephropathy (DN), to model the progression of structural changes which occur in DN.  When plotted in a 2-dimentional latent space, human and mouse glomeruli, show separation with some overlap, suggesting that the data is continuous, and can be statistically correlated.  When DN stage is plotted in this latent space, trends in disease pathology are visualized.

Examining structural changes in diabetic nephropathy using inter-nuclear distances in glomeruli: A comparison of variously automated methods

O. Simon, R. Yacoub, S. Jain, J. E. Tomaszewski, and P. Sarder
Conference Paper [15] Proc. of SPIE–Medical Imaging 2018: Digital Pathology, vol. 10581, pp. 1015810B:1–10, Houston, Texas, USA, Feb. 2018.


In diabetic nephropathy (DN), hyperglycemia drives a progressive thickening of and damage to the glomerular filtration surfaces, as well as mesangial expansion and a constriction of capillary lumens.  This leads at first to high blood pressure, increased glomerular filtration and micro-proteinuria, and later (if untreated) to severe proteinuria and end-stage renal disease (ESRD).  Though, it is well known that DN is accompanied by marked histopathological changes, the assessment of these structural changes is to a degree subjective and hence varies between pathologists.  In this work, we make a first study of glomerular changes in DN from a graph-theoretical and distance-based standpoint, using minimal spanning trees (MSTs) and distance matrices to generate statistical distributions that can potentially provide a "fingerprint" of DN.  We apply these tools to detect notable differences between normal and DN glomeruli in both human disease and in a streptozotocin-induced (STZ) mouse model.  We also introduce an automated pipeline for rapidly generating MSTs and evaluating their properties with respect to DN, and make a first pass at three-dimensional MST structures.  We envision these approaches may provide a better understanding not only of the processes underway in DN progression, but of key differences between actual human disease and current experimental models.

Computational analysis of the structural progression of human diabetic nephropathy glomeruli

B. Ginley, J. E. Tomaszewski, K. Y. Jen, A. Fogo, S. Jain, and P. Sarder
Conference Paper [14] Proc. of SPIE–Medical Imaging 2018: Digital Pathology, vol. 10581, pp. 1015810A:1–7, Houston, Texas, USA, Feb. 2018.


The glomerulus is the primary compartment of blood filtration in the kidney.  It is a sphere of bundled, fenestrated capillaries that selectively allows solute loss.  Structural damages to glomerular micro-compartments lead to physiological failures which influence filtration efficacy.  The sole way to confirm glomerular structural damage in renal pathology is by examining histopathological or immunofluorescence stained needle biopsies under a light microscope.  However, this method is extremely tedious and time consuming, and requires manual scoring on the number and volume of structures.  Computational image analysis is the perfect tool to ease this burden.  The major obstacle to development of digital histopathological quantification protocols for renal pathology is the extreme heterogeneity present within kidney tissue.  Here we present an automated computational pipeline to 1) segment glomerular compartment boundaries and 2) quantify features of compartments, in healthy and diseased renal tissue.  The segmentation involves a two stage process, one step for rough segmentation generation and another for refinement.  Using a Naïve Bayesian classifier on the resulting feature set, this method was able to distinguish pathological stage IIa from III with 0.89/0.93 sensitivity/specificity and stage IIb from III with 0.7/0.8 sensitivity/specificity, on n = 514 glomeruli taken from n = 13 human biopsies with diagnosed diabetic nephropathy, and n = 5 human renal tissues with no histological abnormalities.  Our method will simplify computational partitioning of glomerular micro-compartments and subsequent quantification.  We aim for our methods to ease manual labor associated with clinical diagnosis of renal disease. 

Multi-radial LBP features as a tool for rapid glomerular detection and assessment in whole slide histopathology images

O. Simon, R. Yacoub, S. Jain, J. E. Tomaszewski, and P. Sarder
Journal Paper [20] Scientific Reports - Nature, vol. 8, pp. 2032:1–11, Feb. 2018.


We demonstrate a simple and effective automated method for the localization of glomeruli in large (~1 gigapixel) histopathological whole-slide images (WSIs) of thin renal tissue sections and biopsies, using an adaptation of the well-known local binary patterns (LBP) image feature vector to train a support vector machine (SVM) model.  Our method offers high precision (>90%) and reasonable recall (>70%) for glomeruli from WSIs, is readily adaptable to glomeruli from multiple species, including mouse, rat, and human, and is robust to diverse slide staining methods.  Using 5 Intel(R) Core(TM) i7-4790 CPUs with 40 GB RAM, our method typically requires ~15 sec for training and ~2 min to extract glomeruli reproducibly from a WSI.  Deploying a deep convolutional neural network trained for glomerular recognition in tandem with the SVM suffices to reduce false positives to below 3%.  We also apply our LBP-based descriptor to successfully detect pathologic changes in a mouse model of diabetic nephropathy.  We envision potential clinical and laboratory applications for this approach in the study and diagnosis of glomerular disease, and as a means of greatly accelerating the construction of feature sets to fuel deep learning studies into tissue structure and pathology. 

Dual-radiolabeled nanoparticle probes for depth-independent in vivo imaging of enzyme activation

K. Black, M. Zhou, P. Sarder, M. Kuchuk, A. Al-Yasiri, S. Gunsten, K. Liang, H. Hennkens, W. Akers, R. Laforest, S. Brody, C. Cutler, and S. Achilefu
Conference Paper [13] Proc. of SPIE–Reporters, Markers, Dyes, Nanoparticles, and Molecular Probes for Biomedical Applications, vol. 10508, pp. 1050805:1–5, San Francisco, California, USA, Jan. 2018.


Quantitative and noninvasive measurement of protease activities has remained an imaging challenge in deep tissues such as the lungs.  Here, we designed a dual-radiolabeled probe for reporting the activities of proteases such as matrix metalloproteinases (MMPs) with multispectral single photon emission computed tomography (SPECT) imaging.  A gold nanoparticle (NP) was radiolabeled with 125I and 111In and functionalized with an MMP9-cleavable peptide to form a multispectral SPECT imaging contrast agent.  In another design, incorporation of 199Au radionuclide into the metal crystal structure of gold NPs provided a superior and stable reference signal in lungs, and 111In was linked to the NP surface via a protease-cleavable substrate, which can serve as an enzyme activity reporter.  This work reveals strategies to correlate protease activities with diverse pathologies in a tissue-depth independent manner. 

Computational detection and quantification of human and mouse neutrophil extracellular traps in flow cytometry and confocal microscopy

B. Ginley, T. Emmons, B. Lutnick, C. Urban, B. H. Segal, and P. Sarder
Journal Paper [19] Scientific Reports - Nature, vol. 7, pp. 17755:1–11, Dec. 2017.


Neutrophil extracellular traps (NETs) are extracellular defense mechanisms used by neutrophils, where chromatin is expelled together with histones and granular/cytoplasmic proteins.  They have become an immunology hotspot, implicated in infections, but also in a diverse array of diseases such as systemic lupus erythematosus, diabetes, and cancer.  However, the precise assessment of in vivo relevance in different disease settings has been hampered by limited tools to quantify occurrence of extracellular traps in experimental models and human samples.  To expedite progress towards improved quantitative tools, we have developed computational pipelines to identify extracellular traps from an in vitro human samples visualized using the ImageStream® platform (Millipore Sigma, Darmstadt, Germany), and confocal images of an in vivo mouse disease model of aspergillus fumigatus pneumonia.  Our two in vitro methods, tested on n = 363/n = 145 images respectively, achieved holdout sensitivity/specificity 0.98/0.93 and 1/0.92.  Our unsupervised method for thin lung tissue sections in murine fungal pneumonia achieved sensitivity/specificity 0.99/0.98 in n = 14 images.  Our supervised method for thin lung tissue classified NETs with sensitivity/specificity 0.86/0.90. We expect that our approach will be of value for researchers, and have application in infectious and inflammatory diseases. 

Cerebral organoids reveal early cortical maldevelopment in schizophrenia — computational anatomy and genomics, role of FGFR1

E. K. Stachowiak, C. A. Benson, S. T. Narla, A. Dimitri, L. Chuye, S. Dhiman, K. Harikrishnan, S. Elahi, D. Freedman, K. Brennard, P. Sarder, and M. K. Stachowiak
Journal Paper [18] Translational Psychiatry - Nature, vol. 7, no. 6, pp. 1–24, Nov. 2017.


Studies of induced pluripotent stem cells (iPSCs) from schizophrenia patients and control individuals revealed that the disorder is programmed at the preneuronal stage, involves a common dysregulated mRNA transcriptome, and identified Integrative Nuclear FGFR1 Signaling a common dysregulated mechanism.  We used human embryonic stem cell (hESC) and iPSC-derived cerebral organoids from four controls and three schizophrenia patients to model the first trimester of in utero brain development.  The schizophrenia organoids revealed an abnormal scattering of proliferating Ki67+ neural progenitor cells (NPCs) from the ventricular zone (VZ), throughout the intermediate (IZ) and cortical (CZ) zones.  TBR1 pioneer neurons and reelin, which guides cortico-petal migration, were restricted from the schizophrenia cortex.  The maturing neurons were abundantly developed in the subcortical regions, but were depleted from the schizophrenia cortex.  The decreased intracortical connectivity was denoted by changes in the orientation and morphology of calretinin interneurons.  In schizophrenia organoids, nuclear (n)FGFR1 was abundantly expressed by developing subcortical cells, but was depleted from the neuronal committed cells (NCCs) of the CZ.  Transfection of dominant negative and constitutively active nFGFR1 caused widespread disruption of the neuro-ontogenic gene networks in hESC-derived NPCs and NCCs.  The fgfr1 gene was the most prominent FGFR gene expressed in NPCs and NCCs, and blocking with PD173074 reproduced both the loss of nFGFR1 and cortical neuronal maturation in hESC cerebral organoids.  We report for the first time, progression of the cortical malformation in schizophrenia and link it to altered FGFR1 signaling.  Targeting INFS may offer a preventive treatment of schizophrenia. 

Common developmental genome deprogramming in schizophrenia — Role of Integrative Nuclear FGFR1 Signaling (INFS)

S. T. Narla, Y-W. Lee, C. A. Benson, P. Sarder, K. J. Brennand, E. K. Stachowiak, and M. K. Stachowiak
Journal Paper [17] Schizophrenia Research, vol. 185, pp. 17–32, Jul. 2017.


The watershed-hypothesis of schizophrenia asserts that over 200 different mutations dysregulate distinct pathways that converge on an unspecified common mechanism(s) that controls disease ontogeny.  Consistent with this hypothesis, our RNA-sequencing of neuron committed cells (NCCs) differentiated from established iPSCs of 4 schizophrenia patients and 4 control subjects uncovered a dysregulated transcriptome of 1349 mRNAs common to all patients.  Data reveals a global dysregulation of developmental genome, deconstruction of coordinated mRNA networks, and the formation of aberrant, new coordinated mRNA networks indicating a concerted action of the responsible factor(s).  Sequencing of miRNA transcriptomes demonstrated an overexpression of 16 miRNAs and deconstruction of interactive miRNA–mRNA networks in schizophrenia NCCs.  ChiPseq revealed that the nuclear (n) form of FGFR1, a pan-ontogenic regulator, is overexpressed in schizophrenia NCCs and overtargets dysregulated mRNA and miRNA genes.  The nFGFR1 targeted 54% of all human gene promoters and 84.4% of schizophrenia dysregulated genes.  The upregulated genes reside within major developmental pathways that control neurogenesis and neuron formation, whereas downregulated genes are involved in oligodendrogenesis.  Our results indicate (i) an early (preneuronal) genomic etiology of schizophrenia, (ii) dysregulated genes and new coordinated gene networks are common to unrelated cases of schizophrenia, (iii) gene dysregulations are accompanied by increased nFGFR1-genome interactions, and (iv) modeling of increased nFGFR1 by an overexpression of a nFGFR1 lead to up or downregulation of selected genes as observed in schizophrenia NCCs.  Together our results designate nFGFR1 signaling as a potential common dysregulated mechanism in investigated patients and potential therapeutic target in schizophrenia. 

Leveraging unsupervised training sets for multi-scale compartmentalization in renal pathology

B. Lutnick, J. E. Tomaszewski, and P. Sarder
Conference Paper [12] Proc. of SPIE–Medical Imaging 2017: Digital Pathology, vol. 10140, pp. 101400I:1–7, Orlando, Florida, USA, Feb. 2017.


Clinical pathology relies on manual compartmentalization and quantification of biological structures, which is time consuming and often error-prone.  Application of computer vision segmentation algorithms to histopathological image analysis, in contrast, can offer fast, reproducible, and accurate quantitative analysis to aid pathologists.  Algorithms tunable to different biologically relevant structures can allow accurate, precise, and reproducible estimates of disease states.  In this direction, we have developed a fast, unsupervised computational method for simultaneously separating all biologically relevant structures from histopathological images in multi-scale.  Segmentation is achieved by solving an energy optimization problem.  Representing the image as a graph, nodes (pixels) are grouped by minimizing a Potts model Hamiltonian, adopted from theoretical physics, modeling interacting electron spins.  Pixel relationships (modeled as edges) are used to update the energy of the partitioned graph.  By iteratively improving the clustering, the optimal number of segments is revealed.  To reduce computational time, the graph is simplified using a Cantor pairing function to intelligently reduce the number of included nodes.  The classified nodes are then used to train a multiclass support vector machine to apply the segmentation over the full image.  Accurate segmentations of images with as many as 106 pixels can be completed only in 5 sec, allowing for attainable multi-scale visualization.  To establish clinical potential, we employed our method in renal biopsies to quantitatively visualize for the first time scale variant compartments of heterogeneous intra- and extraglomerular structures simultaneously.  Implications of the utility of our method extend to fields such as oncology, genomics, and non-biological problems.

Automatic computational labeling of glomerular textural boundaries

B. Ginley, J. E. Tomaszewski, and P. Sarder
Conference Paper [11] Proc. of SPIE–Medical Imaging 2017: Digital Pathology, vol. 10140, pp. 101400G:1–7, Orlando, Florida, USA, Feb. 2017.


The glomerulus, a specialized bundle of capillaries, is the blood filtering unit of the kidney.  Each human kidney contains about 1 million glomeruli.  Structural damages in the glomerular micro-compartments give rise to several renal conditions; most severe of which is proteinuria, where excessive blood proteins flow freely to the urine.  The sole way to confirm glomerular structural damage in renal pathology is by examining histopathological or immunofluorescence stained needle biopsies under a light microscope.  However, this method is extremely tedious and time consuming, and requires manual scoring on the number and volume of structures.  Computational quantification of equivalent features promises to greatly ease this manual burden.  The largest obstacle to computational quantification of renal tissue is the ability to recognize complex glomerular textural boundaries automatically.  Here we present a computational pipeline to accurately identify glomerular boundaries with high precision and accuracy.  The computational pipeline employs an integrated approach composed of Gabor filtering, Gaussian blurring, statistical F-testing, and distance transform, and performs significantly better than standard Gabor based textural segmentation method.  Our integrated approach provides mean accuracy/precision of 0.89/0.97 on n = 200 Hematoxylin and Eosin (H&E) glomerulus images, and mean 0.88/0.94 accuracy/precision on n = 200 Periodic Acid Schiff (PAS) glomerulus images.  Respective accuracy/precision of the Gabor filter bank based method is 0.83/0.84 for H&E and 0.78/0.8 for PAS.  Our method will simplify computational partitioning of glomerular micro-compartments hidden within dense textural boundaries.  Automatic quantification of glomeruli will streamline structural analysis in clinic, and can help realize real time diagnoses and interventions.

Identification and characterization of neutrophil extracellular trap shapes in flow cytometry

B. Ginley, T. Emmons, P. Sasankan, C. Urban, B. H. Segal, and P. Sarder
Conference Paper [10] Proc. of SPIE–Medical Imaging 2017: Digital Pathology, vol. 10140, pp. 101400D:1–7, Orlando, Florida, USA, Feb. 2017.


Neutrophil extracellular trap (NET) formation is an alternate immunologic weapon used mainly by neutrophils.  Chromatin backbones fused with proteins derived from granules are shot like projectiles onto foreign invaders.  It is thought that this mechanism is highly anti-microbial, aids in preventing bacterial dissemination, is used to break down structures several sizes larger than neutrophils themselves, and may have several more uses yet unknown.  NETs have been implied to be involved in a wide array of systemic host immune defenses, including sepsis, autoimmune diseases, and cancer.  Existing methods used to visually quantify NETotic versus non-NETotic shapes are extremely time-consuming and subject to user bias.  These limitations are obstacles to developing NETs as prognostic biomarkers and therapeutic targets.  We propose an automated pipeline for quantitatively detecting neutrophil and NET shapes captured using a flow cytometry-imaging system.  Our method uses contrast limited adaptive histogram equalization to improve signal intensity in dimly illuminated NETs.  From the contrast improved image, fixed value thresholding is applied to convert the image to binary.  Feature extraction is performed on the resulting binary image, by calculating region properties of the resulting foreground structures.  Classification of the resulting features is performed using Support Vector Machine.  Our method classifies NETs from neutrophils without traps at 0.97/0.96 sensitivity/specificity on n = 387 images, and is 1500X faster than manual classification, per sample.  Our method can be extended to rapidly analyze whole-slide immunofluorescence tissue images for NET classification, and has potential to streamline the quantification of NETs for patients with diseases associated with cancer and autoimmunity. 

Unsupervised labeling of glomerular boundaries using Gabor filters and statistical testing in renal histology

B. Ginley, J. E. Tomaszewski, R. Yacoub, F. Chen, and P. Sarder
Journal Paper [16] Journal of Medical Imaging - SPIE, vol. 4, no. 2, pp. 021102:1–13, Feb. 2017.


The glomerulus is the blood filtering unit of the kidney.  Each human kidney contains ~1 million glomeruli.  Several renal conditions originate from structural damage to glomerular microcompartments, such as proteinuria, the excessive loss of blood proteins into urine.  The gold standard for evaluating structural damage in renal pathology is histopathological and immunofluorescence examination of needle biopsies under a light microscope.  This method is limited by qualitative or semiquantitative manual scoring approaches to the evaluation of glomerular structural features.  Computational quantification of equivalent features promises to improve the precision of glomerular structural analysis.  One large obstacle to the computational quantification of renal tissue is the identification of complex glomerular boundaries automatically.  To mitigate this issue, we developed a computational pipeline capable of extracting and exactly defining glomerular boundaries.  Our method, composed of Gabor filtering, Gaussian blurring, statistical F-testing, and distance transform, is able to accurately identify glomerular boundaries with mean sensitivity/specificity of 0.88/0.96 and accuracy of 0.92, on n = 1000 glomeruli images stained with standard renal histological stains.  Our method will simplify computational partitioning of glomerular microcompartments hidden within dense textural boundaries.  Automatic quantification of glomeruli will streamline structural analysis in clinic and can pioneer real-time diagnoses and interventions for renal care. 

Automated renal histopathology: Digital extraction and quantification of renal pathology

P. Sarder*, B. Ginley*, and J. E. Tomaszewski
Conference Paper [9] Proc. of SPIE–Medical Imaging 2016: Digital Pathology, vol. 9791, pp. 97910F:1–12, San Diego, California, USA, Mar. 2016.


The branch of pathology concerned with excess blood serum proteins being excreted in the urine pays particular attention to the glomerulus, a small intertwined bunch of capillaries located at the beginning of the nephron.  Normal glomeruli allow moderate amount of blood proteins to be filtered; proteinuric glomeruli allow large amount of blood proteins to be filtered.  Diagnosis of proteinuric diseases requires time intensive manual examination of the structural compartments of the glomerulus from renal biopsies.  Pathological examination includes cellularity of individual compartments, Bowman's and luminal space segmentation, cellular morphology, glomerular volume, capillary morphology, and more.  Long examination times may lead to increased diagnosis time and/or lead to reduced precision of the diagnostic process.  Automatic quantification holds strong potential to reduce renal diagnostic time.  We have developed a computational pipeline capable of automatically segmenting relevant features from renal biopsies.  Our method first segments glomerular compartments from renal biopsies by isolating regions with high nuclear density.  Gabor texture segmentation is used to accurately define glomerular boundaries.  Bowman's and luminal spaces are segmented using morphological operators.  Nuclei structures are segmented using color deconvolution, morphological processing, and bottleneck detection.  Average computation time of feature extraction for a typical biopsy, comprising of ~12 glomeruli, is ~69 s using an Intel(R) Core(TM) i7-4790 CPU, and is ~65X faster than manual processing.  Using images from rat renal tissue samples, automatic glomerular structural feature estimation was reproducibly demonstrated for 15 biopsy images, which contained 148 individual glomeruli images.  The proposed method holds immense potential to enhance information available while making clinical diagnoses.

Protonation and trapping of a small pH-sensitive near-infrared fluorescent molecule in acidic tumor environment delineates diverse tumors in vivo

R. Gilson, R. Tang, A. Som, C. Klajer, P. Sarder, G. Sudlow, W. Akers, and S. Achilefu
Journal Paper [15] Molecular Pharmaceutics, vol. 12, no. 12, pp. 4237–4246, Oct. 2015.


Enhanced glycolysis and poor perfusion in most solid malignant tumors create an acidic extracellular environment, which enhances tumor growth, invasion, and metastasis.  Complex molecular systems have been explored for imaging and treating these tumors.  Here, we report the development of a small molecule, LS662, that emits near-infrared (NIR) fluorescence upon protonation by the extracellular acidic pH environment of diverse solid tumors.  Protonation of LS662 induces selective internalization into tumor cells and retention in the tumor microenvironment.  Noninvasive NIR imaging demonstrates selective retention of the pH sensor in diverse tumors, and two-photon microscopy of ex vivo tumors reveals significant retention of LS662 in tumor cells and the acid tumor microenvironment.  Passive and active internalization processes combine to enhance NIR fluorescence in tumors over time.  The low background fluorescence allows tumors to be detected with high sensitivity, as well as dead or dying cells to be delineated from healthy cells.  In addition to demonstrating the feasibility of using small molecule pH sensors to image multiple aggressive solid tumor types via a protonation-induced internalization and retention pathway, the study reveals the potential of using LS662 to monitor treatment response and tumor-targeted drug delivery.

Evaluation of dynamic optical projection of acquired luminescence for sentinel lymph node biopsy in large animals

E. Ringhausen, T. Wang, J. Pitts, P. Sarder, and W. Akers
Journal Paper [14] Technology in Cancer Research & Treatment, pp. 1533034615604978:1–9, Sep. 2015.


Open surgery requiring cytoreduction still remains the primary treatment course for many cancers.  The extent of resection is vital for the outcome of surgery, greatly affecting patients' follow-up treatment including need for revision surgery in the case of positive margins, choice of chemotherapy, and overall survival.  Existing imaging modalities such as computed tomography, magnetic resonance imaging, and positron emission tomography are useful in the diagnostic stage and long-term monitoring but do not provide the level of temporal or spatial resolution needed for intraoperative surgical guidance.  Surgeons must instead rely on visual evaluation and palpation in order to distinguish tumors from surrounding tissues.  Fluorescence imaging provides high-resolution, real-time mapping with the use of a contrast agent and can greatly enhance intraoperative imaging.  Here we demonstrate an intraoperative, real-time fluorescence imaging system for direct highlighting of target tissues for surgical guidance, optical projection of acquired luminescence (OPAL).  Image alignment, accuracy, and resolution was determined in vitro prior to demonstration of feasibility for operating room use in large animal models of sentinel lymph node biopsy.  Fluorescence identification of regional lymph nodes after intradermal injection of indocyanine green was performed in pigs with surgical guidance from the OPAL system.  Acquired fluorescence images were processed and rapidly reprojected to highlight indocyanine green within the true surgical field.  OPAL produced enhanced visualization for resection of lymph nodes at each anatomical location.  Results show the optical projection of acquired luminescence system can successfully use fluorescence image capture and projection to provide aligned image data that is invisible to the human eye in the operating room setting.

Molecular probes for fluorescence lifetime imaging

P. Sarder, D. Maji, and S. Achilefu
Journal Paper [13] Bioconjugate Chemistry, vol. 26, no. 6, pp. 963–974, May 2015.


Visualization of biological processes and pathologic conditions at the cellular and tissue levels largely relies on the use of fluorescence intensity signals from fluorophores or their bioconjugates.  To overcome the concentration dependency of intensity measurements, evaluate subtle molecular interactions, and determine biochemical status of intracellular or extracellular microenvironments, fluorescence lifetime (FLT) imaging has emerged as a reliable imaging method complementary to intensity measurements.  Driven by a wide variety of dyes exhibiting stable or environment-responsive FLTs, information multiplexing can be readily accomplished without the need for ratiometric spectral imaging.  With knowledge of the fluorescent states of the molecules, it is entirely possible to predict the functional status of biomolecules or microevironment of cells.  Whereas the use of FLT spectroscopy and microscopy in biological studies is now well-established, in vivo imaging of biological processes based on FLT imaging techniques is still evolving.  This review summarizes recent advances in the application of the FLT of molecular probes for imaging cells and small animal models of human diseases.  It also highlights some challenges that continue to limit the full realization of the potential of using FLT molecular probes to address diverse biological problems and outlines areas of potential high impact in the future.

Community detection for fluorescent lifetime microscopy image segmentation

D. Hu*, P. Sarder*, P. Ronhovde, S. Achilefu, and Z. Nussinov
Conference Paper [8] Proc. of SPIE–Three-Dimensional and Multidimensional Microscopy: Image Acquisition and Processing XXI, vol. 8949, pp. 89491K:1–13, San Francisco, California, USA, Feb. 2014.


Multiresolution community detection (CD) method has been suggested in a recent work as an efficient method for performing unsupervised segmentation of fluorescence lifetime (FLT) images of live cell images containing fluorescent molecular probes.  In the current paper, we further explore this method in FLT images of ex vivo tissue slices.  The image processing problem is framed as identifying clusters with respective average FLTs against a background or solvent in FLT imaging microscopy (FLIM) images derived using NIR fluorescent dyes.  We have identified significant multiresolution structures using replica correlations in these images, where such correlations are manifested by information theoretic overlaps of the independent solutions (replicas) attained using the multiresolution CD method from different starting points.  In this paper, our method is found to be more efficient than a current state-of-the-art image segmentation method based on mixture of Gaussian distributions. It offers more than 1:25 times diversity based on Shannon index than the latter method, in selecting clusters with distinct average FLTs in NIR FLIM images.

Quantitative determination of maximal imaging depth in all-NIR multiphoton microscopy images of thick tissues

P. Sarder, W. Akers, G. Sudlow, S. Yazdanfar, and S. Achilefu
Conference Paper [7] Proc. of SPIE–Multiphoton Microscopy in the Biomedical Sciences XIV, vol. 8948, pp. 894827:1–7, San Francisco, California, USA, Feb. 2014.


We report two methods for quantitatively determining maximal imaging depth from thick tissue images captured using all–near-infrared (NIR) multiphoton microscopy (MPM).  All-NIR MPM is performed using 1550 nm laser excitation with NIR detection.  This method enables imaging more than five-fold deep in thick tissues in comparison with other NIR excitation microscopy methods.  In this study, we show a correlation between the multiphoton signal along the depth of tissue samples and the shape of the corresponding empirical probability density function (pdf) of the photon counts.  Histograms from this analysis become increasingly symmetric with the imaging depth.  This distribution transitions toward the background distribution at higher imaging depths.  Inspired by these observations, we propose two independent methods based on which one can automatically determine maximal imaging depth in the all-NIR MPM images of thick tissues.  At this point, the signal strength is expected to be weak and similar to the background.  The first method suggests the maximal imaging depth corresponds to the deepest image plane where the ratio between the mean and median of the empirical photon-count pdf is outside the vicinity of 1.  The second method suggests the maximal imaging depth corresponds to the deepest image plane where the squared distance between the empirical photon-count mean obtained from the object and the mean obtained from the background is greater than a threshold.  We demonstrate the application of these methods in all-NIR MPM images of mouse kidney tissues to study maximal depth penetration in such tissues.

Near-infrared fluorescence quenching properties of copper (II) ions for potential applications in biological imaging

D. Maji, M. Zhou, P. Sarder, and S. Achilefu
Conference Paper [6] Proc. of SPIE–Reporters, Markers, Dyes, Nanoparticles, and Molecular Probes for Biomedical Applications, vol. 8956, pp. 89560K:1–6, San Francisco, California, USA, Feb. 2014.


Fluorescence quenching properties of copper(II) ions have been used for designing Cu(II) sensitive fluorescent molecular probes.  In this paper, we demonstrate that static quenching plays a key role in free Cu(II)-mediated fluorescence quenching of a near infrared (NIR) fluorescent dye cypate.  The Stern-Volmer quenching constant was calculated to be KSV = 970,000 M-1 in 25 mM MES buffer, pH 6.5 at room temperature.  We synthesized LS835, a compound containing cypate attached covalently to chelated Cu(II) to study fluorescence quenching by chelated Cu(II).  The fluorescence quenching mechanism of chelated Cu(II) is predominantly dynamic or collisional quenching.  The quenching efficiency of chelated Cu(II) was calculated to be 58%±6% in dimethylsulfoxide at room temperature.  Future work will involve further characterization of the mechanism of NIR fluorescence quenching by Cu(II) and testing its reversibility for potential applications in designing fluorophore-quencher based molecular probes for biological imaging.

Simultaneous detection of multiple biological targets using optimized microfluidic microsphere-trap arrays

X. Xu, Z. Li, P. Sarder, N. Kotagiri, and A. Nehorai
Journal Paper [12] Journal of Micro/Nanolithography, MEMS, and MOEMS, vol. 13, no. 1, pp. 13017:1–10, Jan. 2014.


We propose an analytical framework to build a microfluidic microsphere-trap array device that enables simultaneous, efficient, and accurate screening of multiple biological targets in a single microfluidic channel.  By optimizing the traps' geometric parameters, the trap arrays in the channel of the device can immobilize microspheres of different sizes at different regions, obeying hydrodynamically engineered trapping mechanism.  Different biomolecules can be captured by the ligands on the surfaces of microspheres of different sizes.  They are thus detected according to the microspheres' positions (position encoding), which simplifies screening and avoids target identification errors.  To demonstrate the proposition, we build a device for simultaneous detection of two target types by trapping microspheres of two sizes.  We evaluate the device performance using finite element fluidic dynamics simulations and microsphere-trapping experiments.  These results validate that the device efficiently achieves position encoding of the two-sized microspheres with few fluidic errors, providing the promise to utilize our framework to build devices for simultaneous detection of more targets.  We also envision utilizing the device to separate, sort, or enumerate cells, such as circulating tumor cells and blood cells, based on cell size and deformability.  Therefore, the device is promising to become a cost-effective and point-of-care miniaturized disease diagnostic tool.

Automatic segmentation of fluorescence lifetime microscopy images of cells using multiresolution community detection–a first study

D. Hu*, P. Sarder*, P. Ronhovde, S. Orthaus, S. Achilefu, and Z. Nussinov
Journal Paper [11] Journal of Microscopy, vol. 253, no. 1, pp. 54–64, Jan. 2014.


Inspired by a multiresolution community detection based network segmentation method, we suggest an automatic method for segmenting fluorescence lifetime (FLT) imaging microscopy (FLIM) images of cells in a first pilot investigation on two selected images.  The image processing problem is framed as identifying segments with respective average FLTs against the background in FLIM images.  The proposed method segments a FLIM image for a given resolution of the network defined using image pixels as the nodes and similarity between the FLTs of the pixels as the edges.  In the resulting segmentation, low network resolution leads to larger segments, and high network resolution leads to smaller segments.  Furthermore, using the proposed method, the mean-square error in estimating the FLT segments in a FLIM image was found to consistently decrease with increasing resolution of the corresponding network.  The multiresolution community detection method appeared to perform better than a popular spectral clustering-based method in performing FLIM image segmentation.  At high resolution, the spectral segmentation method introduced noisy segments in its output, and it was unable to achieve a consistent decrease in mean-square error with increasing resolution.

Dynamic optical projection of acquired luminescence for aiding oncologic surgery

P. Sarder, K. Gullicksrud, S. Mondal, G. Sudlow, S. Achilefu, and W. Akers
Journal Paper [10] Journal of Biomedical Optics, vol. 253, no. 12, pp. 12501:1–3, Dec. 2013. [Selected as one of the top 10 most downloaded articles on the journal's webpage (Mar. 2014 to Apr. 2014).]


Optical imaging enables real-time visualization of intrinsic and exogenous contrast within biological tissues.  Applications in human medicine have demonstrated the power of fluorescence imaging to enhance visualization in dermatology, endoscopic procedures, and open surgery.  Although few optical contrast agents are available for human medicine at this time, fluorescence imaging is proving to be a powerful tool in guiding medical procedures.  Recently, intraoperative detection of fluorescent molecular probes that target cell-surface receptors has been reported for improvement in oncologic surgery in humans.  We have developed a novel system, optical projection of acquired luminescence (OPAL), to further enhance real-time guidance of open oncologic surgery.  In this method, collected fluorescence intensity maps are projected onto the imaged surface rather than via wall-mounted display monitor.  To demonstrate proof-of-principle for OPAL applications in oncologic surgery, lymphatic transport of indocyanine green was visualized in live mice for intraoperative identification of sentinel lymph nodes.  Subsequently, peritoneal tumors in a murine model of breast cancer metastasis were identified using OPAL after systemic administration of a tumor-selective fluorescent molecular probe.  These initial results clearly show that OPAL can enhance adoption and ease-of-use of fluorescence imaging in oncologic procedures relative to existing state-of-the-art intraoperative imaging systems.

All–near-infrared multiphoton microscopy interrogates intact tissues at deeper imaging depths than conventional single and two photon near-infrared excitation microscopes

P. Sarder*, S. Yazdanfar*, W. Akers, R. Tang, G. Sudlow, C. Egbulefu, and S. Achilefu
Journal Paper [9] Journal of Biomedical Optics, vol. 18, no. 10, pp. 106012:1–11, Oct. 2013. [Selected as one of the top 10 most downloaded articles on the journal's webpage (Nov. 2013 to Feb. 2014).]


The era of molecular medicine has ushered in the development of microscopic methods that can report molecular processes in thick tissues with high spatial resolution.  A commonality in deep-tissue microscopy is the use of near-infrared (NIR) lasers with single- or multiphoton excitations.  However, the relationship between different NIR excitation microscopic techniques and the imaging depths in tissue has not been established.  We compared such depth limits for three NIR excitation techniques: NIR single-photon confocal microscopy (NIR SPCM), NIR multiphoton excitation with visible detection (NIR/VIS MPM), and all-NIR multiphoton excitation with NIR detection (NIR/NIR MPM).  Homologous cyanine dyes provided the fluorescence.  Intact kidneys were harvested after administration of kidney-clearing cyanine dyes in mice.  NIR SPCM and NIR/VIS MPM achieved similar maximum imaging depth of ~100 μm.  The NIR/NIR MPM enabled greater than fivefold imaging depth (>500 μm) using the harvested kidneys.  Although the NIR/NIR MPM used 1550-nm excitation where water absorption is relatively high, cell viability and histology studies demonstrate that the laser did not induce photothermal damage at the low laser powers used for the kidney imaging.  This study provides guidance on the imaging depth capabilities of NIR excitation-based microscopic techniques and reveals the potential to multiplex information using these platforms.

Performance analysis and design of position-encoded microsphere arrays using the Ziv-Zakai bound

X. Xu, P. Sarder, N. Kotagiri, S. Achilefu, and A. Nehorai
Journal Paper [8] IEEE Trans. on NanoBioscience, vol. 12, no. 1, pp. 29–40, Mar. 2013.


Position-encoded microsphere arrays are a promising technology for identifying biological targets and quantifying their concentrations.  In this paper we analyze the statistical performance of these arrays in imaging targets at typical low signal-to-noise ratio (SNR) levels.  We compute the Ziv-Zakai bound (ZZB) on the errors in estimating the unknown parameters, including the target concentrations.  We find the SNR level below which the ZZB provides a more accurate prediction of the error than the posterior Cramér-Rao bound (PCRB), through numerical examples.  We further apply the ZZB to select the optimal design parameters of the microsphere array device and investigate the effects of the experimental variables such as microscope point-spread function.  An imaging experiment on microspheres with protein targets verifies the optimal design parameters using the ZZB.

Optimization of microfluidic microsphere-trap arrays

X. Xu*, P. Sarder*, Z. Li*, and A. Nehorai
Journal Paper [7] Biomicrofluidics, vol. 7, no. 1, pp. 014112:1–16, Feb. 2013. [Selected among the 20 most cited articles (2014); Selected for the Research Highlights on the journal's webpage, and also selected as one of the top 20 most downloaded articles (Mar. 2013 to Sep. 2013); Selected among the 10 best "Editors' Pick" articles (2012-13).]


Microarray devices are powerful for detecting and analyzing biological targets.  However, the potential of these devices may not be fully realized due to the lack of optimization of their design and implementation.  In this work, we consider a microsphere-trap array device by employing microfluidic techniques and a hydrodynamic trapping mechanism.  We design a novel geometric structure of the trap array in the device, and develop a comprehensive and robust framework to optimize the values of the geometric parameters to maximize the microsphere arrays' packing density.  We also simultaneously optimize multiple criteria, such as efficiently immobilizing a single microsphere in each trap, effectively eliminating fluidic errors such as channel clogging and multiple microspheres in a single trap, minimizing errors in subsequent imaging experiments, and easily recovering targets.  We use finite element simulations to validate the trapping mechanism of the device, and to study the effects of the optimization geometric parameters.  We further perform microsphere-trapping experiments using the optimized device and a device with randomly selected geometric parameters, which we denote as the un-optimized device.  These experiments demonstrate easy control of the transportation and manipulation of the microspheres in the optimized device.  They also show that the optimized device greatly outperforms the un-optimized device by increasing the packing density by a factor of two, improving the microsphere trapping efficiency from 58% to 99%, and reducing fluidic errors from 48% to a negligible level (less than 1%).  The optimization framework lays the foundation for the future goal of developing a modular, reliable, efficient, and inexpensive lab-on-a-chip system.

Optimization of microfluidic trap-based microsphere arrays

X. Xu*, P. Sarder*, Z. Li*, and A. Nehorai
Conference Paper [5] Proc. of SPIE–Microfluidics, BioMEMS, and Medical Microsystems XI, vol. 8615, pp. 86150V:1–12, San Francisco, California, USA, Feb. 2013.


We build a microfluidic trap-based microsphere array device.  In the device, we design a novel geometric structure of the trap array and employ the hydrodynamic trapping mechanism to immobilize the microspheres.  We develop a comprehensive and robust framework to optimize the values of the geometric parameters to maximize the microsphere arrays' packing density.  We also simultaneously optimize multiple criteria, such as efficiently immobilizing a single microsphere in each trap, effectively eliminating fluidic errors such as channel clogging and multiple microspheres in a single trap, minimizing errors in subsequent imaging experiments, and easily recovering targets.  Microsphere-trapping experiments have been performed using the optimized device and a device with un-optimized geometric parameters.  These experiments demonstrate easy control of the transportation and manipulation of the microspheres in the optimized device.  They also show that the optimized device greatly outperforms the un-optimized one.

Microfluidic microsphere-trap arrays for simultaneous detection of multiple targets

X. Xu, Z. Li, N. Kotagiri, P. Sarder, S. Achilefu, and A. Nehorai
Conference Paper [4] Proc. of SPIE–Microfluidics, BioMEMS, and Medical Microsystems XI, vol. 8615, pp. 86151E:1–11, San Francisco, California, USA, Feb. 2013.


Microsphere arrays can be used to effectively detect, identify, and quantify biological targets, such as mRNAs, proteins, antibodies, and cells.  In this work, we design a microfluidic microsphere-trap array device that enables simultaneous, efficient, and accurate screening of multiple targets on a single platform.  Different types of targets are captured on the surfaces of microspheres of different sizes.  By optimizing the geometric parameters of the traps, the trap arrays in this device can immobilize microspheres of different sizes at different regions with microfluidic hydrodynamic trapping.  The targets are thus detected according to the microspheres' positions (position-encoding), which simplifies screening and avoids errors in target identification.  We validate the design using fluid dynamics finite element simulations by COMSOL Multiphysics software using microsphere of two different sizes.  We also performed preliminary microspheretrapping experiments on a fabricated device using microspheres of one size.  Our results demonstrate that the proposed device can achieve the position-encoding of the microspheres with few fluidic errors.  This device is promising for simultaneous detection of multiple targets and become a cheap and fast disease diagnostic tool.

Fluorescence lifetime imaging microscopy using near-infrared contrast agents

R. Nothdurft*, P. Sarder*, S. Bloch, J. Culver, and S. Achilefu
Journal Paper [6] Journal of Microscopy, vol. 247, no. 2, pp. 202–207, Aug. 2012.


Although single-photon fluorescence lifetime imaging microscopy (FLIM) is widely used to image molecular processes using a wide range of excitation wavelengths, the captured emission of this technique is confined to the visible spectrum.  Here, we explore the feasibility of utilizing near-infrared (NIR) fluorescent molecular probes with emission >700 nm for FLIM of live cells.  The confocal microscope is equipped with a 785 nm laser diode, a red-enhanced photomultiplier tube, and a time-correlated single photon counting card.  We demonstrate that our system reports the lifetime distributions of NIR fluorescent dyes, cypate and DTTCI, in cells.  In cells labelled separately or jointly with these dyes, NIR FLIM successfully distinguishes their lifetimes, providing a method to sort different cell populations.  In addition, lifetime distributions of cells co-incubated with these dyes allow estimate of the dyes' relative concentrations in complex cellular microenvironments.  With the heightened interest in fluorescence lifetime-based small animal imaging using NIR fluorophores, this technique further serves as a bridge between in vitro spectroscopic characterization of new fluorophore lifetimes and in vivo tissue imaging.

Statistical design of position-encoded microsphere arrays at low target concentrations

X. Xu, P. Sarder, and A. Nehorai
Conference Paper [3] 45th Asilomar Conf. on Signals, Systems, and Computers, pp. 1694–1698, Pacific Grove, California, USA, Nov. 2011.


We design microsphere arrays with predetermined positions of microspheres, for capturing targets at low concentrations.  To optimize the design parameters, we compute the Ziv-Zakai bound (ZZB) on the errors in estimating the target concentrations.  We numerically demonstrate our design by computing the minimal distance between the microspheres and the optimal imaging temperature, for a desired level of errors.  We also validate that, at low target concentrations, the statistical design using the ZZB is more precise than that using the posterior Cramér-Rao bound.  We further quantitatively evaluate the effect of the fluorescence microscope point-spread function on the design performance, which provide useful guides to the device design and implementation.  The key advantages of the proposed microsphere arrays are error-free target identification, simplified data analysis, high packing density, and reduced cost.

Statistical design of position-encoded microsphere arrays

P. Sarder and A. Nehorai
Journal Paper [5] IEEE Trans. on NanoBioscience, vol. 10, no. 1, pp. 16–29, Mar. 2011.


We propose a microsphere array device with microspheres having controllable positions for error-free target identification.  We conduct a statistical design analysis to select the optimal distance between the microspheres as well as the optimal temperature.  Our design simplifies the imaging and ensures a desired statistical performance for a given sensor cost.  Specifically, we compute the posterior Cramér-Rao bound on the errors in estimating the unknown target concentrations.  We use this performance bound to compute the optimal design variables.  We discuss both uniform and sparse concentration levels of targets, and replace the unknown imaging parameters with their maximum likelihood estimates.  We illustrate our design concept using numerical examples.  The proposed microarray has high sensitivity, efficient packing, and guaranteed imaging performance.  It simplifies the imaging analysis significantly by identifying targets based on the known positions of the microspheres.  Potential applications include molecular recognition, specificity of targeting molecules, protein-protein dimerization, high throughput screening assays for enzyme inhibitors, drug discovery, and gene sequencing.

Gene reachability using Page ranking on gene co-expression networks

P. Sarder, W. Zhang, J. P. Cobb, and A. Nehorai
Book Chapter [1] Link Mining: Models, Algorithms, and Applications | Ch. 21 | pp. 557–568 | P. S. Yu, J. Han, and C. Faloutsos, Eds. | Springer, New York | 2010.


We modify the Google Page-Rank algorithm, which is primarily used for ranking web pages, to analyze the gene reachability in complex gene co-expression networks.  Our modification is based on the average connections per gene.  We propose a new method to compute the metric of average connections per gene, inspired by the Page-Rank algorithm.  We calculate this average as eight for human genome data and three to seven for yeast genome data.  Our algorithm provides clustering of genes.  The proposed analogy between web pages and genes may offer a new way to interpret gene networks.

Estimating sparse gene regulatory networks using a Bayesian linear regression

P. Sarder, W. Schierding, J. P. Cobb, and A. Nehorai
Journal Paper [4] IEEE Trans. on NanoBioscience, vol. 9, no. 2, pp. 121–131, June 2010.


In this paper, we propose a gene regulatory network (GRN) estimation method, which assumes that such networks are typically sparse, using time-series microarray datasets.  We represent the regulatory relationships between the genes using weights, with the "net" regulation influence on a gene's expression being the summation of the independent regulatory inputs.  We estimate the weights using a Bayesian linear regression method for sparse parameter vectors.  We apply our proposed method to the extraction of differential gene expression software selected genes of a human buffy-coat microarray expression profile dataset of ventilator-associated pneumonia (VAP), and compare the estimation result with the GRNs estimated using both a correlation coefficient method and a database-based method ingenuity pathway analysis.  A biological analysis of the resulting consensus network that is derived using the GRNs, estimated with both our and the correlation-coefficient methods results in four biologically meaningful subnetworks.  Also, our method performs either better than or competitively with the existing well-established GRN estimation methods.  Moreover, it performs comparatively with respect to: 1) the ground-truth GRNs for the in silico 50- and 100-gene datasets reported recently in the DREAM3 challenge and 2) the GRN estimated using a mutual information-based method for the top-ranked Bayesian analysis of time series (a Bayesian user-friendly software for analyzing time-series microarray experiments) selected genes of the VAP dataset.

Statistical design of a 3D microarray with position-encoded microspheres

P. Sarder and A. Nehorai
Conference Paper [2] Proc. Third International Workshop on Computational Advances in Multi-Sensor Adaptive Processing (CAMSAP), pp. 161–164, Aruba, Dutch Antilles, Dec. 2009.


We propose a three-dimensional (3D) microarray device with fixed distance between the microspheres.  The microspheres in the new device have controllable positions.  To design the layout of our proposed device, we consider an optimal statistical performance analysis for imaging the microspheres.  We compute the posterior Cramér-Rao bound on the error in estimating the unknown imaging parameters and use the bound to compute the minimal distance between the microspheres, which provides a desired level of statistical imaging performance.  We illustrate our concept using a numerical example.  The proposed microarray has high sensitivity, efficient packing density, and guaranteed imaging performance.  It identifies targets based on the microspheres' locations, allowing significantly simplified imaging analysis.

Estimating locations of quantum-dot–encoded microparticles from ultra-high density 3D microarrays

P. Sarder and A. Nehorai
Journal Paper [3] IEEE Trans. on NanoBioscience, vol. 7, no. 4, pp. 284–297, Dec. 2008.


We develop a maximum likelihood (ML)-based parametric image deconvolution technique to locate quantum-dot (q-dot) encoded microparticles from three-dimensional (3-D) images of an ultra-high density 3-D microarray.  A potential application of the proposed microarray imaging is assay analysis of gene, protein, antigen, and antibody targets.  This imaging is performed using a wide-field fluorescence microscope.  We first describe our problem of interest and the pertinent measurement model by assuming additive Gaussian noise.  We use a 3-D Gaussian point-spread-function (PSF) model to represent the blurring of the widefield microscope system.  We employ parametric spheres to represent the light intensity profiles of the q-dot-encoded microparticles.  We then develop the estimation algorithm for the single-sphere-object image assuming that the microscope PSF is totally unknown.  The algorithm is tested numerically and compared with the analytical Cramér-Rao bounds (CRB).  To apply our analysis to real data, we first segment a section of the blurred 3-D image of the multiple microparticles using a k-means clustering algorithm, obtaining 3-D images of single-sphere-objects.  Then, we process each of these images using our proposed estimation technique.  In the numerical examples, our method outperforms the blind deconvolution (BD) algorithms in high signal-to-noise ratio (SNR) images.  For the case of real data, our method and the BD-based methods perform similarly for the well-separated microparticle images.

Estimating gene signals from noisy microarray images

P. Sarder, A. Nehorai, P. H. Davis, and S. Stanley
Journal Paper [2] IEEE Trans. on NanoBioscience, vol. 7, no. 2, pp. 142–153, June 2008.


In oligonucleotide microarray experiments, noise is a challenging problem, as biologists now are studying their organisms not in isolation but in the context of a natural environment.  In low photomultiplier tube (PMT) voltage images, weak gene signals and their interactions with the background fluorescence noise are most problematic.  In addition, nonspecific sequences bind to array spots intermittently causing inaccurate measurements.  Conventional techniques cannot precisely separate the foreground and the background signals.  In this paper, we propose analytically based estimation technique.  We assume a priori spot-shape information using a circular outer periphery with an elliptical center hole.  We assume Gaussian statistics for modeling both the foreground and background signals.  The mean of the foreground signal quantifies the weak gene signal corresponding to the spot, and the variance gives the measure of the undesired binding that causes fluctuation in the measurement.  We propose a foreground-signal and shape-estimation algorithm using the Gibbs sampling method.  We compare our developed algorithm with the existing Mann-Whitney (MW)- and expectation maximization (EM)/iterated conditional modes (ICM)-based methods.  Our method outperforms the existing methods with considerably smaller mean-square error (MSE) for all signal-to-noise ratios (SNRs) in computer-generated images and gives better qualitative results in low-SNR real-data images.  Our method is computationally relatively slow because of its inherent sampling operation and hence only applicable to very noisy-spot images.  In a realistic example using our method, we show that the gene-signal fluctuations on the estimated foreground are better observed for the input noisy images with relatively higher undesired bindings.

Performance analysis of quantifying fluorescence of target-captured microparticles from microscopy images

P. Sarder and A. Nehorai
Conference Paper [1] Proc. Fourth IEEE Workshop on Sensor Array and Multi-Channel Processing, pp. 289–293, Waltham, Massachusetts, USA, Jul. 2006.


Fluorescence microscopy imaging is widely used in biomedical research, astronomical speckle imaging, remote sensing, positron-emission tomography, and many other applications.  In companion papers, we developed a maximum likelihood (ML)-based image deconvolution technique to quantify fluorescence signals from a three-dimensional (3D) image of a target captured microparticle ensemble.  We assumed both the additive Gaussian and Poisson statistics for the noise.  Imaging is performed by using a confocal fluorescence microscope system.  Potential application of microarray technology includes security, environmental monitoring, analyzing assays for DNA or protein targets, functional genomics, and drug development.  We proposed a new parametric model of the fluorescence microscope 3D point-spread function (PSF) in terms of basis functions.  In this paper, we present a performance analysis of the ML-based deconvolution techniques for both the noise models.

Deconvolution methods for 3D fluorescence microscopy images: An overview

P. Sarder and A. Nehorai
Journal Paper [1] IEEE Signal Processing Magazine, vol. 23, no. 3, pp. 32–45, May 2006.


This paper presents an overview of various deconvolution techniques of 3D fluorescence microscopy images.  It describes the subject of image deconvolution for 3D fluorescence microscopy images and provides an overview of the distortion issues in different areas.  The paper presents a brief schematic description of fluorescence microscope systems and provides a summary of the microscope point-spread function (PSF), which often creates the most severe distortion in the acquired 3D image.  Finally, it discusses the ongoing research work in the area and provides a brief review of performance measures of 3D deconvolution microscopy techniques.  It also provides a summary of the numerical results using simulated data and presents the results obtained from the real data.