TR&D3: Image processing and analysis, with an emphasis on analysis of cell and tissue organization in support of modeling

FigIV.3Example synthetic 3D image for HeLa cells The image shows the nuclear membrane in red, the cell membrance in blue, and lysosomal membranes in green. It was randomly generated from a model learned from real 3D microscope images using the approach described in Peng and Murphy. (Peng,T. and R.F.Murphy. 2011. Image-derived, three-dimensional generative models of cellular organization. Cytometry A 79:383-391.)

Microscope images provide information about biological systems that is typically unavailable from any other source. Over the past thirty years the development of new microscopy methods, the advent of digital recording, and the development of high throughput microscopes created the ability to routinely collect terabytes of images containing detailed molecular and structural information. However, the breaking of the logjam in acquiring images has led to a bottleneck in analyzing them.


In TR&D3, we are addressing three specific, critical challenges in image processing and analysis: methods to convert diverse, micron resolution images to use in the modeling of cellular processes at molecular resolutions; paths to reconstructing the three-dimensional connectivity of neuronal tissue and relating the structure to neuronal activity; and  tools to provide fast access to various types of processed images whose sizes strain current methods, especially for interactive visualization. Overcoming these challenges will enable a dramatic increase in the information that can be extracted from images and in the routine use of cutting-edge modeling tools by cell biologists, developmental biologists, and neurobiologists.

To meet these challenges, we are focusing on three areas:

  • The development and distribution of software to build models of cellular and subcellular organization from fluorescent microscope images

    This software will include interfaces which provide new capabilities to cell simulation software such as MCell, Virtual Cell, and Smoldyn. It will improve existing object-based models of protein distribution at a single time point, and include tools to estimate generative spatial models of single protein concentrations at a single time point, generative spatiotemporal models for single proteins, and dependencies between spatiotemporal models for different proteins.

  • The creation of high efficiency registration and analysis algorithms for petavoxel image sets, especially image sets from serial section electron microscopy (ssEM).

    These algorithms will be implemented across a wide range of computing platforms and will improve the detection of alignment points, leading toward the fully automatic assembly of datasets exceeding one petavoxel. These algorithms will also include semi-automated methods to align datasets from different sources (such as EM and optical microscopy), and to trace and segment neural pathways and other structures within registered ssEM datasets.

  • The development of a new multi-platform framework called the Virtual Volume Filesystem (VVFS), enabling the efficient delivery of images from large datasets (~100 gigbytes or larger), especially the volumetric data produced by DBP5.

    The VVFS will allow users to enter data in optimized VVFS formats and insert algorithms into the VVFS pipeline to create customized, on-the-fly transformations. Results will be delivered as virtual files to analysis programs on users’ computing platforms. As an example application that uses the system, a Virtual Volume Viewer will be implemented to provide interactive viewing of VVFS datasets while navigating in arbitrary 3D orientations.


Relationship between cell and nuclear shape demonstrated We showed (Johnson et al, Molecular Biology of the Cell, in press) for the first time that cell shape can be predicted from nuclear shape (and vice versa). This relationship was diminished by altering protein C1QBP or various drugs. We also describe a generative model of the kinetics of shape change. The software is available in CellOrganizer v2.4.

Punctate subcellular patterns resolved through generative modeling A critical component of modeling the subcellular organization of cells is to learn proper models not just of the structure and position of organelles but also of how the spatial distributions of different organelles are related. The patterns of many small organelles and structures, such as lysosomes and RNA-processing bodies are difficult to distinguish visually and therefore image-based protein annotations are frequently quite general. We have therefore used images from the Human Protein Atlas to construct models of the subcellular distribution of punctate structures that capture the relationship of puncta to positions of microtubules and the nuclear and plasma membranes (Johnson et al, PLoS Computational Biology, submitted). We then used these models to assign 240 proteins with high confidence to one of the six organelles. Furthermore, we can use these models to create synthetic cell instances containing all six of these organelles (plus microtubules and the nucleus), enabling complex, spatially-realistic simulations of reactions involving these structures. We also extended these results to create Poisson and Markov point process models and showed that the positions of punctate organelles depend upon microtubules but not endoplasmic reticulum (Li et al, Cytometry, submitted). Software will be released in CellOrganizer v2.5.

Spatiotemporal model of signaling at the T cell synapse With DBP4, we have completed a major study of the regulation of actin dynamics by signaling molecules during antigen presentation. We analyzed over a thousand movies of actin and eight core actin regulators under control and costimulation-blocked conditions. Our computational analysis identified diminished recruitment of WAVE2 and Cofilin to F-actin as the dominant difference upon costimulation blockade. Reconstitution of WAVE2 and Cofilin activity restored the defect in actin signalling dynamics upon costimulation blockade (Roybal et al, Nature Cell Biology, submitted).

Identification of novel cytoplasmic complexes using cell fractionation and microscopy Current approaches for identifying and characterizing protein complexes emphasize direct measurement of protein-protein interactions but are subject to concerns about specificity and significance of measured interactions. We therefore developed an orthogonal approach in which proteins that cofractionate and share similar appearing objects in microscope images are inferred to be part of a complex. We have validated this approach by showing that many of the complexes we identify correspond to known complexes. Basically, the knowledge of which proteins coelute enables us to identify a subset of objects in images (by their appearance and distribution) that presumably correspond to those complexes (Naik et al, Nature Methods, in preparation).

High efficiency image registration We released AlignTK software based on Pearson correlation and spring model relaxation to iteratively converge on the global shape. Further development and testing was also done on SWIFT, our system which uses spatial frequency band amplitude scalings to perform high confidence image matching and applies Z direction averaging and Kalman smoothing to fit a global shape model. SWIFT was used in a collaboration with Florian Engert and David Hildebrand to perform a Zebrafish reconstruction from a combination of electron and 2-photon microscopy images consisting of over 17,000 sections

Major new release of CellOrganizer (v2.0) 

AlignTK 1.0.0 released

Protein distribution during T cell synapse formation Successful modeling and comparison of spatiotemporal patterns of protein distribution during T cell synapse formation (with DBP4). Manuscript in preparation.

Neuronal differentiation modeling New collaborative project begun on modeling of neuronal differentiation. Extensive image collection created for PC12 cells during NGF-induced differentiation, and a generative, statistical model of changes in cell and nuclear shape and mitochondrial distribution created. Manuscript in preparation.

More efficient diffeomorphic shape model learning software developed; being incorporated into the next release of CellOrganizer.

Promising results on signal whitening approach to registration Robustness and correctness testing on cutting edge 10,000 section 100TB dataset. Performance testing and optimization are underway during further development.


An illustration of a model of the distribution of Arp3 in the helper T cell [being built in collaboration with DBP4]. Left: 40 seconds before immunological synapse formation. Going across rows left to right and progressing top to bottom, each subimage shows a cross section of the 3D protein distribution averaged across 17 cells. Each slice is perpendicular to the synapse (the face of the half ellipsoid template) and shows the synapse as the top edge of the shape shown. The synapse has not yet formed, so actin branching is occurring uniformly all around the periphery of the cell. Right: Model at the time of immunological synapse formation. Actin branching appears to be concentrated near the synapse as expected.


TR&D3 Research Highlights

Mouse visual cortex
Anatomy and Function of an Excitatory Network in the Visual Cortex

MMBioS researcher Greg Hood’s collaboration with Wei-Chung Allen Lee of Harvard University and R. Clay Reid of the Allen Institute for Brain Science concerning the reconstruction of an excitatory nerve-cell network in the mouse brain cortex at a subcellular level using the AlignTK software has been published in Nature. Read more


langmead2 200Sparse Graphical Models of Protein:Protein Interactions

DgSpi is a new method for learning and using graphical models that explicitly represent the amino acid basis for interaction specificity and extend earlier classification-oriented approaches to predict ΔG of binding.  Read more


CellOrganizerCellorganizer 2.0 Major Release

A major new release of the CellOrganizer system for creating image-derived models of cell shape and organization has just been published.  Read more.

View all Research Highlights