Introduction

Viv

A WebGL-powered toolkit for interactive visualization of high-resolution, multiplexed bioimaging datasets.

About

Viv is a JavaScript library for rendering OME-TIFF and OME-NGFF (Zarr) directly in the browser. The rendering components of Viv are packaged as deck.gl layers, making it easy to compose with existing layers to create rich interactive visualizations.

More details and related work can be found in our paper and original preprint. Please cite our paper in your research:

Trevor Manz, Ilan Gold, Nathan Heath Patterson, Chuck McCallum, Mark S Keller, Bruce W Herr II, Katy Börner, Jeffrey M Spraggins, Nils Gehlenborg, "Viv: multiscale visualization of high-resolution multiplexed bioimaging data on the web." Nature Methods (2022), doi:10.31219/10.1038/s41592-022-01482-7

💻 Related Software

💥 In Action

• Vitessce visualization framework
• HuBMAP Common Coordination Framework Exploration User Interface (CCF EUI)
• OME-Blog OME-NGFF and OME-NGFF HCS announcements
• ImJoy I2K Tutorial
• Galaxy Project includes Avivator as default viewer for OME-TIFF files
• 10x Genomics uses Viv in their viewer for Xenium In Situ Analysis Technology: demo

💾 Supported Data Formats

Viv's data loaders support OME-NGFF (Zarr), OME-TIFF, and Indexed OME-TIFF*. We recommend converting proprietrary file formats to open standard formats via the bioformats2raw + raw2ometiff pipeline. Non-pyramidal datasets are also supported provided the individual texture can be uploaded to the GPU (< 4096 x 4096 in pixel size).

Please see the tutorial for more information.

*We describe Indexed OME-TIFF in our paper as an optional enhancement to provide efficient random chunk access for OME-TIFF. Our approach substantially improves chunk load times for OME-TIFF datasets with large Z, C, or T dimensions that otherwise may incur long latencies due to seeking. More information on generating an IFD index (JSON) can be found in our tutorial or documentation.

💽 Installation

You will also need to install deck.gl and other peerDependencies manually. This step prevent users from installing multiple versions of deck.gl in their projects.

Breaking changes may happen on the minor version update. Please see the changelog for information.

📖 Documentation

Detailed API information and example sippets can be found in our documentation.

🏗️ Development

This repo is a monorepo using pnpm workspaces. The package manager used to install and link dependencies must be pnpm.

Each folder under packages/ are a published as a separate packages on npm under the @vivjs scope. The top-level package @hms-dbmi/viv exports from these dependencies.

To develop and test the @hms-dbmi/viv package:

1. Run pnpm install in viv root folder
2. Run pnpm dev to start a development server
3. Run pnpm test to run all tests (or specific, e.g., pnpm test --filter=@vivjs/layers)

🛠️ Build

To build viv's documentation and the Avivator website (under sites/), run:

📄 Sending PRs and making releases

For changes to be reflected in package changelogs, run npx changeset and follow the prompts.

Note not every PR requires a changeset. Since changesets are focused on releases and changelogs, changes to the repository that don't effect these won't need a changeset (e.g., documentation, tests).

The Changesets GitHub Action will create and update a PR that applies changesets versions of @vivjs/ packages to NPM.

🌎 Browser Support

Viv supports both WebGL1 and WebGL2 contexts, to provides coverage across Safari, Firefox, Chrome, and Edge. Please file an issue if you find a browser in which Viv does not work.

Getting Started

Start a simple web server to make your data avaiable to the browser via HTTP.

If following the Data Preparation tutorial, the server provides Viv access to the OME-TIFF via URL:

• http://localhost:8000/LuCa-7color_Scan1.ome.tif (OME-TIFF)

If you wish to use the SideBySideViewer, simply replace PictureInPictureViewer with SideBySideViewer and add props for zoomLock and panLock while removing overview and overviewOn.

API Structure

API Structure

Viewer

Viv provides four high-level React components called "Viewers": PictureInPictureViewer, SideBySideViewer, VolumeViewer and VivViewer. The first three wrap the VivViewer and use the View API for composing different layouts. VivViewer wraps a DeckGL component and handles managing the complex state (i.e multiple "views" in the same scene) as well as resizing and forwarding rendering props down to the underling layers via the View API.

View

A "View" in Viv is a wrapper around a deck.gl View that exposes an API to manage where an image is rendered in the coordinate space. A View must inherit from a VivView and implement:

• 1.) a filter for updating ViewState
• 2.) a method for instantiating a View
• 3.) a method for rendering Layers

Views are used by Viv's Viewer components. For example, the OverviewView is used in the PictureInPictureViewer to provide a constant overview of the high resolution image. The SideBySideView has supports locked and unlocked zoom/pan interactions within the SideBySideViewer.

Layer

Viv implements several deck.gl Layers, for rending RGB and multi-channel imaging data. These layers can be composed like any other layer in the deck.gl ecosystem. The main three layers for use are MultiscaleImageLayer (for tiled, pyramidal images), ImageLayer (for non-pyramidal and non-tiled iamges), and VolumeLayer (for volumetric ray casting rendering) which accept PixelSource arguments for data fetching. These layers handle the complexity of data fetching and setting up the rendering by wrapping the XRLayer, XR3DLayer and BitmapLayer, which are the lower level rendering layers.
The XRLayer (eXtended Range Layer) and XR3DLayer enable multi-channel additive blending of Uint32, Uint16, Uint8 and Float32 data on the GPU.

A crucial part of the layer is the extensions prop - these control the per-fragment (pixel) rendering. The default on all layers is ColorPaletteExtension and it will provide a default colors prop - thus all that is necessary for controlling rendering is the contrastLimits. But if you wish to do something different, for example to use a "colormap" like viridis, you will need to pass in extensions: [new AdditiveColormapExtension()] and colormap: viridis. Please see deck.gl's documentation for more information.

Loader (Pixel Sources)

Viv wraps both Tiff- and Zarr-based data sources in a unified PixelSource interface. A pixel source can be thought of as a multi-dimensional "stack" of image data with labeled dimensions (usually ["t", "c", "z", "y", "x"]). A multiscale image is represented as list of pixel sources decreasing in shape. Viv provides several helper functions to intialize a loader via url: loadOmeTiff, loadBioformatsZarr, loadOmeZarr, and loadMultiTiff. Each function returns a Promise for an object of shape { data: PixelSouce[], metadata: M }, where M is a JavaScript object containing the format-specific metadata for the image. For OME-TIFF and Bioformats-Zarr, and MultiTiff the metadata is identical (OME-XML representation), for OME-Zarr,the metadata is that for a multiscale group (more information: https://ngff.openmicroscopy.org/latest/). This metadata can be useful for creating UI componenets that describe the data source.

Custom Shaders

2D

Viv's shaders in 2D can be modified via deck.gl shader hooks. The LensExtension, AdditiveColormapExtension, and ColorPaletteExtesnion (default) in Viv implement shader hooks. Implementing your own shader hook requires extending the standard layer extension with a getShaders method that returns an inject function for one of the following supported hooks:

DECKGL_PROCESS_INTENSITY(inout float intensity, vec2 contrastLimits, int channelIndex)

This hook allows for custom processing of raw pixel intensities. For example, a non-linear (or alternative) transformation function may be provided to override the default linear ramp function with two contrast limit endpoints. This hook is available on all layers in all modes. By default, the layer provides a reasonable function for this.

DECKGL_MUTATE_COLOR(inout vec4 rgba, float intensity0, float intensity1, float intensity2, float intensity3, float intensity4, float intensity5, vec2 vTexCoord)

This hook allows for users to mutate conversion of a processed intensity (from DECKGL_PROCESS_INTENSITY) into a color. This is only available in 2D layers. An implementation for this hook is required by all Viv extensions.

DECKGL_FILTER_COLOR(inout vec4 color, FragmentGeometry geometry)

Please see deck.gl's documentation as this is a standard hook. This hook may be used for example to apply a gamma transformation to the output colors.

3D

Viv's shaders can be modified in 3D similar to the above, with the exception of DECKGL_MUTATE_COLOR. Instead, at least one provided extension must implement _BEFORE_RENDER, _RENDER and _AFTER_RENDER. Specifically, one extension must have opts.rendering as an object with at least the _RENDER property a string that contains valid glsl code. For example, the MaximumIntensityProjectionExtension uses _BEFORE_RENDER to set up an array which will hold the found maximum intensities. _RENDER fills that array as maximum intensities are found. And finally _AFTER_RENDER will place those intensities in the color or gl_FragColor buffer to be rendered.

Data preparation

This guide demonstrates how to generate a pyramidal OME-TIFF with Bio-Formats that can be viewed with Avivator. Viv also supports OME-NGFF, but tooling to generate the format remains limited as the specification matures. We will update this tutorial accordingly when a method for generating OME-NGFF is endorsed by the Open Microscopy Environment.

Getting Started

NOTE: If you wish to view an image located on a remote machine accessible via SSH, please see the note at the end of this document.

This tutorial requires Bio-Formats bioformats2raw and raw2ometiff command-line tools. It's easiest to install these tools using conda, but binaries also are available for download on the corresponding github repositories.

Input data

Bio-Formats is an incredibly valuable toolkit and supports reading over 150 file formats. You can choose one of your own images, but for the purposes of this tutorial, we will use a multiplexed, high-resolution Perkin Elmer (1.95 GB) image made available under CC-BY 4.0 on OME.

Pyramid Generation

First use bioformats2raw to convert the .qptiff format to an intermediate "raw" format. This representation includes the multiscale binary pixel data (Zarr) and associated OME-XML metadata.

The next step is to convert this "raw" output to an OME-TIFF.

Note: LZW is the default if you do not specify a --compression option (the syntax requires an "=" sign, like --compression=zlib).

You may also use bfconvert (Bioformats >= 6.0.0) to generate an OME-TIFF.

All the above arguments are necessary except for -compression which is optional (default uncompressed). In order for an image to be compatible with Viv:

• -pyramid-scale must be 2
• -tilex must equal -tiley (ideally a power of 2)
• -pyramid-resolutions must be computed using the image dimensions and tile size. For example, for a 4096 x 4096 with tile size of 512, 3 = log2(ceil(4096 / 512)) resolutions should work well. a power of 2), and the -pyramid-resolutions argument should be adjusted to match the size of your image and your choice of tile size. For example, if you have a 4096 x 4096 and tile size of 512, 3 = log2(ceil(4096 / 512)) resolutions should work well. For the LuCa-7color_Scan1.qptiff image, 6 = max(log2(ceil(12480 / 512)), log2(ceil(17280 / 512))) resolutions work best as the image is 12480 x 17280 in size. There is currently no "auto" feature for inferring the number of pyramid resolutions. Without the compression set, i.e -compression LZW, the output image will be uncompressed.

There is a 2GB limit on the total amount of data that may be read into memory for the bfconvert cli tool. Therefore for larger images, please use bioformats2raw + raw2ometiff.

NOTE: Viv currently uses geotiff.js for accessing data from remote TIFFs over HTTP and support the three lossless compression options supported by raw2ometiff - LZW, zlib, and Uncompressed as well as jpeg compression for 8 bit data. Support for JPEG-2000 for >8 bit data is planned. Please open an issue if you would like this more immediately.

Indexed OME-TIFF

The TIFF file format is not designed for the cloud, and therefore certain images are less suitable to be natively read remotely. If your OME-TIFF image contains large non-XY dimensions (e.g. Z=100, T=50), you are likely to experience latencies when switching planes in Avivator due to seeking the file over HTTP. We recommend generating an index (offsets.json) that contains the byte-offsets for each plane to complement OME-TIFF, solving this latency issue and enabling fast interactions.

Alternatively you may use our web application for generating the offsets.

⚠️ IMPORTANT ⚠️ Avivator requires the offsets.json file to be adjacent to the OME-TIFF on the server in order to leverage this feature. For example, if an index is generated for the dataset in this tutorial, the following directory structure is correct:

data
├── LuCa-7color_Scan1.offsets.json
└── LuCa-7color_Scan1.ome.tif

This index can be reused by other Viv-based applications and even clients in other languages to improve remote OME-TIFF performance. If using Viv, you must fetch the offsets.json directly in your application code. See our example for help getting started

Viewing in Avivator

⚠️ Warning ⚠️ This section only works in Chrome, Firefox, and Edge (not Safari) due to differences in how browser restrict websites hosted at https:// URLs (Avivator) from issuing requests to http:// (the local data server) as a security measure. The supported browsers allow requests to http:// from https:// under the special case of localhost, whereas Safari prevents all requests to http://. As a workaround, you can start an Avivator client at http://, but we suggest trying a different supported browser. Alternatively, you can drag-and-drop an image (no local server) into the viewer in any browser.

There are a few different ways to view your data in Avivator.

If you have an OME-TIFF saved locally, you may simply drag and drop the file over the canvas or use the "Choose file" button to view your data.

Otherwise Avivator relies on access to data over HTTP, and you can serve data locally using a simple web-server. It's easiest to use http-server to start a web-server locally, which can be installed via npm or Homebrew if using a Mac.

Install http-server

Starting a Server

From within this directory, start a local server and open Avivator in your browser.

This command starts a web-server and makes the content in the current directory readable over HTTP. Once the server is running, open Avivator and paste http://localhost:8000/LuCa-7color_Scan1.ome.tif into the input dialog to view the OME-TIFF generated in this tutorial. For convenience, you can also create a direct link by appending an image_url query parameter:

• http://avivator.gehlenborglab.org/?image_url=http://localhost:8000/LuCa-7color_Scan1.ome.tif (OME-TIFF)

Troubleshooting: Viv relies on cross-origin requests to retrieve data from servers. The --cors='*' flag is important to ensure that the appropriate Access-Control-Allow-Origin response is sent from your local server. In addition, web servers must allow HTTP range requests to support viewing OME-TIFF images. Range requests are allowed by default by http-server but may need to be enabled explicitly for your production web server.

Viewing an Image via SSH

It is possible to generate the datasets in this tutorial on a remote machine and view them in Avivator via SSH and port forwarding. For example, you can follow this tutorial on a remote machine within SHH, linking your local port 12345 to the remote's local 8000.

Since your local port 12345 is linked to the remote 8000 via SSH, you can now view the remote dataset locally via your localhost:12345 in Avivator: that is, you paste http://avivator.gehlenborglab.org/?image_url=http://localhost:12345/LuCa-7color_Scan1.ome.tif into your browser instead of http://avivator.gehlenborglab.org/?image_url=http://localhost:8000/LuCa-7color_Scan1.ome.tif as is written at the end of the tutorial.

Other Examples

Other sample OME-TIFF data can be downloaded from OME-TIFF sample data provided by OME and viewed with Viv locally (without needing to run Bio-Formats).

• MitoCheck
• Artificial Datasets

3D Rendering

Viv has the capability to do volume ray-casting on in-memory volumetric data. It also exposes an API for applying arbitrary affine transformations to the volume when rendering. Napari is another popular tool for doing both of these, but there are some key differences. Viv follows the convention of 3D graphics to first have the x-axis running left-right across the screen, then the y-axis running up and down, and then the z-axis running into and out of the screen (all following the right-hand-rule for standard orientation). Napari orients volumes in order to respect broadcasting conventions with numpy - that is their axis order is actually zyx, which is the reverse of Viv.

If you have a homogeneous matrix that you are using in Napari, the best way to make it usable in Viv is to go back through the steps by which you got the matrix, reversing your z and x operations. However, if this is not possible, the following steps should produce something that looks identical as it swaps the row and column space (i.e you want to change the order of both how the matrix interprets its inputs and how it affects its outputs):

If you would like more information, please visit our github page.

Viewers (React Components)

Views

Utility Methods

Layers

The documentation in this section shows each layer as an object with properties although this is not accurate. These are deck.gl layer classes and the properties are those of the props that are passed into the constructor of the deck.gl layer classes. For more information, see deck.gl's documentation on how to use layers or the layer class. We welcome contributions to improve the docs, whether it be small fixes or a new docs site that would allow us to show these classes properly. Thanks!

Extensions

Loaders

Misc