https://www.ximea.com/support/projects/allprod/wiki/bonus_software_for_scientific_cmos_cameras

GPU Software for processing sCMOS cameras data¶

Scientific research is a highly demanding field that requires the newest cameras to produce ever lower noise, higher resolution, increased sensitivity and all that at high fast speed.
The ones able to deliver such parameters become indispensable in microscopy, experiments with cold atom gases, astronomy, photonics, etc.
Apart from the exceptional hardware side, there is a desire for high performance of the software part to process large data streams in realtime and with high precision.
To cover these necessities, XIMEA has introduced the xiJ family of cameras with outstanding features.

Since the beginning, scientific grade cameras from XIMEA have always focused on providing the lowest noise because in quantitative applications, like single molecule imaging and super resolution microscopy imaging, reducing the noise to a minimum is one of the top priorities.
In addition to that, the xiJ camera models come with powerful onboard FPGA, high end filters and housing materials, efficient Peltier cooling modules, precise calibrations, customized sensor timing, high frame rates, triggering capabilities, proprietary readout modes and last but not least - highest quality sensor grades including the backside illuminated versions.

Every xiJ camera is carefully fine-tuned to deliver impressive linearity, especially during long exposure times at low light, improving photo response non-uniformity (PRNU) and dark signal non-uniformity (DSNU), reducing fixed pattern noise (FPN) and minimizing pixel differences.
The xiJ models are equipped with proprietary "CLEANPATH" readout and sensor driving circuitry, which results in low readout noise architecture and high fidelity in color reproduction with clear 14 and 16 bits/pixel images in various modes.

Applications for sCMOS cameras¶

There are a plethora of scientific imaging tasks which are or could be handled by such cameras:

Laser Scanning Confocal Microscopy
Biophysics and Biophotonics
Live-Cell Microscopy and Live-Cell Imaging
Biological and Biomedical Sciences
Light Sheet Fluorescence Microscopy
Bioimaging and Biosensing
Neuroimaging
Digital Microscopy

Image processing Software for xiJ cameras¶

As usual with high performance cameras, the xiJ models generate a lot of data.
The typical bandwidth could be deduced from 90 fps * 4.1 MPix * 2 Byte/Pix = 800 MByte/s.
In the case of 16-bit modes, this could present a bottleneck in saving such data stream to SSD, especially for two-camera system and long-term recording, which is usual for microscopy applications.
For one day of operation and recoding duration, storage could become a problem.

Such two-camera setup produces around 5.76 TB per hour and implementing real-time compression could help to cut the storage cost.
Utilization of JPEG or H.265 encoding algorithms will not work for 16bit since these don't support more than 12-bit data.
The best choice then could be JPEG2000 compression which is working natively with 16-bit images.
Tests with NVIDIA GeForce GTX 1080 resulted in approximately 240 fps for lossy JPEG2000 encoding with a compression ratio near 20.

Here is the JPEG2000 benchmark comparison for widespread J2K encoders: https://www.fastcompression.com/benchmarks/benchmarks-j2k.htm
This is the outcome achievable on CPU because corresponding JPEG2000 implementations with OpenJPEG, Jasper, J2K or Kakadu are slower.
JPEG2000 lossless compression algorithm is also available, but it offers a smaller compression ratio, usually in the range of 2-2.5 times.
Still, it's a useful option to store original compressed data without any losses which could be mandatory for a particular image processing workflow.

The optimal compression ratio for lossy JPEG2000 encoding should be defined by checking different quality metrics and their correspondence to a particular task to be solved.
However, there is currently no better alternative for fast JPEG2000 compression for 16-bit data, so JPEG2000 is preferred and recommended with the following image processing modules to be added to the full pipeline for better image quality:

Dynamic Bad Pixel Correction
Data linearization with 1D LUT
Dark Frame Subtraction
Flat Field Correction (vignette removal)
White/Black Points
Exposure Correction
Curves and Levels
Denoising
Crop, Flip/Flop, Rotate 90/180/270, Resize
Geometric transforms, Rotation to an arbitrary angle
Sharp
Gamma Correction
Realtime Histogram and Parade
Mapping and monitor output
Output JPEG2000 encoding (lossless or lossy)

Fastvideo SDK¶

The above image processing pipeline could be fully implemented on GPU to achieve realtime performance or even faster.
It could be done with Fastvideo SDK and NVIDIA GPU.
This SDK is supplied with sample applications in source codes, so users can create their own GPU-based application very fast.
Fastvideo SDK is available for Windows, Linux, L4T.

There is also a gpu-camera-sample application which is based on Fastvideo SDK.
You can download source codes and/or binaries for Windows from the following link on Github: https://github.com/fastvideo/gpu-camera-sample
Binaries are able to work with raw images in PGM format (8/12/16-bit), even without a camera.
User can add support for XIMEA cameras to process images in realtime on NVIDIA GPU.

Notes¶

The performance of JPEG2000 codec strongly depends on GPU, image content, encoding parameters and complexity of the full image processing pipeline.
To scale the performance, users can also utilize several GPUs for image processing at the same time.
Multiple GPU processing options are part of Fastvideo SDK.
If you have any questions, please fill the Contact form

Summary¶

High quality and high performance software to work with scientific sCMOS cameras:

- Lossless J2K encoding for 16-bit frames in realtime
- Full image processing pipeline implementation on GPU is possible on laptop, desktop, server, mobile.
- Fast viewer for raw or compressed footages
- Solutions for multicamera systems

Benefits:

- Fast Image Processing on GPU
- High quality images
- Low latency
- Reduced TCO
- Lower time to market

Software is based on Fastvideo SDK, to discuss your needs or to get SDK together with sample applications, please contact Fastvideo company.