Mastering Flat Frames (1/3): Gain matching

In order to understand the nuances of astrophotography, a deeper dive is sometimes required. Recently, I ran into a situation just like that when preparing for the arrival of a new telescope. It started with some basic questions on which gain and darks to use for my Flats and why. And what would be the best flat panel for a large aperture telescope. It led me to a path of multiple experiments with sometimes surprising results. Through these experiments, I have been able to better understand and master my Flat frames and panels.

In a series of three blogs, I would like to share these experiences. Hopefully they can help you in making better informed decisions about how you would like to take your Flat frames. Topics covered are: which Gain settings to use, which Dark frames to use and which Flat Panel to use.

So, don't miss out! Subscribe to the newsletter to stay up to date with these and other new postings.

For fully automated imaging, some Flat panels can automatically close the front of the tube and turn on an evenly distributed light source. Brightness can be adjusted per filter, for maximum flexibility to optimise exposure times.

Fundamentals

Flat frames are probably the most important calibration frames used in astrophotography. There are many tutorials that describe basic concepts. For example the video from Cuiv, The Lazy Geek is worth watching. In short, Flat frames are designed to correct for ‘obstructions’ in the optical path of a telescope. Be it dust on sensor or filters, light fall-off in lenses or shadows from any physical obstruction.

A couple of ground rules exist for good reason. One needs an evenly lit light-source (e.g. a T-shirt over the telescope or dedicated Flat Panel). Exposures should reach about 25-50% of maximum illumination of the sensor (for most cameras an ADU value between 17,000 and 32,000). Focus is the same as for Light frames. Depending on the light source, Flats can be a fraction of a second or many seconds. Flat frames are calibrated using gain/offset/temperature matched dark and/or bias frames (see the third blog in this series). Multiple (25 is a good number) Flat frames are combined in one stack to minimise noise. Ideally Flats are taken before/after each session, but if the optical train remains intact (e.g. observatory) and no dust can get into the system, Flats may last substantially longer. So far pretty unambiguous.

Gain matching

But one topic of ambiguity is whether Flat frames should be Gain-matched with their corresponding Light frames. In various discussions on the topic, it turned out that there is no real consensus about this topic. The ‘yes’ argument is typically ‘to be safe’ while the ‘no’ argument is often ‘don’t see a reason why’. None of them are really convincing arguments.

To dive into the topic further, I recently started a discussion on Astrobin. Interestingly, this led to some interesting insights, a series of experiments and eventually a somewhat definitive answer.

Pixel-Response Non Uniformity (PRNU)

The insight that was provided had to do with Pixel-Response Non Uniformity (PRNU), a form of fixed pattern noise. Fixed pattern noise is well known for sensors that don’t receive light. It is one of the reasons why we shoot Dark frames. But something similar happens when a sensor is illuminated. Each pixel will respond slightly different to the same light. And Flats correct for this PRNU, just as they do for dust bunnies and light-falloff. Not completely though, as PRNU varies with illumination. Flats are usually between 25-50% of maximum illumination and thus somewhere in between bright stars (close to maximum) and faint nebulosity (only a few %). So perhaps not a perfect correction, but an approximation.

PRNU is also influenced by the gain setting of a sensor. A different gain can result in a different fixed pattern noise. And this is the reason why in theory a Flat frame should be gain-matched with its Light frames. But theory is not practise. So the question is: do we notice this effect? Well, there is only one way to find out, and that is to do the experiment. So that is what I did.

Experiment

Using the ZWO ASI533MM Pro camera, three sets of 25 Flats from the Luminance filter were compared, at Gain 0, 100 and 300. Target exposures for each Flat were 22,000-24,000 ADU. Corresponding Flat-Darks (Darks with matching time/gain/offset/temperature) were taken to calibrate the Flats. After calibration, Flats were stacked. All processing was done in PixInsight.

The results are shown in the image below. When comparing the Flats at Gain 0 and Gain 100, there is a clear difference in the overall ‘look’. At Gain 0 the image looks a bit more ‘cloudy’, while at Gain 100 the greys have a much finer structure to them. However, at Gain 100 there appear to be some horizontal bands, some narrow, some a lot wider. Unfortunately the details are a bit obscured by the many dust bunnies in the image. But this can be evened out. By subtracting both images (and applying a small pedestal to prevent clipping), the dust bunnies disappear. And what’s left is the actual fixed pattern noise difference from illumination. The differences in brightness are only small. A quick check with some previews suggests that the absolute variations are only about 10% compared to the shadows projected by the dust bunnies. So it is unclear how much of this would be visible in actual Light frames, but the pattern is definitely there.

Stacks of 25 Flats taken at Gain 0 (left) and Gain 100 (middle) show very different uniformity in PRNU. When subtracting the two images (right), all typical Flat features such as dust bunnies even out and what is left is the difference in uniformity. It shows a ‘blotchy’ pattern and horizontal banding.

In theory this fixed pattern noise is gain dependent. So the experiment was repeated at Gain 300. Upon first inspection, the images at Gain 100 and Gain 300 look fairly similar. The Gain 300 image is a bit grainier, but nothing uncommon for Gain 300 images. But the real surprise comes when we subtract the Gain 100 and Gain 300 images from each other. The result is a completely flat image, with no pattern in it whatsoever. In other words, whatever the fixed pattern was at Gain 100, at Gain 300 it is exactly the same.

Stacks of 25 Flats taken at Gain 100 (left) and Gain 100 (middle) show very similar uniformity in PRNU. When subtracting the two images (right), all typical Flat features such as dust bunnies even out and what is left is the difference in uniformity. It shows a completely uniform distribution and no pattern of any kind.

High Conversion Gain (HCG) mode

So why is there such a difference between Gain 0 on one hand and Gain 100 and 300 on the other hand? The answer can be found in a review of the ASI294MM camera by Christian Buil. For that camera he had noticed similar differences in illumination pattern between gains lower than 120 and gains higher than 120.

These modern camera’s have so-called dual-gain settings. In Low Conversion Gain (LCG) mode, sensors are capable of capturing bright objects better and have maximum dynamic range. In High Conversion Gain mode (HCG), the read noise is greatly reduced, with only a modest drop in dynamic range, ideal for low light astro-photos. Cameras with this dual-gain feature can be identified from their gain/read-noise curve. It always shows a dramatic drop somewhere halfway the curve. The camera usually switches automatically, depending on which gain it is set to. This gain setting can be different between different camera’s. For the ASI294MM this is 120. The ASI533MM switches at Gain 100, and the effects seen here are consistent with what Christian demonstrated for the ASI294MM at Gain 120.

Conclusions

The data show differences in fixed pattern noise under illumination (PRNU) between Gain 0 (HCG mode off) and Gain 100 (HCG mode on). The data show no differences between Gain 100 and Gain 300 (both HCG mode on). So in as much as PRNU is Gain dependent, this cannot be demonstrated in a real life setting. But PRNU is clearly dependent on which HCG mode is selected.

In other words:

Flats and Lights should be ‘HCG-mode matched’, not necessarily ‘gain-matched’.

A practical consequence is that with dim Flat panels, there is no problem cranking up the Gain to e.g. 300 for Flats of narrowband filters if the narrowband imaging is done at Gain 100. This has very positive practical consequences as we will see in the second blog of this series.

Update (12 February 2023)

After publishing this blog, a detailed discussion resulted on Cloudy Nights with other astrophotographers performing similar experiments. And that triggered me to do some some additional experiments. If you’re interested please have a look at the CN thread for all the details. At large most of the conclusions from this blog are confirmed. But there is one additional aspect that became apparent. While HCG-mode matched flats are not gain-dependent for their flat fielding corrections, there were some indications that higher gain flats might lead to some extra noise. The question then becomes how much of a negative effect this has on the final images. On a recent image this was tested. A set of 23 H-alpha frames taken at Gain100 was calibrated with both Flats at Gain 100 and at Gain 300. Visually the resulting images showed no difference at all. Since a visual comparison on a stretched image can be deceiving, the statistics were also compared and appeared very similar. The median signals were 57.068 vs 56.947 and the Mean Absolute Deviations (MAD) were 5.499 and 5.464 respectively, essentially identical.

View fullsize

Statistics of a stack of H𝛼 images calibrated using Flats at Gain 100 (left) and Gain 300 (right) don’t show an increase in noise.

Nevertheless, perhaps its best to change the conclusion mentioned above a little. Probably a better conclusion would be:

Flats and Lights should be ‘HCG-mode matched’, not necessarily ‘gain-matched’. If not gain-matched, there may be risk of small noise increase, but this could not be confirmed in testing.