My task was to take 3 grayscale images, each corresponding to the red, green, and blue channels for a colored image, and align them to construct that colored image. The difficulty here lies in that these filters must be aligned properly, and we need to try various alignments and assess their effectiveness using some similarity metric.
The similarity metric I used was normalized cross correlation (NCC). Intuitively, NCC yields a high similarity score for an alignment of 2 channels when high values are matched with high values, and low values are matched with low values.
Crucially, when aligning color channels, the "similarity" we are looking for is not in the RGB values themselves but rather the structural similarity of the images. For example, for an image with a green hill, the RGB values for that hill would be close to (255, 0, 0), but NCC would punish the correct alignment of the red and green filters since their values are so different. Thus, I applied Canny edge detection on each channel, an algorithm that identifies the edges or boundaries of objects within an image. Then, I applied NCC on the "edge versions" of the channels so that they would be aligned based on structural similarity.
Original images
NCC on direct RGB values
NCC on Canny edge detection image
For small .jpg images, I searched a range of displacements, such as [-15, 15] pixels, for the red or green filters relative to the blue filter. For larger .tif images, I used a recursive image pyramid search. Image pyramid search works by repeatedly downscaling an image to create a pyramid of progressively smaller images, then using the best displacement vector for the previous resolution level as a starting point for the search for the next level. This significantly reduces the search space of displacement vectors by progressively honing in on high potential vectors.
