The baseline for upscaling and colorization is a naive approach used as a reference point for comparison. Upscaling simply applies a basic upscaling method, such as bilinear or bicubic interpolation, to the input image. Colorization is performed either by using the color version of the image as a reference or by simply returning the grayscale image as the result.
Using the color version of the image as a reference for the oracle would provide an unfair advantage, as it would essentially be using the ground truth information to perform the task. Therefore, the oracle is evaluated with and without using the color image as a reference to assess its performance objectively. As can be seen from the table, even when using the color image as a reference, the results are not perfect due to information loss during resizing.
| Entire test dataset | ||||
|---|---|---|---|---|
| with color reference | without color reference | |||
| Method | RGB MSE | Grayscale MSE | RGB MSE | Grayscale MSE |
| nearest | 0.00302 | 0.00297 | 0.01256 | 0.00296 |
| bilinear | 0.00149 | 0.00148 | 0.01107 | 0.00148 |
| bicubic | 0.00141 | 0.00140 | 0.01100 | 0.00140 |
| area | 0.00174 | 0.00171 | 0.01131 | 0.00171 |
| Used for validation subset (512 images) | ||||
| with color reference | without color reference | |||
| Method | RGB MSE | Grayscale MSE | RGB MSE | Grayscale MSE |
| nearest | 0.00299 | 0.00293 | 0.01337 | 0.00293 |
| bilinear | 0.00148 | 0.00146 | 0.01190 | 0.00146 |
| bicubic | 0.00139 | 0.00138 | 0.01182 | 0.00138 |
| area | 0.00173 | 0.00170 | 0.01213 | 0.00170 |
(data was collected via scripts/test-oracle.py)
In brief, the RGB MSE value of 0.01100 can be considered the primary baseline since it represents the ability to restore color from grayscale images. Achieving a value close to 0.003 is considered close to ideal, indicating high accuracy in restoring the color channels.
Monitoring the Grayscale MSE is also important. Naive upscale gives 0.00297 Grayscale MSE, so if this metric is higher, it means that the network completely fails to capture the color information.
Both the RGB MSE and Grayscale MSE metrics provide valuable insights into the performance of the network.