I was excited and happy to see the beta gave us a new method for converting from RGB to indexed color, and have been trying the octree conversion. Thanks to the devs for their hard work! I have occasionally gotten some unexpected results. I believe this is known already and I do not like to complain about a feature until I’ve done a little digging. So here are some very non-scientific test-runs with where I’ve gotten so far.
I’m using Aseprite v 1.3 beta 4 on Windows 10.
For a palette, I used Adam C. Younis’s Apollo. I chose this palette because it was designed for general use across many scenes in a video game; it includes a skin-tone ramp and an extended low saturation ramp.
For a test image, I downloaded some Chrono Trigger screen captures from here. They were scaled up 400%, so I resized them 25% in Aseprite using nearest-neighbor then put four together into one image.
This is the result of Aseprite’s old conversion method (table-based) with no dithering:
This is Aseprite’s new conversion method (octree) with no dithering:
I can’t post the homebrew conversion script in a single gist, because too much is involved. I needed a 3D vector, an axis aligned bounding box (AABB), an octree, a color class that could handle conversions from sRGB to CIE LAB, and a dialog script.
Below are test conversions. All color representations search for the nearest color with squared Euclidean distance; there are other distance metrics possible (Chebyshev, Manhattan, Minkowski).
The first picture uses standard RGB (sRGB).
In linear RGB:
In CIE XYZ:
and in CIE LAB:
If you want to see a more neutral diagram–similar to a part of Lospec’s DB Palette Analysis–here’s a test sheet with 6 faces from a flattened sRGB cube:
Here is the Aseprite table approach:
Here is the Aseprite octree approach:
Compare with the homebrew results. In standard RGB:
In linear RGB:
In CIE XYZ:
In CIE LAB:
For both Aseprite and homebrew, I focused on elements of the courtroom scene (bottom-left): did the conversion pick up on the green banners hung over the juror’s boxes and did it recreate the stained glass window behind the judge well. To my eye, there’s a blue shift in the Chrono Trigger Aseprite indexed conversions.
The sRGB cube faces make me wonder if there may even be a bug: the (R, G, 0) and (R, G, 255) faces in the left column look ok. The (R, 0, B) and (R, 255, B) faces in the middle look askew to me.
These are some limitations of the homebrew implementation: it processes only one cel at a time; it doesn’t actually convert the image to indexed color mode; it’s very slow; it doesn’t handle alpha at the moment (I may just retain the source alpha in the future); there’s no dithering to compensate for its limitations; depending on which palette and image you pick, the results may be just as lousy no matter the representation.
Lastly, I discovered that GIMP has a indexed mode conversion. That could be a useful control for experiments. I don’t know about other programs like Krita and so on.
I can’t tell from the Chrono Trigger screen cap, but the sRGB faces image leads me to believe that GIMP is using CIE LAB color space. It also looks like the color has been quantized.