I created a tool for reducing an to a custom palette here
The ReadMe explains its use:
This tool is designed to reduce/convert the current palette of the active sprite to a custom input palette. This is ideal for workflows that generate large palette sizes, or just for cleaning up existing artwork It works by analyzing the pixels of the current project and converting them to one of the input colors. It will swap to the color of the custom palette closest to its own value (either RBG or LAB values).
Example palette reduction using a 8 color palette:
Welcome aboard and congratulations on releasing your Aseprite extension!
Regarding any tips, I’m not in the least an expert on performance, so take all advice to follow with a grain of salt.
I’d say start by having a look at this post:
As far as I know, color matching can be sped up with a space partitioning strategy. I’m aware of octrees, I believe people use k-d trees as well, and there may be yet others.
I also suggest finding the unique colors in an image, then matching the palette to uniques, rather than looping over all the pixels. Unique colors can be found by using a boolean as a dummy value for each key, e.g., dictionary[rgbaInteger] = true. Palette matches can be assigned to colors from the image in a similar manner, e.g., dictionary[uniqueImageColor] = paletteMatchInteger, then pixelIterator(dictionary[pixelIterator()]). In the examples, by integer I’m referring to the 32-bit integers that store the color channels, each in [0, 255], in the order 0xAABBGGRR.
Perhaps not as important: if you use Euclidean distance, try seeing if the distance-squared gives you the same results. It might spare you some math.sqrt calls. Also, if I remember correctly, you don’t need to find the math.abs of the delta when the exponent is even, i.e. is 2. For example, check that with local deltasq = delta * delta. You may want to retain the abs, however, if you want to generalize to some other distance metric later. To see what I mean, compare the formulas for Manhattan, Minkowski, Chebyshev and Euclidean distance.
(The distance metric for CIE LAB is a whole other rabbit hole, and I prefer to simplify by choosing a variant LAB space instead of tackling it directly.)
The go-to doc for general Lua performance tips is a pdf by Roberto Ierusalimschy: https://www.lua.org/gems/sample.pdf . It includes advice such as localizing global functions that are used in loops, e.g., local sqrt = math.sqrt and simplifying tables, e.g., rgbaTable[1 + i * channelCount + rChannelOffset] = red instead of rgbaTable[1 + i] = { r = red, g = green, b = blue }.
Measure elapsed time for your script. I just use os.clock, then subtract the start time from the end. Are there better ways? Probably.
Lastly, I guess others were looking into / found a way to write in another programming language, with the Lua handling primarily the UI? Not sure though. See Calling a C function from Lua inside Aseprite? .
These are excellent tips! The next thing I was going to change was an rbga dictionary to only calculate for unique colors so I’m glad I was on the right track there!
Thank you for all of these great suggestions, I will tinker with them asap
It works by analyzing the pixels of the current project and converting them to one of the input colors. It will swap to the color of the custom palette closest to its own value (either RBG or LAB values).