Would you mind posting some example images?
I’ve seen the problem you describe before firsthand, and threads about palette matching have been made before, such as https://community.aseprite.org/t/converting-to-index-color-gives-incorrect-results . Aside from making sure that a transparent color, clear black, is at palette index 0, which you mentioned, I don’t have a lot of other decent ideas.
-
Try a similar feature in other software, like palettize map in Krita. Importing palettes is not as intuitive; see the docs for Manage Resources. GIMP also has a convert to indexed color mode feature.
-
I’ve found that matching in a different color space, such as LAB, can improve the results, but not always.
-
Manually replacing problem colors with a tolerance might help, but would be tedious. Again, if available, use a color replace function based in LAB, not RGB.
I initially thought that a Floyd-Steinberg might help, but with the Aseprite built-in match it didn’t.
Just in case, here is an example set using Calm 48 and a still life by Ruysch:
Calm 48 palette.
Aseprite’s built-in match.
Custom match using a Lua script, SR LAB 2 and a cylindrical distance metric.
Krita, LAB space.
GIMP.