Aseprite indexed color mode images are 1 byte per pixel, meaning [0, 255] is the limit, even if you load a palette greater than that. RGB color mode images do not support variable bits per channel; it’s 4 bytes per pixel in active memory, 1 byte per channel.
Two suggestions would be (1.) Create an indexed image that uses a subset of the 9-bit RGB palette. (2.) Create a 32-bit RGB color mode image that simulates 9-bit by compressing and then re-expanding each color channel.
| Step |
Decimal |
Hex |
Diff |
| 0 |
0 |
00 |
|
| 1 |
36 |
24 |
36 |
| 2 |
73 |
49 |
37 |
| 3 |
109 |
6D |
36 |
| 4 |
146 |
92 |
37 |
| 5 |
182 |
B6 |
36 |
| 6 |
219 |
DB |
37 |
| 7 |
255 |
FF |
36 |
The expansions depend on what you use as a reference. Sometimes I see a consistent step of 36 that sums to 252 rather than the alternating 36, 37 step above. A Wikipedia reference image uses 0, 36, 72, 109, 145, 182, 218, 255. I’ve also seen a step of 34 here.
If you’re interested in those options, there are some scripts here that might be useful. The test image is from the Nasa Webb telescope. Looks like this forum has converted some of these images to jpegs, though, so the exact result will not carry across. 
Aseprite palette matching result, no dither, 256 colors.
Quantize color. No reference palette needed. 223 out of 512 colors used.
Palette to cel, no dither. 179 out of 512 used.
Floyd-Steinberg dither with SR LAB 2 color model. 202 out of 512 used.
If you extract the palette from option 2 and the palette is within the 256 color limit, then you can use that as a basis for option 1. Aseprite has a built-in palette from sprite feature in the palette options menu. The scripts repo linked above also has a palette from cel script.
I know that the ppm file format supports custom channel maximums. This means that your image would actually be stored in 9-bit RGB, 3-bits per channel, if you used a max of 7, as 2 ^ 3 - 1. It’s up to the importer to expand that color data as needed. There’s a Lua script to import and export to ppm files at this Github repo.
Krita has a palettize filter which doesn’t change the color mode, but looks like it does a decent job. It also has dither pattern options. GIMP has the same limitation as Aseprite. You can convert an image to indexed, but can’t even select palettes of more than 256 colors.
