Hi @hsandt,
A lot depends on whether you need to detect exact duplicates or nearly-matching colors in your palette.
Exact duplicates are easy to sift with a Lua script, as a color’s 32-bit integer representation can be used as a hash, or key, to a dictionary. See, for example, this post with a script to concatenate two palettes, excluding any duplicates
The biggest wrinkle you usually run into is how to handle colors with zero alpha, but potentially different RGB channels.
(Note: If you try the script linked above, a file preset string must be exactly right when loading palette from a preset, as a single typo can crash Aseprite 
).
Nearly-matching colors is a much more complex problem. This is because digital color can be represented multiple ways. Some ways are better suited finding near matching colors according to human perception, but come at the cost of extra computation time and learning on your part to understand them. You can get a taste of what’s involved here:
Aseprite’s scripting API does not provide much to handle this complexity, so you have to decide what’s worth your time and would give good-enough results.
As far as on project start vs. on export, what I’ve done to resolve other unrelated issues is to make my own new file dialog and GPL export (GPL is a file format that stores palettes), then assign custom hotkeys. If you have some coding background and interest in writing your own, take a look at Palette, which has a resize and a saveAs method; Sprite, which has a load from file constructor; and Dialog, which includes a file select widget.