You could use the free command line video encoder ffmpeg (https://www.ffmpeg.org/). That will take png sequences and encode them into just about any video standard you could think of.
The basic command you’d need to make an mp4 would be:
ffmpeg -i img%03d.png -c:v libx264 -vf fps=25 -pix_fmt yuv420p out.mp4
Where your sequence of PNG files would be named ‘img001.png’, ‘img002.png’ etc
And the output would be a x264 encoded 25 fps mp4 file called out.mp4
Edit: Aseprite doesn’t pad frame numbers with zeros so you don’t need the ‘03’ in the input file name, just the ‘%d’ part.