Typical workflow in many drawing programs is to use X to toggle between fg and bg colors, but also to use C to toggle painting with mask. This helps a lot if you’re often alternating between two arbitrary fg and bg colors, but also wanting to erase using the same brush. Not having to reserve bg color for mask, and to repeatedly use the color picker or palette to alternate feels much smoother.
Swapping to eraser isn’t quite sufficient, since you usually want to keep the alpha brush shape and eraser shape separate, and requires swapping the active tool which isn’t as intuitive.