Odd hotkey behavior when using mouse and tablet

I often use the mouse to scroll or click on things while using a wacom tablet for most of the drawing.

Sometimes while switching between the two, hotkeys will not function. For example the eraser will be selected, and yet pressing the pencil hotkey will not switch to it. Then if I draw on the screen for a second with the tablet, then the hotkeys start working again, even if I use the mouse.

Happens enough to be annoying, but I have no clue what causes it.