E increases my brush/eraser size instead of switching to eraser

like the title says sometimes (not always) when I press E it increases my brush/eraser size instead of switching to the eraser and it kinda slows my progress whenever it happens.
I checked the keybinds and E is set to eraser so what could be the issue?