See mock-up (left image is current implementation, right image is proposed implementation). Indenting the layer colors for nested layers would help a good deal with visual parsing of groups, as it makes the grouping hierarchy obvious at a glance.
Wow, I never worked with so many layers.
When I am 6 or 8 it starts to be difficult to handle everything, look for each layer, have enough space on the screen etc …