CGA programs were notorious for palette swaps between the 16 basic colors. AFAIK EGA was the same issue: something like 256 total colors but only could show 16 at a time. It’s not until XVGA, I think, that you get parity between palette and total color space — effectively removing the need for constant palette swapping. Palette swaps were still used in many 2d games (and indeed, any game that has sprites still uses palette swaps in theory) because it allowed them to create more varieties of graphics without creating new graphics.

Anyway, I think CGA was actually 3 colors + black (two bits for color in the actual graphics.) You could map those bits differently if you wished.