Font Color in cells in Pivot Tables

In Pivot Tables, the font color for certain fields are determined by the 'Color by' for the cell and NOT the color of the tag field.  See attached screenshot- where Executive Sponsor is a custom User field and that black text is barely readable in the dark grey tag but if I change the Color By cell color to a dark color, it will change the font to white.

This idea is to make the color of the font for a field in a pivot table based on the field color (tag fields) and NOT the 'color by' color set in the pivot table.