Pivot charts are a different kind of animal than regular charts. They are great in that they are "quick and dirty" but they have some limitations.

You may have better luck overlaying the pivot table results with named dynamic ranges and use the names to to run the charts. I can't say more than that without seeing a sample file.