OK, well i couldn't find how to fix this. So instead i used a workaround.

I setup a 4 cell range next to the pivot chart, referenced the values in the pivot table with some formula so the titles and values of this range update dynamically as the chart does - then pointed my bar chart at this range. Because there are always two series and two values (although one value may be blank) - the chart doesn't seem to get confused.