Got it figured out. I deleted the textbox, and then re-created it, making sure to have picked inside the chart first.
Now when the chart moves, so does the text box.