Displaying data labels only from separate column

I hope this has not already been asked, but I have been searching for the last several hours and I have not yet found a solution.

I have an chart in Excel that is displaying data according to a selected column from the worksheet. On a separate column, I have data labels only on certain cells in the column (the rest of the column being blank). The two columns line up with each other such that the data labels should appear at the same point as the data on the other column. I am having a very hard time figuring out how to display these data labels on the chart. Any help you can provide will be greatly appreciated.

Example:

Column A
A1 1
A2 2
A3 3
A4 4
A5 5

Column B
B1
B2 Label
B3
B4 Label
B5