Pivot Table to roll up data and display in pivot chart

I'm using Excel 2010 to build a report on our service ticket data. In the end I want to find what the average time is per user for the date range that is selected.

I created a sample sheet with a pivot table, then manually summed up to create the column that I would like. I have tried to use "grand totals" but when i change it to average then the other columns change from sum to average. I think I may need to use a calculated field but I haven't been able to figure that out yet.

I would like to get the average column to be displayed in a pivot chart. I don't really need the daily time spent in the pivot chart.

Thanks in advance for your help!