Why is this code slow?

For some reason my cursor takes more time to move from cell to cell than I think it should. It is a noticeable time delay.
I have narrowed it down to the following code.
It runs from every Worksheet_SelectionChange().

Sub My_SelectionChange(ByVal Target As Range)
    With Target.Worksheet
        On Error Resume Next
        .Range("Last_Selection_Row").Value = .Range("Selection_Row").Value
        .Range("Last_Selection_Col").Value = .Range("Selection_Col").Value
        .Range("Selection_Row").Value = Target.Row
        .Range("Selection_Col").Value = Target.Column
    End With
End Sub

I only notice the slowdown on one worksheet. I'm guessing that it has something to do with my Conditional Formatting. I have lots of cells that change their cell color when the cursor is on their row.

Is there a faster way to accomplish this?