Sum based on whether multiple cells don't contain multiple values

So I can't wrap my head around this one. I'm trying to get the total of all cells on Sheet 1 Column B that do not equal the values on sheet 2 (D2:D5) whether positive or negative that are labeled as Non-Union in Sheet 1 Column A to populate in cell E5 on Sheet2.

I know this sounds pretty confusing but I've attached a sample spreadsheet that I hope makes it easier to understand. Any help on this is much appreciated.