COUNTIFS with cell ranges

[1.zer0]

I have this formula that I botched up and it's not working how I want. I don't know what the problem is though?

=COUNTIFS(INDIRECT("'"&TEXT((B$3:B22),"dd-mm-yy")&"'!E:E"),"Yes",INDIRECT("'"&TEXT((B$3:B22),"dd-mm-yy")&"'!F:F"),"No")

Based on the number it is giving me and what I actually want I think I've deduced it to the cell ranges of (B$3:B22)

The end result I want is to give me the amount of combined "Yes's" in one column and "No's" in another where they are matched, for multiple sheets.

So if:

Row1 Yes No. This would be counted.
Row2 Yes Yes. This wouldn't be counted.
Row 3 No Yes. This is an impossibility and so is irrelevant.
Row 4 No No or "". This wouldn't be counted.

I have uploaded the file to help clarity.

Many Thanks