this will work for the 1st part...
=AVERAGEIFS(B1:B9,A1:A9,">="&8/24,A1:A9,"<="&20/24)

Tring to figure out why the opposite wont work...
=AVERAGEIFS(B1:B9,A1:A9,"<"&8/24,A1:A9,">"&20/24)
gives #DIV/0