I don't believe so or that it's necessary. I just always make sure my PK is an actual date column and not a key like 20170401 or something like that, as I've found many of the DAX capabilities won't work correctly if you don't use a date column when relating tables.
The only other thing I can think of for you to check is to see if there are other roundabout relationships between the date table and another table that then joins back to your fact table directly or indirectly. If there's any bi-directional relationship there, it might affect what you're trying to do even though your direct relationship between date and fact table is single...
I had same issue before and I found out that as @bblais mentioned, I had two tables havng relationship with dates table. Once I set corss filter direction single for both of them. problem was resolved.