I do find this confusing even though I understand the grammatical logic behind it. But my (temporary) solution is to get away from thinking in English and adopt the Spanish viewpoint. So I think “I please you” (te gusto); “you please me” (me gustas); “he pleases them” (les gusta) etc. , rather than "you fancy me"......
I believe it’s better as a general principle to try to think in the target language, rather than translate from your own language into the target language.
Hope this helps.