I'm usually pretty lame at showing my gratitude to friends as well, but I say it if need be and they know why it sounds awkward since they know i'm not great with that sort of thing. I may even verbalize my discomfort with the situation depending on the moment if I think it will make things less strange. Them having an idea on you character and that it's awkward for you could help ease the comfort.
As for help on remembering, or trying to understand when it's appropriate i'd try to take my time to think about if you need the extra help and maybe make a mental checklist of your own. Did they help you through something? Did they do anything for you in any way? If so I make sure I always thank the person. Maybe you can set up some guidelines to start off and go from there? It could help you feel social norms out better and help condition you so that it becomes easier over time goes by to "feel" out.
I personally always thank waiters a lot. Every time they ask if I need anything, refill my cup, or bring my food. They're providing a service for me. With friends, i'd go by if they're going out of their way for you, care to help you, or try to do something for you in any way. If they're expensing any energy for me in anyway, or I assume they are, I thank them. A mental checklist isn't foolproof, but hey, it's a start. I doubt that helped but those are just my 4 cents.
Good luck with it man. Cheers.