If I want to say "Thanks for your help", which sentence is correct and which one is more commonly used?
It seems that the first one is more uncommon but I'm not really sure.
In my opinion, 帮忙 and 帮助 are different.
帮忙: [verb]to help some do something .This word itself has declared its object[忙] already. I think the phrase "do sb a favor" is a more appropriate equivalent in English.
帮助: [verb]to help. [noun]help. This word needs an object
You would say,
你可以来帮忙吗？ Could you come and help?
你可以帮助*我*一下吗？Could you help me?
我帮不上忙。 I am not able to help.
我无法帮助*你*。(Actually, 我帮不了你 is more common) I am not able to help.
Imagine that you are going to prepare a meal in your house,to entertain your friends, but you have a lot of work(buying, washing, cutting, cooking...) to do , so you ask me for help. You would say,
Huang,明早你可以来帮忙吗？Could you please come and help(do me a favor) tomorrow morning? By using 帮忙, you should not express its object, and that doesn't matter, because in the context, I understand what I can help you. I say,
没问题。我能帮助你什么呢？=没问题，我能帮什么忙呢？No problem! What can I help you?
你可以帮助我去超市买菜。 You can help me buy the foods in the supermarket. Here, you won't say 帮忙, because you have to express the object explicitly.
Now, for your question, I think the 2nd sentence is more common, though the first one could be understood. I guess you would not say "Thanks for your doing me a favor" in English, would you?
You can also say other words to express this meaning,
谢谢(or 谢了 if you are familiar with the interlocutor) 辛苦你了 辛苦了 麻烦你了 麻烦了