The reason for English speakers having difficulties with Korean or Japanese is that English follows a SVO syntax structure (Subject Verb Object: I eat apple) while Korean or Japanese uses a SOV syntax structure (subject object verb: I apple eat 나는 사과를 먹는다/私はリンゴを食べる). The verb for Korean/Japanese comes last, not right after the subject noun.
If you've intuitively had verbs follow your initial pronouns, it's going to be very difficult to grasp the flow of the language in another tongue. Also, Korean or Japanese is contextual as well, meaning that sometimes the subject isn't actually said sometimes, because it's already being inferred.
Honestly I don't think the language part isn't the hardest aspect, I think it's cultural mannerisms and etiquette and the soft philosophies behind the language or culture that's the hardest part to grasp.
Japanese for example is very nonconfrontational, and a lot of the language seems like it's just muttering factual statements without the input of emotions, but it's inferred from the language in an indirect way what the speaker is trying to convey. Not very direct, but implicit. Korean on the other hand is very developed emotionally in that the cultural format of the language prioritizes emotional standing over factual statements. A lot of the speaking revolves around emotional tenor rather than some kind of back and forth you have in English. In English you can be blunt about things, like for example you can simply say that the movie sucked. The listener can then reply 'oh why do you think so?'. But in Korean, you usually say "I think that the movie wasn't that interesting" or, "For me, the movie wasn't great, that's what I think".
In English a lot of it is roundabout, but once you realize the relationships dynamics in the culture, that's when you realize why they speak the way they do.