Understand what kind of people Tao Yuanming was would help you understand his poem. Tao Yuanming was a solitary. He spent his whole life to practice his creed －－ “不为五斗米折腰”，he prefer to live far way from worldly people, especially shame on live for making money. He wish he can live alone in the wild, but he can't escape from living pressures, he need all kind of materials like other worldly people need. He had to live near other people although he really hate to live with them. But he can't stop yearning for that. So he wrote that
心遠地自偏 => 让心（飘向我向往的）远方，（我住的地方）也就变成了那个（我向往的）偏僻的远方。 Let heart roam to the faraway (which I wish to live), then I am living there.