I know it's probably not the best way to say it. Perhaps something more elaborate like "他想要在家里“ would sound better. But is the barebones version 他要在家 incorrect? To be clear, I mean this as a complete sentence, not the first half of a sentence like 他要在家看书， but just a complete sentence: "他要在家。” Maybe he's homesick, or something like that.
If you want to convey literal homesickness you can say 想家, or more generally 懷鄉 for hometown/land.
The phrase you wrote is correct, but with no other context I would read it as needing to stay home (for something)-- I personally would not think in the direction of homesick or the like for that phrasing.
Chinese is highly context reliant, so a good phrase for wanting to stay home for homesickness, wanting to stay home to study, or wanting to stay home to rest are unlikely to be covered by just one term/phrase.
Some non homesick examples:
我想留在家好好休息 I want to stay at home to rest
我想留好好看著家 I want to stay at home to watch over the house (while everyone else is out etc.)
我想留在家讀完大學了 I want to stay at home to finish college.
If you have a specific scenario in mind it'll help in determining the best way to say something :)
呆在家里 (dāi zài jiā lǐ) might be what you're looking for. For example, "我只是想呆在家里" could be translated to "I just want to stay at home," and "我放假通常呆在家里" could be translated to "I usually stay home on my days off."
Reverso-Context is a good resource for questions like these. You can find my examples there, plus more