I always assume that the personal moving uphill should yield b/c 1) it is easier for the uphill hiker to see/find a more adequate place to step aside since the trail is 'moving slower' for them, 2) they are more likely to desire a quick rest anyhow, and 3) it is less straining on the uphill hiker's knees to stop for a bit.
Of course, if the downhiller steps aside well in advance, then I will hike uphill past them, thanking them en route--as well as several other mitigating circumstances.
I don't think that larger groups should have an automatic right-of-way. If anything, quite the opposite, as a single hiker can move more quickly and adeptly around them. Especially when a single is trying to pass them going the same direction!