A soft fork is when previously valid blocks are now invalid whereas a hard fork means that previously invalid forks are now valid.
A hard fork is necessary when a certain rule needs to be removed, for (a stupid) example: Nonces can now be base 16 and include alphabetic characters as well.
A hard for will split the community temporarily, until 100% of nodes update to allow for the previously invalid blocks. This can lead to very large parallel chains of blocks occupying the same height in the blockchain, which will eventually break off to become stale and potentially remove tons of transactions when the fork is resolved. The alternative is the formation of an entirely new token because the community has split as a result of disagreeing on what they believe should be valid.