Well… no, you don’t.
The Post Office and Pimento Grove are exceptions to the rule for The Simpsons Tapped Out. All your other buildings come as part of a quest, and have to be unlocked in order. Sure, you can build additional houses, Kwik-E-Marts, Krusty Burgers, and Gulp ‘n’ Blows when you get to the right XP level, but the first one of each of these has to be unlocked at the right time.
The Post Office and Pimento Grove work differently. They’re not part of a quest and you unlock these simply by hitting level 23 and level 25 respectively. And if you choose not to build them, your progress won’t be affected. While both buildings earn you additional rent, the Post Office doesn’t come into use as part of the game until you get to level 25 and unlock Smithers (where it’s needed for his quest), and Pimento Grove isn’t used by any of your characters at all.
So, if you’re trying to save your money for the buildings that you actually need to unlock to keep moving ahead with your game, you really won’t need the Post Office until you unlock Smithers – and, asd to say, you really don’t need Pimento Grove at all. They’re nice to have though – and they do help your conformity bonus.