I think people talking about premature optimisation are often talking about micro-optimisations. Those are almost always unnecessary until you’ve identified choke points. Optimising the overall architecture of the code base on the other hand, is in my opinion something that should be thought about before you even start coding. That’s where the major gains can often be done anyway.