Keeping the structure at temperature takes more energy than reheating the structure to temperature. (assuming similar outside temperatures at both ends of the journey) Yes it does take more time to reheat the structure than the air, but that again is a matter of comfort and not of energy consumption.
The energy utilized to raise the item (structure) to the original temperature was the same in both scenarios and is often forgotten by the individual who suggests not lowering the temperature. The energy required to
keep anything at an elevated temperature is higher than if you allow the temperature of that item to drop for a period of time and then reheat it. If that were not the case, we would have a brand new source of energy on this planet by tapping into this "magic energy" that people believe exists in structures.
Please, do not confuse energy consumption with comfort - they are often inversely proportional - one sacrifices (what most people call) comfort for the sake of energy savings. Although large temperature setbacks on a daily basis do not save
much additional energy, they do save
some additional energy and when extended for longer periods of time - a 2 week vacation - a lot of energy is saved.
You simply need to look at your heating bill in the winter if you go away for an extended period and lower the thermostat, or your cooling bill (that portion for A/C) in the summer if you have A/C and raise the thermostat. The bill would include the reheating or recooling of the structure. Of course, these bills would need to be adjusted for heating/cooling degree days. If you had a really mild winter or a cool summer, the savings would not be immediately visible if you didn't account for heating/cooling degree days.