I always found it misleading to think about it as "calories" instead of just grams. We don't transform matter into energy. We just rearrange molecules around, mostly breaking big molecules into smaller molecules, storing some and excreting the rest. Some energy gets liberated in the process, but that's only the energy that was binding the molecules together. I dunno, maybe it's just me.