Sudsy Residue

So now there are all these antibacterial liquid soaps with scents and everything. (I personally have a favourite -- Target brand melon scented antibacterial liquid hand soap -- but that's irrelevant.) The thing is, soap is there to wash the crap off your hands, but when you're done, you're not just left with your own clean hands; you have your own clean hands and the (albeit pleasant) scent of fruits or teas or berries or something.

Does that mean the soap has left something on your hands? So aren't you just swapping crap for something else (soap additives)? Sure, with moisturizing soaps, that makes sense (and I assume the moisturizers know to stay, while the soap part knows to bugger off with the dirt.) How do they do that, to design soap that goes away, but leaves pure clean scent without residue??