Took me a bit to figure this out too, you need to have a Unit inside a city with a barrack. When you hit shift-U it will offer to upgrade all of the same type of units in cities with barracks.
Its a nice balance change, no more units magically upgrading in the field.
If you're in the barracks city, you can also right-click the unit at the bottom of the city screen and select upgrade.
It seems like upgrades always either lose 1 level of veterancy, or they drop to conscript (2 hp) .. I generally have only upgraded Regulars so far, not sure which.