There's always crossing with 20+ attack units
Seriously, it depends on their stance with you, what you're using and how close to a city you get.
A worker often gets a warning after a turn or two and is booted a similar time afterwards.
Gracious AIs may ignore a unit for a turn but will quickly drop to polite.
A transport is counted as being full of nasty invading units.
And remember, police your own borders just as vigilantly!