I hate starting near the Romans and the Japanese.
Romans are a pain. Not only their unit is good on the defence as well as on the offence, you'll be often defeated, and their militaristic skills will upgrade their unit to Veteran or Elite. Good ways to stop this are to launch pinpoint operations on their iron resources. Or , even better, wait till the Mid Ages , esp. If you're playing with an Asian civ.
Which bring me to my next contender to the throne of the pain in the ass: The Japanese.
These guys are generic in the ancient world, They fight just like the next guy... but in the early middle ages springs The Samurai. This badass mofo is a ***** to play against. He needs no horse resources ( usually , a minor benefit ) AND he's a kickass defensive unit , to cling onto the city he's just captured, and we must not forget , honed his skills to elite level ,as he was almost certainly produced in a city with some cheap barracks, and recieved an upgrade during the fight with you. And when he's dug in, You mustn't forget that he's not your average grunt, mustering a 2nd speed level, so don't even think about trying to attack his ass with knights , constantly sending the multitude of badly damaged units for a fix up at your nearest friendly city, until one of your Raiders/Elephants/Knights somehow miraculously pounds his way through the smooth silky robes of the fearless warrior. These guys fight to the end, And the end is bad for you, usually. Playing with them as a civ , Is fun fun fun, though.
my only hint for you: Either make them small before middle ages , or wait to the arrival of the Cavalry, literally.