A potential fix for AI worker inefficiency is to artificially give AI foresight.
worker management is all about knowing how many workers you want in the future and keeping a checkbox in your head checked at all times that because you need there workers for the "future" if they don't have a job to do, the last thing you should do is disband them.
1) Every 5 or so or however many tunrs, the AI asses its empire and decide on its optimum worker count. Once it is decided, this number is FIXED.
If workers are lost due to war, they are replaced after a given fraction of the workers are lost.
Alternatively
2) We can use a worker maximization model with constraints, like a standard linear programming problem. Constraints to the model would include terrain improvements required, ratio of workers to empire size, ratio of workers to military size, existing workers, and gold income.
The gold constraint should be somewhat loose, perhaps with a nice range of say 20 gold.
This way, a sudden bump in the budget of 2 gold from a city improvement doesn't immediately require the disbanding of worker units.
The 2nd alternative sounds more flexible and follow the spirit of the game AI, which was to create a set of basic instructions, without too much specifics. So naturally, in playtestings, there may be tweaks that is required and who knows what else might happen.