I'm not 100% sure about this, but I don't think specialists make citizens unhappy -- at least not directly. However, if you have a luxury rate of over 0% and the tile you pull the worker from was generating commerce, then the city will have less commerce to go into luxuries, which will reduce happiness. Also, if you change an entertainer into another type of specialist, happiness will obviously go down.
However:
I have noticed that the least happy citizens in a city are assigned the specialist jobs. But specialists are always considered content (I think). So in addition to its regular job, a specialist makes 1 unhappy citizen -- the specialist itself -- content (assuming that at least 1 citizen in the city is unhappy).
And while the output of a scientist or tax collector is unaffected by improvements, it is unaffected by corruption as well. So in high-corruption cities, specialists are often more productive than regular workers.
My problem with specialists is as follows:
Suppose that I want to change all of the tax collectors in my empire to scientists, to focus on research. Then I have to go into all of my cities and change them manually. Arg!
The only difference between these two specialists is on my empire as a whole -- they change nothing locally in their home cities. So why can't I just decide to change my x tax collectors and y scientists into x-c tax collectors and y+c scientists, without having to manually reset them? Or equivalently, why not just have a generic tax collector/scienist specialist instead of the two separate ones, and let the player decide how much of the income they generate goes into the treasury and how much goes into research?