It certainly can be very polluting, in part because China doesn't give a fuck but in part because of the unusual methods required to extract them. And you're right, most of them are not rare. The main problem is that they're very, very difficult to identify and then purify/separate. Most rare earth elements belong to the lanthanide group and they're chemically similar to one another because most of them have identical outer electron shells. It's REALLY tough to identify them using traditional chemical methods. Consequently, extraction requires very specialized equipment and techniques, something we have in rather short supply here in the states.