It's interesting that what we consider to be the exploitation of these workers, may be, from the worker's perspective, a great opportunity. If the employment is truly fair to the worker, then I agree, we are no more entitled to keep that job in the U.S. than anyone else in the world.
However, I don't think it is often the case that workers are treated fairly elsewhere in the world. They take these 'opportunities' because they don't have a choice between starving or working. The cocoa and chocolate trade is one such business. It is absolutely shocking the kind of abuse that workers are put through to provide cocoa to markets in the rest of the world. There are literally hundreds of markets and products that are better and worse, as well.