Influence via insulation

Rabbi Yosef ben Kisma was offered all the gold and silver in the world to leave his city of Torah scholars and settle elsewhere, and he refused. He could have built a yeshiva there and lifted everyone around him. Why didn't he? The Mishnah's wording of the offer holds a small redundancy. It also settles a tension in Parshas Behaaloscha: the Levi'im are told to be set apart, and yet they must teach everyone. And how can anyone follow the Rambam's advice to live among good people, if everyone does the same?