I think that the library-agnostic aspect is a neat thing to tout and is a feature which makes newcomers and decision-makers within a company perk up. It kind of gives CanJS that immediate "wow" factor for anybody unfamiliar with it. However, I agree wholeheartedly that its unnecessary to support all of these libraries if nobody is using anything but jQuery (Zepto).
Maybe you can silently stop updating Dojo and YUI and see if any support tickets come up - that will really tell you if it hurts anybody's feelings.