You're applying a particular nuance of "fond". "Not fond of" in this context does not mean "doesn't like" or even "doesn't much like" but rather, "doesn't like a lot of". The word is most commonly translated "addicted".
"Fond" usually means "like a lot". "Not fond", these days, is more...