Sounds like your lactase persistence is diminishing or something.
http://en.wikipedia.org/wiki/Lactase_persistence