I was an athlete, and I thought the overall experience was great. Sometimes there is less fan support during midterms, etc., but overall the student body is small and tight-knit, and there will almost always be friends in the stands.
Even when it is hard, you are with such an incredible group of people that the entire experience is amazing. There is room for whatever you commit to making time for - athletics, socialization, volunteering, etc.