I agree: swimming will make you more fit, and works great as general aerobic or anaerobic exercise, but won't necessarily build up muscles beyond a reasonable baseline. However, I'd argue that it works far better than walking or running if you want general calorie burning.
On a different note, though: swimming works very well for building lung capacity, which cross-trains very well with other exercises.
I can attest to the final point. I had to give up running for a month to heal from shin splints and I started swimming and biking to train in the meantime. My previous 5k personal best was 24:00 and when I ran it again after that month of swimming/biking I ran it in 21:52 with relative ease. I attribute it to the swimming which kicked my ass.
On a different note, though: swimming works very well for building lung capacity, which cross-trains very well with other exercises.