Thanks everyone for chiming in!
I don't want to remove the siding for the reasons so well-put by goldensmom. I think it might structurally damage the house, and anyhow, the siding is in fine condition.
RCWRTR - thanks so much for offering your expertise! I'll see if I can get a picture posted.
The siding was put on at least 30 years ago and is dull and chalky - if you touch it a white residue comes off. I'm sure washing would help somewhat, but I'm really not crazy about the color. I usually like to wait a year before making any big decisions about a house, but I have found that making a cosmetic change is easier to do when you are moving in. There is so much money being thrown around that an extra couple of thousand dollars to make the house look really pretty seems very reasonable, whereas a year on I'm feeling more tight-fisted - and I really do want a home that makes me smile when I look at it.
I am seeing that paint on aluminum lasts anywhere from 5-25 years depending on preparation and type of paint used, and from my research there is some disagreement among experts on what type of primer, or if a primer is even necessary. So I'll need to find a really experienced painter and get a great warrantee! I do not want to paint every 5 years. As it needs to be sprayed (not brushed) I would not be able to do it myself. I've pretty much decided to paint - I will post a picture when I get a chance and get some feedback.
I'm not that interested in adding shutters and what not - it really is the basic color that's the problem for me. (Though I do want to get a more attractive screen door...)