There is much confusion about the proper use of the alt attribute. The first confusion is reference to the alt attribute as the alt tag. This is more or less okay, but..... if you do a google search on alt attribute you will get results that are generated by more knowledgeable writers. So, while it is okay to use alt tag it would be better to at least know the proper term. If for no other reason when you see alt attribute used in an article you will know that they are talking about your good old friend alt tag. :)
Before we talk about the more philosophical aspects of the alt attribute let's get right to the meat of the matter.
The alt attribute is displayed when the image is not. The alt attribute is literally the alternative to the image.
Reasons for seeing alt text:
First and foremost consider that the alt text is being used by screen readers. So be compassionate. The alt attribute is NOT the place to do keyword stuffing. Please have mercy on those using screen reader software.
Some people look at webpages with image download turned off by choice. Therefore they will see the alt text. They may have turned of image download for any one of many valid (and/or invalid) reasons. The choice was theirs. The alt text should make sense for these folks. You can turn off image download in your browser to see what this looks like. If you are a stuffer, hopefully you will be suitably shamed into better alt attribute behavior after looking at your pages in this fashion.
For those will slow internet connections, they will see the alt text temporarily while the image is being downloaded. When my broadband is acting more like narrowband dial-up, I will often see the alt text on the page while waiting for the image to download.