for myself, as long as the person is not the focus of the photograph, then I most likely would not ask their permission to shoot or keep the photo. also if they are not immediately recognizable then for me I have no problem shooting it. some examples are if you are shooting a nice mural on a wall or something, and a lady walks by dressed in something that hightlights or meshes with the wall. if she is a considerable distance away and not looking at the camera, then I dont have a problem shooting it and not saying anything. she is an element of the photo, but her face is not the focus.
if i were shooting some street portrait type work, or shots where the person is the focus of the picture. i would feel compelled to at least let them know, either after the fact if the moment of the shot deemed it necessary, or prior to shooting it, if I want to do a street portrait, etc.
that is how i approach this type of photography.
as for what is morally or legally correct. there is just so many variables that play into it. i.e. the culture of the country you are shooting, etc.