A telephoto is essentially a telescope. It allows you to get a better view of a distant object by manipulating the light to seemingly bring that object closer. The effect of this is that it compresses the distance between the viewer and the object. You can see this in some movies, such as the original Vanishing Point, in which the car is approaching at 120mph, but the distance is so compressed by the lens that it looks like it is barely moving. With a long lens and a lot of depth-of-field (small apperture that puts more of the image in focus), movement of near and distant objects looks odd because of the way the distance between them is compressed in the image.
For instance, imagine you have a 4x lens. There is a person 20' away from you and a person 100' away from you, almost hidden by the nearer person. In your video, they appear to be 5' and 25' away. Now, while you are recording, move the camera one foot to the right. As you look with your eyes, they separate by a normal distance, but, in the video, the person who appears to be 25' away moves by a distance that the viewer would expect a person 100' away to move – that is, the separation looks to be much greater than it should be, based on normal perspective.