Index fossils are the remains of organisms that lived for a geologically short time but spread over a wide area, so finding one in a rock layer pins down that layer’s relative age. Good index fossils are abundant, easy to identify, and widely distributed; ammonites, graptolites, trilobites, and microfossils such as foraminifera are classic examples.
They matter because rock layers of the same age can look completely different from place to place. By matching index fossils, geologists in the field of biostratigraphy can correlate strata between distant sites, define biozones, and subdivide the geologic time scale.
Source: Biostratigraphy, Wikipedia
Image: Pleuroceras solare ammonite by H. Zell, CC BY-SA 3.0, via Wikimedia Commons.