How do I find out what compression has been used to write a dataset?

Within an application, you can open the dataset with H5Dopen(), query the dataset's creation property list with H5Dget_create_plist() to get the number of filters defined for the dataset with H5Pget_nfilters(). Then you can loop from 0 to n-1 and call H5Pget_filter() to retrieve information about each filter.

See the Examples by API on the HDF5 Examples page for example programs that write and read compressed datasets using compression filters that are in the HDF5 library.

The FAQ Using Compression in HDF5 has links to information on compression, including how to read datasets with user defined compression filters.