Performance-wise, how does HDF5 compare to a relational database?
It really depends on your application. HDF5 is tuned to do efficient I/O and storage for "big" data (hundreds of megabytes and more). It will not work well for small reads/writes.
It doesn't have indexing capabilities, though we are working on some limited features.
HDF5 was designed to complement DBs and not to compete with them.