2.1. Overcoming Binary Portability Issues
[2. Overview]

A number of libraries have been developed to read and write binary files that are portable over multiple computing platforms. Two popular libraries include NetCDF and HDF5. After performing a search for other alternatives, we concluded that these two libraries have the most stability and support. We evaluated NetCDF and HDF5. Both libraries had similar IO performance times. HDF5 has more flexibility for data storage, compression, and data mining. HDF5 supports data folders and data structures, making it more customizable. Therefore, we have decided to utilize the HDF5 library for our generic model data API.

HDF5 is developed at the National Center for Supercomputing Applications at the University Of Illinois at Urbana-Champaign (see http://hdf.ncsa.uiuc.edu/HDF5/). HDF5 has many users both in government and private organizations. HDF5 is a free library developed primarily with funding from the Department of Energy and NASA. It is a powerful library that includes the following features:

Data is organized in a structure similar to a directory system. This makes it possible to easily store related data in a single file in an organized manner.

Data is stored in a binary format for fast file IO.

Large files (> 2 Gig) can be managed easily.

Data is platform and language independent due to automatic conversions performed by the HDF5 library. Platforms include AIX, Cray, FreeBSD, HP-UX, IRIX, Linux, OSF1, Solaris, ASCI TFLOPS, and Windows.

The data can be easily compressed using tools embedded in HDF5.

Data is stored in array like entities called datasets. A dataset can be read whole or data mined for specific elements, slices, etc.

Datasets can include metadata, called attributes by HDF5, to describe the data.