957:
157:
267:
45:
239:
HDF is self-describing, allowing an application to interpret the structure and contents of a file with no outside information. One HDF file can hold a mix of related objects which can be accessed as a group or as individual objects. Users can create their own grouping structures called "vgroups."
248:(Scientific Dataset) objects support arbitrary named attributes, while other types only support predefined metadata. Perhaps most importantly, the use of 32-bit signed integers for addressing limits HDF4 files to a maximum of 2 GB, which is unacceptable in many modern scientific applications.
211:
The quest for a portable scientific data format, originally dubbed AEHOO (All
Encompassing Hierarchical Object Oriented format) began in 1987 by the Graphics Foundations Task Force (GFTF) at the National Center for Supercomputing Applications (NCSA). NSF grants received in 1990 and 1992 were
195:
for general use. HDF is supported by many commercial and non-commercial software platforms and programming languages. The freely available HDF distribution consists of the library, command-line utilities, test suite source, Java interface, and the Java-based HDF Viewer (HDFView).
243:
The HDF4 format has many limitations. It lacks a clear object model, which makes continued support and improvement difficult. Supporting many different interface styles (images, tables, arrays) leads to a complex API. Support for metadata depends on which interface is in use;
293:
In addition to these advances in the file format, HDF5 includes an improved type system, and dataspace objects which represent selections over dataset regions. The API is also object-oriented with respect to datasets, groups, attributes, types, dataspaces and property lists.
312:
data such as stock price series, network monitoring data, and 3D meteorological data. The bulk of the data goes into straightforward arrays (the table objects) that can be accessed much more quickly than the rows of an
290:. Metadata is stored in the form of user-defined, named attributes attached to groups and datasets. More complex storage APIs representing images and tables can then be built up using datasets, groups and attributes.
256:
The HDF5 format is designed to address some of the limitations of the HDF4 library, and to address current and anticipated requirements of modern systems and applications. In 2002 it won an
188:, it is supported by The HDF Group, a non-profit corporation whose mission is to ensure continued development of HDF5 technologies and the continued accessibility of data stored in HDF.
228:
HDF4 is the older version of the format, although still actively supported by The HDF Group. It supports a proliferation of different data models, including multidimensional arrays,
440:
856:
257:
185:
987:
1002:
317:
database, but B-tree access is available for non-array data. The HDF5 data storage mechanism can be simpler and faster than an SQL
282:
This results in a truly hierarchical, filesystem-like data format. In fact, resources in an HDF5 file can be accessed using the
485:
enables users to manage HDF5 files through a high-level language (similar to SQL) in C, C++, Java, Python, C#, Fortran and R.
997:
236:
for reading, writing, and organizing the data and metadata. New data models can be added by the HDF developers or users.
898:
829:
233:
813:
992:
365:
853:
220:(EOS) project. After a two-year review process, HDF was selected as the standard data and information system.
609:
450:
446:
620:(a high-level interface with advanced indexing and database-like query capabilities). HDF4 is available via
546:
649:
530:
523:
binding that takes a different approach from the official HDF5 Java binding which some users find simpler
520:
399:
340:
Dataset data cannot be freed in a file without generating a file copy using an external tool (h5repack).
574:
504:
454:
468:
678:
635:
426:
355:
672:
513:
191:
In keeping with this goal, the HDF libraries and associated tools are available under a liberal,
17:
639:
922:
778:
217:
119:
8:
184:) designed to store and organize large amounts of data. Originally developed at the U.S.
956:
156:
730:
52:
98:
756:, The Netcdf Java library reads HDF5, HDF4, HDF-EOS and other formats using pure Java
629:
568:
463:
962:
759:
666:
498:
127:
693:
690:
A data visualization program that reads the HDF, HDF5 and netCDF data file formats
31:
860:
817:
229:
329:
Criticism of HDF5 follows from its monolithic design and lengthy specification.
873:
790:
278:
Groups, which are container structures that can hold datasets and other groups
263:
HDF5 simplifies the file structure to include only two major types of object:
981:
458:
934:
833:
748:
705:
643:
621:
492:
192:
950:
902:
810:
232:, and tables. Each defines a specific aggregate data type and provides an
143:
598:
417:
396:
HDF5 Dimension Scale (H5DS) – allows dimension scales to be added to HDF5
318:
309:
173:
617:
421:
266:
625:
592:
375:
88:
628:
for both Python 2 and Python 3. The popular data manipulation package
554:
550:
717:
564:
711:
540:
509:
430:
702:
A browser and editor for HDF5 and PyTables files written in Python
439:
introduced support for HDF5 export using an implementation called
44:
687:
675:
HDF5 Plugin for Apache Drill enables SQL Queries over HDF5 Files.
560:
488:
371:
305:
653:
476:
753:
714:
A browser for HDF files specifically for synchrotron X-ray data
616:(both high- and low-level access to HDF5 abstractions) and via
588:
584:
436:
337:, so client applications may be expecting ASCII in most places.
298:
199:
The current version, HDF5, differs significantly in design and
972:
578:
720:
For viewing HDF files in the browser or in Visual Studio Code
390:
360:
334:
283:
563:
can gain HDF support through third-party libraries, such as
742:
736:
699:
604:
536:
472:
411:
213:
433:, with a high-level h5py style D wrapper under development
314:
200:
652:
can gain HDF support through third-party libraries like
595:– use HDF5 as primary storage format in recent releases
533:
implementation providing read-only access to HDF5 files
482:
383:
HDF5 Image (H5IM) – a C interface for images or rasters
613:
443:(Scientific Data Format) with release Dymola 2016 FD01
216:
investigated 15 different file formats for use in the
967:
501:
uses HDF5 as primary storage format since version 3.5
745:(GRIdded Binary), a data format used in meteorology
682:Interoperable HDF5 data product creation GUI tool
526:
380:HDF5 Lite (H5LT) – a light-weight interface for C
275:Datasets, which are typed multidimensional arrays
979:
632:can import from and export to HDF5 via PyTables.
393:to handle "packet" data, accessed at high-speeds
389:HDF5 Packet Table (H5PT) – interfaces for C and
186:National Center for Supercomputing Applications
601:offers immediate analysis of HDF and HDF5 data
386:HDF5 Table (H5TB) – a C interface for tables
349:
308:to index table objects, HDF5 works well for
212:important to the project. Around this time
955:
549:support for HDF5 is available through the
155:
265:
34:, the file system used in Apache Hadoop.
405:
14:
980:
830:"Are there limitations to HDF4 files?"
93:application/x-hdf, application/x-hdf5
968:HDF-EOS Tools and Information Center
203:from the major legacy version HDF4.
871:
24:
988:C (programming language) libraries
762:- Google's data interchange format
708:A netCDF, HDF and GRIB Data Viewer
696:A browser and editor for HDF files
25:
1014:
942:
739:, a data format used in astronomy
333:HDF5 does not enforce the use of
1003:Meteorological data and networks
811:How is HDF5 different from HDF4?
206:
43:
779:Java-based HDF Viewer (HDFView)
669:HDF5 Connector for Apache Spark
301:, version 4, is based on HDF5.
928:
916:
891:
865:
847:
822:
804:
783:
772:
13:
1:
766:
344:
7:
998:Earth sciences data formats
854:R&D 100 Awards Archives
724:
667:Apache Spark HDF5 Connector
459:bindings for BEAM languages
324:
10:
1019:
29:
925:Mathematica documentation
414:uses HDF5 as main storage
350:Officially supported APIs
138:
125:
115:
107:
97:
87:
51:
42:
673:Apache Drill HDF5 Plugin
660:
166:Hierarchical Data Format
89:Internet media type
38:Hierarchical Data Format
30:Not to be confused with
973:Open Navigation Surface
874:"Moving away from HDF5"
251:
223:
791:"History of HDF Group"
684:( no longer available)
638:offers support in the
519:JHDF5, an alternative
297:The latest version of
271:
218:Earth Observing System
120:Scientific data format
993:Computer file formats
923:HDF Import and Export
493:bindings to the C API
431:bindings to the C API
270:HDF Structure Example
269:
679:HDF Product Designer
514:full support of HDF5
406:Third-party bindings
878:cyrille.rossant.net
39:
27:Set of file formats
872:Rossant, Cyrille.
859:2011-01-04 at the
816:2009-03-30 at the
731:Common Data Format
612:supports HDF5 via
272:
53:Filename extension
37:
464:GNU Data Language
288:/path/to/resource
258:R&D 100 Award
163:
162:
108:Developed by
103:\211HDF\r\n\032\n
16:(Redirected from
1010:
959:
954:
953:
951:Official website
937:
932:
926:
920:
914:
913:
911:
910:
901:. Archived from
895:
889:
888:
886:
884:
869:
863:
851:
845:
844:
842:
841:
832:. Archived from
826:
820:
808:
802:
801:
799:
797:
787:
781:
776:
760:Protocol Buffers
499:Huygens Software
304:Because it uses
193:BSD-like license
159:
154:
151:
149:
147:
145:
131:
83:
79:
75:
71:
67:
63:
59:
47:
40:
36:
21:
1018:
1017:
1013:
1012:
1011:
1009:
1008:
1007:
978:
977:
949:
948:
945:
940:
933:
929:
921:
917:
908:
906:
899:"JHDF5 library"
897:
896:
892:
882:
880:
870:
866:
861:Wayback Machine
852:
848:
839:
837:
828:
827:
823:
818:Wayback Machine
809:
805:
795:
793:
789:
788:
784:
777:
773:
769:
727:
663:
408:
352:
347:
327:
254:
226:
209:
142:
126:
81:
77:
73:
69:
65:
61:
57:
35:
28:
23:
22:
15:
12:
11:
5:
1016:
1006:
1005:
1000:
995:
990:
976:
975:
970:
965:
960:
944:
943:External links
941:
939:
938:
927:
915:
890:
864:
846:
821:
803:
782:
770:
768:
765:
764:
763:
757:
751:
746:
740:
734:
726:
723:
722:
721:
715:
709:
703:
697:
691:
685:
676:
670:
662:
659:
658:
657:
647:
633:
607:
602:
596:
582:
572:
558:
544:
534:
524:
517:
507:
502:
496:
486:
480:
466:
461:
444:
434:
424:
415:
407:
404:
403:
402:
397:
394:
387:
384:
381:
378:
369:
363:
358:
351:
348:
346:
343:
342:
341:
338:
326:
323:
280:
279:
276:
253:
250:
225:
222:
208:
205:
172:) is a set of
161:
160:
140:
136:
135:
132:
123:
122:
117:
116:Type of format
113:
112:
109:
105:
104:
101:
95:
94:
91:
85:
84:
55:
49:
48:
26:
9:
6:
4:
3:
2:
1015:
1004:
1001:
999:
996:
994:
991:
989:
986:
985:
983:
974:
971:
969:
966:
964:
963:What is HDF5?
961:
958:
952:
947:
946:
936:
935:PDL::IO::HDF5
931:
924:
919:
905:on 2020-12-01
904:
900:
894:
879:
875:
868:
862:
858:
855:
850:
836:on 2016-04-19
835:
831:
825:
819:
815:
812:
807:
792:
786:
780:
775:
771:
761:
758:
755:
752:
750:
747:
744:
741:
738:
735:
732:
729:
728:
719:
716:
713:
710:
707:
704:
701:
698:
695:
692:
689:
686:
683:
680:
677:
674:
671:
668:
665:
664:
655:
651:
648:
645:
641:
637:
634:
631:
627:
623:
619:
615:
611:
608:
606:
603:
600:
597:
594:
590:
586:
583:
580:
576:
573:
570:
566:
562:
559:
556:
552:
548:
545:
542:
538:
535:
532:
528:
525:
522:
518:
515:
511:
508:
506:
503:
500:
497:
494:
490:
487:
484:
481:
478:
474:
470:
467:
465:
462:
460:
456:
452:
448:
445:
442:
438:
435:
432:
428:
425:
423:
419:
416:
413:
410:
409:
401:
398:
395:
392:
388:
385:
382:
379:
377:
373:
370:
367:
364:
362:
359:
357:
354:
353:
339:
336:
332:
331:
330:
322:
320:
316:
311:
307:
302:
300:
295:
291:
289:
286:-like syntax
285:
277:
274:
273:
268:
264:
261:
259:
249:
247:
241:
237:
235:
231:
230:raster images
221:
219:
215:
207:Early history
204:
202:
197:
194:
189:
187:
183:
179:
175:
171:
167:
158:
153:
141:
137:
133:
129:
124:
121:
118:
114:
111:The HDF Group
110:
106:
102:
100:
96:
92:
90:
86:
56:
54:
50:
46:
41:
33:
19:
930:
918:
907:. Retrieved
903:the original
893:
881:. Retrieved
877:
867:
849:
838:. Retrieved
834:the original
824:
806:
794:. Retrieved
785:
774:
749:HDF Explorer
688:HDF Explorer
681:
577:through the
457:may use the
328:
303:
296:
292:
287:
281:
262:
255:
245:
242:
238:
227:
210:
198:
190:
181:
177:
174:file formats
169:
165:
164:
99:Magic number
622:Python-HDF4
599:Mathematica
418:Common Lisp
319:star schema
310:time series
128:Open format
982:Categories
909:2009-06-10
840:2009-03-29
767:References
376:Fortran 90
345:Interfaces
150:/solutions
646:packages.
565:h5labview
557:packages.
541:hdf5-json
422:hdf5-cffi
146:.hdfgroup
883:21 April
857:Archived
814:Archived
725:See also
700:ViTables
618:PyTables
581:library.
579:lua-hdf5
539:through
510:IGOR Pro
479:package.
420:library
325:Feedback
796:15 July
706:Panoply
694:HDFView
624:and/or
561:LabVIEW
529:A pure
512:offers
491:offers
489:Haskell
429:offers
372:Fortran
306:B-trees
139:Website
18:HDF-EOS
754:NetCDF
630:pandas
610:Python
593:Octave
589:Scilab
585:MATLAB
569:lvhdf5
516:files.
453:, and
451:Elixir
447:Erlang
437:Dymola
368:- .NET
299:NetCDF
733:(CDF)
718:h5web
661:Tools
644:hdf5r
640:rhdf5
626:PyHDF
547:Julia
483:HDFql
473:gonum
335:UTF-8
284:POSIX
152:/hdf5
78:.hdf5
66:.hdf4
885:2016
798:2014
743:GRIB
737:FITS
712:silx
654:hdf5
650:Rust
642:and
614:h5py
605:Perl
567:and
555:JLD2
553:and
551:HDF5
537:JSON
531:Java
527:jHDF
521:Java
477:hdf5
412:CGNS
400:Java
252:HDF5
224:HDF4
214:NASA
182:HDF5
178:HDF4
148:.org
82:.he5
70:.he2
58:.hdf
32:HDFS
591:or
575:Lua
505:IDL
475:'s
455:LFE
441:SDF
391:C++
366:CLI
361:C++
315:SQL
234:API
201:API
170:HDF
144:www
134:Yes
74:.h5
62:.h4
984::
876:.
587:,
471:-
469:Go
449:,
374:,
321:.
260:.
246:SD
180:,
80:,
76:,
72:,
68:,
64:,
60:,
912:.
887:.
843:.
800:.
656:.
636:R
571:.
543:.
495:.
427:D
356:C
176:(
168:(
130:?
20:)
Text is available under the Creative Commons Attribution-ShareAlike License. Additional terms may apply.
.svg){kind=logo}
