By Paul Hamer
IntroductionFSL is in the process of generalizing its systems to support a wider community of users. Its Central Facility, part of Information and Technology Services, is moving toward highly modular system architectures that are customized to support particular operations. Support is provided to scientists and engineers who develop forecast systems for use now and in the future by the National Weather Service and other areas of NOAA, state and local governments, and private industry.
To accommodate ever-increasing volumes and types of data received at FSL, the data ingest, routing, and product generation systems have been converted from a legacy configuration to a system that is highly configurable and supports the processing of real-time and case study datasets. This article highlights the development of the Object Data System (ODS).
Design StrategiesThe Object Data System was developed in 2002 with the goals to reduce the time necessary to introduce new datasets and to create a system with more flexible software. For ODS, existing software packages (such as Unidata's Local Data Manager) were leveraged and the metadata were decoupled from the datasets themselves (see article by Ara Howard in this issue).
FSL's legacy system, Networked Information and Management client-Based User Service (NIMBUS), was developed in the early 1990s to accommodate conversion of proprietary operating systems to an open architecture under Unix. The basic concept was for processes to pass data via a router, the "cloud server," to other processes registered to receive these data (Figure 1). NIMBUS worked well initially, but as it became more costly to introduce new datasets and software development time increased, a more effective alternate system needed to be implemented.
Figure 1. Schematic of the legacy system, NIMBUS.
First, the "cloud server" was removed from the distribution scheme. LDM was chosen to handle data ingest and routing processes because it is widely used, software packages exist for handling common meteorological datasets, and as an open source tool, it can be extended to better support ODS. Extensions and software adjustments are regularly discussed with Unidata staff and if implemented, they are introduced in the next LDM version before release to the user community.
Existing ingest and processing were refactored to reduce the cost of software and to shorten the software development lifecycle. Initial efforts to reduce costs involved the introduction of open source software, such as the GNU Compiler Collection (GCC) and other GNU tools, for development and deployment. The laboratory also switched to the Linux platform run with less costly hardware while maintaining software portability to other platforms. The largest cost reduction was possible by reducing the software development time through the use of Object-Oriented Analysis and Design (OOAD) coupled with open source tools.
The design of ODS was primarily driven by seeking known "patterns" of software design for solutions to the problems faced in carrying out its function of supplying required data. We identified those patterns and implemented solutions accordingly. For example, access to data from LDM is achieved through use of the observer pattern, which allows clients to "see" LdmData objects in real time as they arrive (Figure 2). From this pattern we were able to develop processes to handle all datasets in a common way and therefore simplify both development and deployment. Other patterns regularly used within ODS are Singleton, Proxy, and Factory, all of which provide well-crafted solutions for our particular problems.
Figure 2. "Observer" pattern for data access.
As the data routing mechanism, LDM could be coupled with a scheme that provided the triggering of jobs on events. This is handled through a product queue action whereby "product keys" result in the spawning of jobs (Figure 3). Processes responsible for product generation (such as netCDF file creation) arbitrate between objects that are proxies for real data types and those responsible for certain products, such as GRIB objects and NetCDF. This enables the arbitration process to delegate many of the operations required in product generation to proxy objects. This vastly simplified architecture supports rapid introduction of new datasets and a more manageable distribution model for the processing of all data.
Figure 3. Object Data System architecture.
ResultsComparisons of the legacy system and ODS are encouraging. A new model ingested as GRIB, for example, requires that only an update be made to tables and a new product description be defined in the meta-data database – no software needs to be written or built. Compared to the legacy system, ingest software would require an update and rebuild along with new product generation software for each new model. Another example of potential significant savings under the new system involves the handling of new model data. In terms of developer effort, it is estimated that ODS requires less than two days to carry out a requirement to make a new model in GRIB available as NetCDF, compared with 30 days under the legacy system to support the same requirement. Similar savings have been realized for other data types.
SummaryThe development of ODS contributes to FSL's goal to generalize its systems to support a wider user community. The increased flexibility of ODS enables it to more easily run outside of the confines of the FSL Central Facility and thus support FSL's mission of technology transfer. Elements of ODS using GNU’s autoconfigure tool have been packaged for outside use. Software packages currently available from FSL include ODS elements such as LdmNexrad2NetCD, which converts Level II Nexrad data to NetCDF, and Grib2NetCDF, which converts GRIB edition 1 to NetCDF. Interested parties should contact Dr. Peter Mandics, FSL’s Chief Information Officer, at Peter.A.Mandics@noaa.gov.
Note: More information about FSL's Central Facility and other related topics is available at http://www.fsl.noaa.gov.
(Paul Hamer is a Systems Analyst in FSL’s Information and Technology Services, headed by Dr. Peter Mandics. Mr. Hamer is also affiliated with the Cooperative Institute for Research in the Atmosphere (CIRA), Colorado State University, Ft. Collins, CO. He can be reached at Paul.Hamer@noaa.gov or by phone, 303-497-6342.)