About DBNet

DBNet is a distributed data communications and processing system. DBNet provides a reliable, scalable, maintainable, and extendable data flow infrastructure. DBNet uses a data/event driven model to provide optimum performance under a variety of operating environments. DBNet is also capable of providing a robust real-time dissemination system for most any kind of data.

The basic operating model provided by DBNet is an event/action model. DBNet performs an action for every event generated within it's dataflow environment. DBNet's versatility and power are defined by the flexibility of it's event/action definitions. An event can be as simple as the availability of a file, with it's corresponding action being the transfer of the file from it's remote system to the local system. One DBNet action can generate a new event, and therefore a new action. Event/action chains can be built to support complex data transfers and processing.

DBNet is accessed by external programs by the use of "bindings". Bindings are programs that inject specific types of data into the DBNet dataflow stream. External processes use DBNet by calling an external binding to generate a DBNet event for the specific data being injected. DBNet currently has support for the following bindings:

• BUFR format messages
• WMO format messages
• X224 raw data
• Remapped Satellite Images
• ALERT format messages

The BUFR binding is used to gather and transmit BUFR data. The ALERT binding is used to signal when a file is available on a computer. The WMO binding is used to ingest WMO data into DBNet. The satellite binding is used to distribute remapped satellite images from various NESDIS computer systems.

The best way to describe DBNet is by example. Consider the following scenario:

The AVN model runs on Cray3. The OSO anonymous FTP server needs the output from the AVN model run. How is the FTP server going to get the data? How is the FTP server going to know when the data is available?

As the model runs, it will call DBNet, via the ALERT binding, as each component created during the AVN run becomes available. DBNet will then send an ALERT message to the FTP server. The FTP server now knows what data is available, and can pull the data from the Cray. This model transfer can be configured to occur every time the AVN model runs.

Now, add the DAS1 server to the list of systems requiring the AVN model data. To notify DAS1 that the data is available, only requires Cray3 to also send the ALERT message to it. So the basic idea is:

Now lets expand the scenario a little. Let's add another host that will get the AVN data from DAS1, instead of Cray3. Because the definition of an ALERT is the notification that a file is available, when the AVN output is pulled from Cray3 to DAS1, a new ALERT is generated. The new ALERT indicates that the AVN data is now available on DAS1. DAS1 can be configured to send this ALERT to the new computer. The new computer will now pull the data from DAS1. By setting up the chain like this, we create a hierarchical data flow, reducing the workload and network traffic on Cray3. This implies that a delay, in forwarding the data, will occur at each level of the hierarchy. Therefore, the most time-critical systems should be served from the highest possible level of the hierarchy.

The basic idea demonstrated above is that once DBNet is configured for a particular kind of data, the transmission details become transparent. DBNet will deal with network outages, computer outages and delays in data availability. DBNet configuration details were left out of our example. DBNet uses two configuration files, the parsing table and the ystem configuration file. The parsing table is used to specify the dataflow configuration. The system configuration file is used for specifying DBNet system tuning parameters. Since every DBNet installation has a unique function, every DBNet has a unique version of the parsing table and system configuration file.

Each computer in the example above has it's own parsing table. The Cray3 parsing table has entries to indicate that the AVN model data should be sent to the OSO FTP server and DAS1. The OSO FTP server parsing table table will have an entry indicating that the AVN model data is to be retrieved and stored locally. The DAS1 parsing table will have an entry to retrieve and store the data locally, and then send an ALERT to the next system in the hierarchy.

The most difficult part of DBNet is maintaining the parsing tables. All of the details we skipped over in the example are in the parsing table. DBNet processes data based on Message IDs and Host IDs. The parsing table defines these IDs. The Host IDs are fairly straight forward. Each host, normally distinguished by it's IP address, has a unique Host ID. The first section of the parsing table defines these Host IDs.

The Message IDs are a bit more complicated. When data is submitted to DBNet there are a minimum of three parameters required. The three parameters are the type, sub-type and filename. These three parameters are used to search the parsing table for a Message ID. The Message ID section of the parsing table contains a minimum of four fields, the type, sub-type, filename and Message ID. For example:

In the AVN model data example above the data is submitted to DBNet via the ALERT binding. The call to the ALERT binding looks like this:

The four parameters are as follows:

1. MODEL -- the type
2. AVN_PGB -- the sub-type
3. J883R -- the job that produced the output, we'll discuss this in greater detail later.
4. /com/avn/prod/avn.19991207/avn.t00z.pgrb.tm00 -- the fully qualified filename.

The results of this call will be the generation of an ALERT message. The Message ID for this ALERT message will be determined by searching the parsing table for an entry matching

The Host ID for this ALERT will be determined by searching the parsing table for the IP address of the computer running the dbn_alert binding. The dbn_alert binding uses the hostname of the computer to determine it's IP address. If both the Host ID and Message ID are found then an ALERT is generated. The ALERT for this file will look like:

where "ALERT" is the message format, "1002" is the Host ID for Cray3, "MAVNPGB" is the message ID for the AVN pressure grib files, and "1.96797.865849413" is used internally by DBNet (see DBNet Alerts for more details).

The parsing table will then be searched again for the commands pertaining to this ALERT. The search is based on the message format (ALERT), the Host ID (1002) and the Message ID (MAVNPGB). All the commands that apply will be queued for execution by DBNet.