Entuity StormWorks Data Structures

Structure of user defined types

Making associations between types

Defining attributes of types

StormWorks streams

The Entuity data model is implemented through StormWorks in two ways:

• StormWorks Data Dictionary, which allows you to interrogate the StormWorks data model.
• StormWorks Data Structures, which provides an overview of the concepts behind StormWorks.

Data type is the underlying concept of StormWorks. Types can be split into the following:

• built-in types, e.g. floats, strings.
• structs, which are defined within the StormWorks configuration files, but are not extended from a type. This means that they are not written in the database, but are used as a method of combining attributes (which is useful in some calculations).
• user-defined types, e.g. module, device. These are defined within the StormWorks configuration files, and are extended from other types. User-defined types are the most important types when defining data export jobs.

Structure of user defined types:

Types are defined through StormWorks configuration files. These flat text files are divided into headed sections. Within each of these headed sections are the definitions for the different types, streams, attributes and other entities that are used to configure StormWorks. These text files also specify the relationships between these types, e.g. which type extends from another, which type is associated with another.

Different aspects of constructing a type:

• Associations define relationships between types, this ties related information collected through different types together. e.g., the types port and device have a many to one relationship. A device can have many ports a port can only have one device.
• Attributes define data that can be polled. Attributes can be held:
  • directly against a type. This implies that history data for that attribute is not required.
  • against a stream within a type. This is for time-series data.
• Streams define properties of the polling process. A stream can only be connected to one type (although through type inheritance it can appear otherwise).
  
  For example, in the inheritance example, the stream chassisInventory is defined against the device type, and is then inherited by the SwitchDevice, RouterDevice and LBDevice types.
  
  StormWorks also has the concept of a virtual stream. A virtual stream does not contain directly polled data, but rather data calculated from polled data.
  
  For example, when Entuity polls for device uptime, Entuity will use this value to check whether the device has been continuously up since it was last polled or whether it has been down. The virtual stream v deviceUpTime maintains the poll timestamp, the amount of time the device was up and the amount of time the device was in an unknown state. You can then use this virtual stream when reporting on device uptime.
• Types can be connected to more than one stream. For example, port could have two streams:
  
  • portData, records inbound and outbound octets, port speed and duplex information collected every two minutes.
  • shortUtilization, records short term utilization (which is actually based on the octets, speed information and time stamp collected through portData) calculataed every two minutes.
• Event State Engine processes the specified poll data from the specified stream.
• Collectors describe a method of how an attribute can be polled. An attribute can have a number of collectors, which allows different methods for attaining the same type of information. Entuity uses the collector's priority level to determine which attribute method to try first. It works through the attribute collectors until a method is successful.
• Transform allows data to be converted from one data type to another, e.g. to change an integer to a string.

Type extension and type inheritance

The majority of StormWorks types are extended from other StormWorks types. They inherit the characteristics of those types, e.g. streams and attributes. This allows type definitions to be built from the general to the specific, which makes for easier type definition and maintenance.

When types are configured and compiled to Entuity, different instances of the same object can have different characteristics. For example, switches, routers and load balancers are classed as devices, but the device instance of each reflects their different inheritances. All are extended from the same type, DeviceEx, which contains generic device information, e.g. attributes, associations that are common to three device types. These common attributes are inherited, and specific device type characteristics are defined against LBDevice and SwitchDevice. RouterDevice only inherits attributes and streams, and does not have specific device type characteristics defined against it.

Using the above device example, the structure of type extension, and attribute and stream inheritance, is as follows:

1. type, the original type from which almost all other types are extended. The exceptions are types used as structs.
   • This has two attributes, id and type. All subsequent types extend from this type, and inherit these attributes.
2. BasicDevice, holds characteristics general to devices.
3. device, holds characteristics general to devices.
   • The BasicDevice and device types inherit the id and type attributes from the type, but also extends the definition. device defines data that can be collected against devices in general. Implicit in this is that the device itself will be extended by types that refer to particular devices. Therefore, device includes the stream chassisInventory that is common to all devices.
4. DeviceEx, extends the device definition.
   • This has additional attributes and streams defined against it. It is also used for reporting, rather than device.
5. SwitchDevice and RouterDevice, characteristics specific to switches and routers, respectively.
   • These types inherit all the characteristics of DeviceEx, Device, BasicDevice and type. They also extend the definition by including characteristics - in this example streams that are unique to each.

Once these types are compiled into Entuity, they affect the characteristics of a discovered device. For example, when Entuity discovers a new switch, it first discovers it as a device, then BasicDevice, then DeviceEx, and then finally as a SwitchDevice.

The object instance that is created is of SwitchDevice, but because it is derived from a composite type, it can still be referred to as a device, and therefore retains its original object identifier.

Each discovery cycle is scheduled, which means that a new cycle only starts when the previous cycle has completed. This explains why the attributes associated with a newly discovered device appear in stages.

Making associations between types:

Associations define relationships between types. They tie together related information that has been collected through different types.

For example, the default StormWorks configuration includes an association between the types port and device. A device has one or more ports, and this relationship is defined through the DevicePort Association. This association identifies device as the primary (parent) type and ports as the secondary (child) type. It also indicates that there is a one-to-many relationship between them.

Defining attributes of types:

StormWorks attributes can be grouped by their usage:

• data attributes store information collected from network objects.
• index attributes identify discovered network objects.
• association attributes make associations between types.

Defining data attributes

Data attributes are the attributes on which we want to collect data, display in Entuity, and report. These attributes define data that can be polled.

• They are held directly against a type. This implies history data for that attribute is not required.
• They are held against a stream within a type. This is for time-series data.

Defining index attributes

Index attributes are populated during the discovery process, through calling the association discoverable (which in turn calls their collector). Index attributes are the key fields of a type's attributes.

For example, it may be the OID that identifies the discovered instance of an object, e.g. of a particular flash card.

Defining association attributes

Association attributes are used to make associations between types. These are the attributes of the association, and do not 'belong' to a type.

Associations involve primary and secondary types, and the association should reflect the relationship between the two types. For example, device to ports is a 'one to many' relationship, where device is the primary type and port is the secondary type. In StormWorks, this association requires two attributes:

• ports holds a list of ports. This is all the ports on the current device. It is used when making the association from the primary type device to port. It is the primary attribute.
• device is a supplied attribute derived from the device type (see sw_device.cfg). It contains all the attribute information relating to the current device. It is used when making the association from the secondary type port to device. It is the secondary attribute.

StormWorks streams:

Streams are the main data gathering mechanism of the StormWorks architecture. Streams define properties of the polling process. A stream can only be connected to one data type (although through type inheritance it can appear otherwise), but a type can have one or more streams connected to it.

Streams consist of a number of attributes, amongst which is a filter that specifies which classes of infrastructure objects are to be monitored. This is an example stream definition:

[Stream rnewsPort]
KeepTime=8h
ObtainRate=3min
StaleAgeOut=2d
Attributes=ifNUcastPkts
ConnectTo=port
Backup=true
Description=RNEWS port data
Filter=eq(obj://ref/device/devType, 168)
ClientData=showInParent=0\ndisplayName=Traffic\nTimestamp.isHidden=1\niconImageURL=http://%host%/EOS/linkstat.gif
EventStreamClientData=isHidden=1

Where:

Instantiating streams

For each discovered object that matches the filter, StormWorks creates a stream collector which collects time series sample data for that object. Stream collectors that are instantiated from the same stream specification are stored in the same table, which has a name in the form of:

dss_*

Where * is the stream name. These tables are created dynamically at startup time. Individual rows are differentiated by their stream collector IDs (column name dsStreamInstId). These in turn reference a row in the dsStreamInst table.

The dsStreamInst table contains the data that connects a stream collector to:

• a unique infrastructure object.
• a stream.
• various data relating to the last time that samples from this stream were collected and written to the database.

To verify that the intended data is being collected by the stream, Entuity compares the summarized samples captured between two contiguous poll periods:

• when the values are different, a new row will be written to the corresponding stream collector table and columns updated in the dsStreamInst table, to reflect the time at which this operation took place.
• when the values are the same, then just the last update time will be changed. This is to avoid storing redundant data.

It is not sufficient to check the table corresponding to stream collectors arising from a stream alone. In order to ascertain if a new sample has been written or only the time stamp relating to the stream collector has been updated, you must check the dsStreamInst table as well.