What are event rules and processing stages?

Introduction to event rules

Rule types and supplied rules

To add a rule

To edit or delete a rule

 

 

 

Introduction to event rules:

Rules provide the logic that underpins the actions of the EMS. You can set conditions and tests to determine when a rule should be applied. You can also define one or more actions within a rule, e.g. to change event severity, discard events, run Groovy scripts etc.

 

Processing stages

Entuity applies rules during the processing of events. Processing is divided into two stages:

 

• Pre Storage - before events are saved to the database. Processing a rule before an event is stored in the database means that the rule can selectively discard events so that they are never stored or used to open incidents. This stage contains the following processing stages by default:
  • Traps
  • Initial Filtering
  • Suppressions
  • Flapping
  • N of M
• Post Storage - after events are saved to the database.

Processing stages are containers for rules, allowing you to structure and order your rules. In the same way that you can set conditions and tests against rules that must be met before rules can run, you can set conditions and tests against processing stages before the rules in those stages can run.

You can add your own processing stages, and edit processing stages within the Pre Storage or Post Storage stages. Note, stages are processed from top to bottom, so you should consider the relationship between different rules and stages. For example:

• within the Initial Filtering processing stage, the Filter Port Status Events rule discards trap-based port status events from those ports that you have configured Entuity to ignore.
• within the Flapping processing stage, the 2 unify rules both handle trap-based port events.

It is more efficient to apply the Port Status Filter rule first, because it discards events from identified ports. If this rule were run after the Flapping rules, then the Flapping rules would be processing events that would afterwards be discarded.

 

Enable/disable rules and processing stages

You can also choose to enable or disable rules and processing stages. By default, all system processing stages are enabled, and all supplied rules (except the N of M rules, which are not enabled). The enable state is indicated by its state icon -  tick for enabled,  paused for not.

 

To add a processing stage

You can add processing stages within the Pre Storage stage and Post Storage stage via the Add Processing Stage button under the Rules tab on the Event Administration page:

1. Navigate to the Rules tab of the Event Administration page.
2. On the left, select either Pre Storage or Post Storage, or any of the existing stages within those two sections.
3. Click Add Processing Stage at the bottom of the page, or via the right click.
   
   
   
   This will open the Add Processing Stage window. 
   
   
   
   
   1. Specify a Name and Description for the stage.
   2. Specify the method of Rules Processing, either:
      • process all for Entuity to process all rules in the stage.
      • finish on first match for Entuity to process the first rule that matches the set Condition.
   3. In the Condition dropdown field, specify the condition applied to tests, either:
      • None - Entuity always processes the rule.
      • All tests must succeed - Entuity only processes the rule when all tests are met.
      • At least one test must succeed - Entuity processes the rule when at least one test succeeds.
4. Click OK to save the new processing stage, otherwise click Cancel.

You can drag a processing stage to whichever position that you want, in both Pre Storage and Post Storage.

 

Schedule rules

You can also define a schedule, which is the time period in which the rule can run. By default, supplied rules do not have a schedule and so, if enabled, are always available to run.

 

 

 

Rule types and supplied rules:

Rules are based on a rule type. There are 6 rule types, 5 having special task-oriented algorithms, and 1 generic rule type.

 

 

Rule Type	Supplied Rules
Deduplication	Deduplicate Enterprise Trap Events. SNMP Authentication Failure.
Event Set Detection	-
Flapping Detection	Detect Port Flapping.
Generic	Filter Port Status Event. Unify Port Up Events. Unify Port Down Events. Discard Unknown Trap. Discard Events From Recently Added Devices.
N of M Suppression	N of M for Processor Utilization. N of M for High Port Utilization. N of M for IP SLA Test Failed. N of M for IP SLA Test High Latency. N of M for IP SLA Creation Failure. N of M for IP SLA High ICPIF. N of M for IP SLA Low MoS. N of M for Network Outage. N of M for Device Reachability.
Trap Processing	-

 

When developing rules, consider:

• which rule type is most appropriate to your task.
• when to apply the rule. The supplied rules are all processed during the Pre Storage stage, but you can also develop rules to apply during the Post Storage stage.
• the order in which rules are applied - their order in the tree is the order in which they are applied.
• if the rules should always be available:
  • you can use the schedule feature to only apply a rule for a set period. For example, you may define a rule that instructs EMS to discard all events for a 4-hour period from an area of your network, because that area is unavailable due to planned maintenance.
  • you can enable and disable rules, and enable and disable the processing stages in which they sit. Rules that are in development should always be disabled. Rules that you do not currently require can be disabled rather than deleted.

 

 

Deduplication rule type:

The Deduplication rule type helps reduce event noise when multiple events are raised from the same network object and threaten to flood the EMS. You can define the rule so it raises an event on the first event, but suppresses repeated occurences of the event from that source until a defined period of time has elapsed and/or a releasing event is raised.

The Deduplicate Enterprise Trap Events rule is applied at the initial filtering stage, and is enabled by default. It protects against a flood of enterprise traps from one source generating a flood of enterprise trap events taht would overwhelm your capacity to manage the network.

You can use the Deduplicate Enterprise Trap Events rule with its default settings to work through how EMS applies deduplication rules:

1. Entuity receives an enterprise trap and then creates an Enterprise Trap event.
2. EMS receives the enterprise trap and applies the Deduplicate Enterprise Trap Events rule.
   Because it is the first event from that source, EMS saves the event to the database, raises the event in Entuity, and generates the associated incident.
3. Within 1 second, Entuity receives another enterprise trap from the same source, and it again creates an Enterprise Trap event.
4. EMS receives the Enterprise Trap event and applies the Deduplicate Enterprise Trap Events rule.
   Because it has already raised an event from that source, EMS checks whether the new event is within the specified Release Period (which by default is 2 minutes). The event was raised within 1 second of the first event and therefore is within the Release Period.
   EMS therefore suppreses the event, and it is not written to the database. Therefore there is no record of the event within Entuity.
5. When 2 minutes have elapsed, EMS raises the last suppressed event, if any.

The Deduplication rule type attributes are as follows:

 

 

Field Name	Field Description
Type	rule type (Deduplication).
enabled	tick to enable this rule.
Name	rule name.
Description	rule description.
Condition	condition applied to tests. Select from: None - Entuity always processes the rule. All tests must succeed - Entuity only processes the rule when all tests are met. At least one test must succeed - Entuity processes the rule when at least one test succeeds.
Tests	define the tests which are applied against the event. This field does not appear if None is selected in the Conditions field above.
Event Type	specify the event to which the deduplication rule applies.
Release Period	specify the time period during which repeated events are suppressed.
Release Event	specify the event that ends the deduplication suppression release period.

  

 

Event Set Detection rule type:

When you know that one type of event is likely to be followed by another type of event, you can use the Event Set Detection rule to test for such an occurence. For example, a fan failure event is likely to be followed by a high temperature warning event.

The Event Set Detection rule type attributes are as follows:

 

 

Field Name	Field Description
Type	rule type (Event Set Detection).
enabled	tick to enable this rule.
Name	rule name.
Description	rule description.
Condition	condition applied to tests. Select from: None - Entuity always processes the rule. All tests must succeed - Entuity only processes the rule when all tests are met. At least one test must succeed - Entuity processes the rule when at least one test succeeds.
Tests	define the tests which are applied against the event. This field does not appear if None is selected in the Conditions field above.
Detection	specify the events to occur within a specified time period that will constitute the event set.
Action Steps	specify the action steps that will ensue as a result of the event set detection.

 

 

Flapping Detection rule type:

Flapping Detection rules detect a sequence of events, usually corresponding to a state oscillating between two values, e.g. the operational state of a physical port going up and down more than a defined number of times within a set period.

When Entuity detects the flapping condition:

1. it suppresses the original events and raises a new event to indicate the flapping state.
2. the rule then continues to monitor the incoming events to determine when the flapping has ended.
3. when the flapping has ended, the most recently-received incoming event is released and all suppression is removed.

The Flapping Detection rule type attributes are as follows:

 

 

Field Name	Field Description
Type	rule type (Flapping Detection).
enabled	tick to enable this rule.
Name	rule name.
Description	rule description.
Condition	condition applied to tests. Select from: None - Entuity always processes the rule. All tests must succeed - Entuity only processes the rule when all tests are met. At least one test must succeed - Entuity processes the rule when at least one test succeeds.
Tests	define the tests which are applied against the event. This field does not appear if None is selected in the Conditions field above.
Detection	specify the events, the number thereof, and the period in which they must be raised for the source object to be considered as flapping.
Derived Event	specify the event that is raised to indicate the flapping state.
Release Event	Entuity v19.0 P03 upwards. Specify the event that is processed after the Flapping Rule stops suppressing the related events. For example: a port starts flapping by switching between 'Port Up' and 'Port Down'. The issue is resolved and the last event is a 'Port Up'. in this case, the Flapping Detection rule catches the first 'Port Up' and 'Port Down' events, and provides a 'Port Flapping' event. The rule then starts suppressing further 'Port Up' and 'Port Down' events. If there are no more of these events in a set time, the rule stops suppressing them. the last 'Port Up' event is released, as well as a 'Port Flapping Finish' event.

  

 

Generic rule type:

You should use the Generic rule type when developing rules for tasks that do not require task-specific algorithms.

The Generic rule type attributes are as follows:

 

 

Field Name	Field Description
Type	rule type (Generic).
enabled	tick to enable this rule.
Name	rule name.
Description	rule description.
Condition	condition applied to tests. Select from: None - Entuity always processes the rule. All tests must succeed - Entuity only processes the rule when all tests are met. At least one test must succeed - Entuity processes the rule when at least one test succeeds.
Tests	define the tests which are applied against the event. This field does not appear if None is selected in the Conditions field above.
Action Steps	specify the action steps that will ensue as a result of this generic rule.

 

One type of Generic rule is the Discard Events From Recently Added Devices rule type. Please see below for further information on this specific rule type.

From Entuity v21.0 P01 upwards, you can create a Generic-type event rule that leverages custom webhook payload data to raise custom webhook events on specified devices. Please see this section for further help and information on attaching webhook payload information to generated events.

 

 

N of M Suppression rule type:

Use N of M rules to reduce event noise. For example, without N of M rules, a busy device might not respond to pings because it is prioritizing its core functionality, and Entuity could then raise multiple spurious events indicating that the device is down, followed by the next successful poll raising a clearing event.

With N of M rules, Entuity does not raise an event for each time a ping failure occurs. Instead, it calculates the percentage of time that ping failures indicates the device was unreachable in a rolling window, e.g. it only raises the event if the associated threshold is exceeded by a defined percentage of the set time period.

After a threshold is crossed and an event is raised, one successful poll will cause Entuity to raise a clearing event. This will then reset the N of M count to zero.

When defining N of M rules, you should consider how often Entuity polls for the metric on the object. For example, Entuity polls for CPU Utilization every 5 minutes. If you want to raise an event when utilization is above the threshold for 15 minutes, then in the Edit Rule window, you would set the Release Condition fields to 100% of 15 minutes.

The N of M Suppression rule type attributes are as follows:

 

 

Field Name	Field Description
Type	rule type (N of M Suppression).
enabled	tick to enable this rule.
Name	rule name.
Description	rule description.
Condition	condition applied to tests. Select from: None - Entuity always processes the rule. All tests must succeed - Entuity only processes the rule when all tests are met. At least one test must succeed - Entuity processes the rule when at least one test succeeds.
Tests	define the tests which are applied against the event. This field does not appear if None is selected in the Conditions field above.
Release Condition	specify the % of a specified time period over which the associated threshold is to be exceeded for the event to be raised.
Opened By	specify the event that opens the N of M rule.
Closed By	specify the event that clears the N of M rule.

  

 

Trap Processing rule type:

In Entuity, you can add Trap Processing rules. Traps is the first substage of the Pre Storage stage. Rules in this stage are therefore the first rules actioned. The Trap Processing rule type is also used when you load a MIB file. Entuity will parse the file, and when instructed, Entuity will interpret the trap definitions and generate a new processing rule for each trap. You can then amend the rule if required.

When you create a custom event, you can establish a varbind value as a contributor to the source identification of the trap. To prevent varbind values being automatically included as part of the source identifier, Extra Identifier is not populated by default. However, when you click Modify, Entuity displays the available varbind values.

The Trap Processing rule type attributes are as follows:

 

 

Field Name	Field Description
Type	rule type (Trap Processing).
enabled	tick to enable this rule.
Name	rule name.
Description	rule description.
Condition	condition applied to tests. Select from: None - Entuity always processes the rule. All tests must succeed - Entuity only processes the rule when all tests are met. At least one test must succeed - Entuity processes the rule when at least one test succeeds.
Tests	define the tests which are applied against the event. This field does not appear if None is selected in the Conditions field above.
Set Event Type	specify the event (or create a new event via the New button) to which the rule applies.
Set Details	specify the details of the event.
Extra Identifier	click Modify to display and select from the available varbind values.
Action Steps	specify the action steps that will ensue as a result of this Trap Processing rule.

EMS includes a Discard Unknown Trap rule. As part of the Initial Filtering stage, when activated it is only applied after other trap processing rules are applied. It therefore only discards alerts from traps without processing rules.

You can create new stages and assign trap processing rules to those stages. However, you should not delete the Traps stage because it is used to hold rules generated automatically when loading trap definitions. If you delete the stage, Entuity will recreate the stage the next time it loads MIBs and traps definitions and has to automatically generate rules.

Multiple traps raising the same event type:

You can map multiple trap OIDs to the same Entuity event type. You might want to do this
when:

• some devices are using a MIB with obsolete trap definitions and other devices are using
  the new trap definition.
• some devices are running SNMPv1 and others SNMPv2, which may mean using different
  traps to return the same information.
• different device vendors have different MIBs, but for the Entuity administrator’s purpose
  they are returning the same information.

The following example uses the BGP4 MIB file, which includes 4 traps:

• Two obsolete, bgpEstablished and bgpBackwardTransition
• Two replacements for the obsolete traps, bgpEstablishedNotification and bgpBackwardTransNotification.

The initial loading of the MIB created rules and custom events for each trap. The intention is to change the rules associated with the obsolete traps so that they call the custom event type associated with the replacement traps.

To amend the trap processing rule for bgpEstablished:

1. From the Main Menu, click Administration > Events Administration.
2. Click the Rules tab, and then from the tree click Pre Storage > Traps.
3. Highlight the Process Trap: bgpEstablished rule and click Edit.
4. From Set Event Type, select the bgpEstablishedNotification event type.
5. Click OK.

You can follow the same process to amend the rule for the bgpBackwardTransition. When the rules are adjusted, the custom events associated with the obsolete trap definitions are now unused. You can delete them through the Events page.

An alternative approach to handling multiple trap OIDs that return the same type of data would be at the incident level. You can allow these traps to raise their own custom event type but associate these event types to the same incident type. The details of the incident would identify the originating trap OID.

Using varbind name values to set event type:

You can use varbind name values to determine the event type to raise. For example the value
may determine the severity level of the raised event or signify whether an object is up or
down.

The initial loading of a MIB creates rules and custom events for each trap. This example:

• loads the RPKI-Router-MIB which then generates a custom event and rule for the pkiRtrCacheServerConnectionStateChange trap.
• rpkiRtrCacheServerConnectionStatus has a value of 1 when up and 2 when down.
• amends and renames the generated rule to signify a down connection state. The rule tests the varbind value, using the Trap Varbind Test and calls a specific custom event type.
• creates a new rule and custom event to signify when the trap reports an up state.
• creates an incident that is raised when the connection is in a down state and closed when it is in an Up state.

To amend the trap processing rule for pkiRtrCacheServerConnectionStateChange:

1. From the Main Menu, click Administration > Events Administration.
2. Click then Trap tab and then Manage MIBs.
   
   
   
   
3. Load the RPKI-Router-MIB and ensure Create Rules and Events from Trap Definitions is selected.
   
   
   
   
4. Click the Rules tab and then from the tree click Pre Storage > Traps.
5. Highlight the Process Trap: pkiRtrCacheServerConnectionStateChange rule and click Edit.
   
   
   
   
6. From the Conditions section, click Add to add a test.
   
   
   
   
7. Set the Type to Trap Varbind Test, from Varbind select pkiRtrCacheServerConnectionStatus, set Operation to equals and value to 2.
   
   
   
   
8. From Set Event Type click New.
   
   
   
   
9. In the Add Custom Event window, define the new event, add the clearing event and add an incident definition. The clearing event will be called by a new processing rule that will test the trap varbind value from the up value of 1.
   
   
   
   
10. Click OK.
11. Define a new trap processing rule. Highlight Traps and from the context menu click Add Rule.
    
    
    
    
12. Specify the Type as Type Processing, and from Apply to Trap select pkiRtrCacheServerConnectionStateChange.
    
    
    
    
13. From the Conditions section click Add to add a test. Set the Type to Trap Varbind Test, from Varbind select pkiRtrCacheServerConnectionStatus, set Operation to equals and value to 1.
    
    
    
    
14. Set Event Type to the previously defined Up event.
15. Click OK.
16. Save and deploy the event project for your changes to take effect.

 

 

Discard Events From Recently Added Devices rule type:

By default, this rule is not enabled. When this rule is enabled, all events for newly added devices are discarded (ignored) for 24 hours. This is primarily to prevent situations whereby newly added devices produce too many discovery events.

Note, from Entuity v19.0 P03 upwards, this event ignores Cisco APIC events (thus allowing these events during and after the first 24 hours).

This can be enabled from the Event Administration page:

1. On the Event Administration page, click the Rules tab.
2. On the left, open the Pre Storage processing stage.
3. Open the Initial Filtering step.
4. Select Discard Events From Recently Added Devices, then click Edit to open the Edit Rule dialog.
5. Tick the enabled box in the top right, and then click OK.
   
   

 

The Discard Events From Recently Added Devices rule type attributes are as follows:

 

Field Name	Field Description
Type	rule type (Generic).
enabled	tick to enable this rule.
Name	rule name (Discard Events From Recently Added Devices).
Description	rule description (by default, this is set to 'Discards events from devices which haven't been managed for full 24 hours').
Condition	condition applied to tests. Select from: None - Entuity always processes the rule. All tests must succeed - Entuity only processes the rule when all tests are met. At least one test must succeed - Entuity processes the rule when at least one test succeeds.
Tests	define the tests which are applied against the event. This field does not appear if None is selected in the Conditions field above. By default, the test is set to: 'Groovy Script: // get time now...'
Action Steps	specify the action steps that will ensue as a result of this Discard Events From Recently Added Devices rule. By default, the 'Discard Event' action step is specified.

 

 

To add a rule:

You can add a rule to any processing stage:

1. Navigate to the Event Administration page. Click the Rules tab.
2. In the tree on the left, select the processing stage to which you wish to add the rule (either Pre Storage or Post Storage).
   
   
   
   
3. Click Add Rule at the bottom of the window or via right click.
   
   
   
   
4. This will open the Add Rule window, which has two tabs: General and Schedule.
   
   
   
   
   • Under the General tab, you can specify the parameters of the new rule. The parameters and fields available will depend on the Type of rule you select (see above for further information on types of rule and their attibutes).
   • Under the Schedule tab, you can specify the schedule under which the rule will run.
5. Click OK to save the rule, otherwise click Cancel.

 

 

To edit or delete a rule:

1. Navigate to the Event Administration page. Click the Rules tab.
2. In the tree on the left, select the processing stage within which is the rule you wish to edit or delete.
3. Select the rule you wish to edit, and click Edit or Delete.
   
   
   
   
   • if editing the rule, make your desired changes and click OK to save, otherwise click Cancel.
   • if deleting the rule, a confirmation window will appear. Click OK to delete, otherwise click Cancel.