Ch. 10 Testing the Threat Evaluation System
The ThreatEvaluation
system of Archipelago was modeled in the last chapter. We now need to
create the feature files that form test cases and help define the remaining behavior of the system. These feature files
often must reference specific data fields in messages. Therefore, our first effort will be to model the data of the
services we wish to test. In this chapter, we’ll be creating feature files for the EngagementTrackPriorityService
and
DefendedAreaTrackPriorityService.
The remaining tests cases and files are left as an exercise to the reader.
Modeling Detailed Data
The data for the EngagementTrackPriorityService
will be modeled first since that is the first feature file we’ll
create. This service has an input of TrackEngagementStatus
and an output of TrackPriority
. These messages are
straightforward.
TrackEngagementStatus.sd
package com.ngc.seaside.archipelago.engagementplanning.datatype
import com.ngc.seaside.archipelago.tracking.datatype.SystemTrackIdentifier
data TrackEngagementStatus {
SystemTrackIdentifier trackIdentifier
int plannedEngagementCount
float probabilityOfKill
}
TrackPriority.sd
package com.ngc.seaside.archipelago.threat.datatype
import com.ngc.seaside.archipelago.tracking.datatype.SystemTrackIdentifier
data TrackPriority {
SystemTrackIdentifier trackIdentifier
string sourceId {
"description": "Identifies the specific service that generated this message."
}
float priority {
"validation": {
"min": 0.0,
"max": 1.0
}
}
}
We can now create a first iteration of the feature file for the EngagementTrackPriorityService
. Since this service
only has a single scenario (calculateTrackPriority
), only one feature file is created. This file will be placed in
the src/test/gherkin
directory within the package/directory structure of com.ngc.seaside.archipelago.threat
. The
name of the file is EngagementTrackPriorityService.calculateTrackPriority.feature
.
EngagementTrackPriorityService.calculateTrackPriority.feature
Feature: EngagementTrackPriorityService calculateTrackPriority
Background:
Given an absolute tolerance of 0.01
Scenario Outline: Priority tests. The track priority should be calculated as
priority = 1.0 - probabilityOfKill
Given a TrackEngagementStatus object
And the trackId is <id>
And the plannedEngagementCount is <count>
And the probabilityOfKill is <probability>
When the TrackEngagementStatus object is received by the service
Then the service should respond with a TrackPriority object
And the trackId should be <id>
And the sourceId should be "service-com.ngc.seaside.archipelago.threat.EngagementTrackPriorityService"
And the priority should be <priority>
Examples:
| id | count | probability | priority |
| 0 | 10 | 0.5 | 0.5 |
| 1 | 24 | 1.0 | 0.0 |
| 2 | 5737 | 0.0 | 1.0 |
| 3 | 24 | 0.25 | 0.75 |
The first test scenario is a scenario output that sets up some input and defines the expected output. We can add a scenario that tests for invalid input data to define how the service handles invalid inputs:
EngagementTrackPriorityService.calculateTrackPriority.feature
Feature: EngagementTrackPriorityService calculateTrackPriority
Background:
Given an absolute tolerance of 0.01
Scenario Outline: Priority tests.
The track priority should be calculated as
priority = 1.0 - probabilityOfKill
Given a TrackEngagementStatus object
And the trackId is <id>
And the plannedEngagementCount is <count>
And the probabilityOfKill is <probability>
When the TrackEngagementStatus object is received by the service
Then the service should respond with a TrackPriority object
And the trackId should be <id>
And the sourceId should be "service-com.ngc.seaside.archipelago.threat.EngagementTrackPriorityService"
And the priority should be <priority>
Examples:
| id | count | probability | priority |
| 0 | 10 | 0.5 | 0.5 |
| 1 | 24 | 1.0 | 0.0 |
| 2 | 5737 | 0.0 | 1.0 |
| 3 | 24 | 0.25 | 0.75 |
Scenario Outline: Invalid probability
Given a TrackEngagementStatus object
When the TrackEngagementStatus object is received by the service
And the probabilityOfKill is <probability>
Then a fault is raised
Examples:
| probability |
| -1 |
| -0.00001 |
| 1.00001 |
| 1e500 |
| 1e-500 |
This scenario uses the step “Then a fault is raised”. We typically use this phrase to indicate the service should raise an exception and halt processing of the request. The infrastructure will handle reporting the actual fault.
Referencing Data from Other Files
The feature file for the DefendedAreaTrackPriorityService
provides us with a more complex example. Like before, we
need to finish modeling the data for this service. Since this service has two inputs, we need to finish the
ImpactAssessment
and SystemTrack
messages. The ImpactAssessment
message is straightforward:
ImpactAssessment.sd
package com.ngc.seaside.archipelago.defendedarea.datatype
import com.ngc.seaside.archipelago.tracking.datatype.SystemTrackIdentifier
data ImpactAssessment {
SystemTrackIdentifier systemTrackIdentifier
float impactProbability
float impactedAreaValue
}
The SystemTrack
message requires more complex data. We need to model its position and state using more complex data
structures. We’ll use a set of vectors to organize the state. The state data objects will reside in the
com.ngc.seaside.archipelago.common.datatype
package.
Vector3.sd
package com.ngc.seaside.archipelago.common.datatype
data Vector3 {
float x
float y
float z
}
StateVector.sd
package com.ngc.seaside.archipelago.common.datatype
import com.ngc.seaside.archipelago.common.datatype.Vector3
data StateVector {
Vector3 ecefPosition
Vector3 ecefVelocity
Vector3 ecefAcc
}
Finally, we just need to update the SystemTrack
data itself.
SystemTrack.sd
package com.ngc.seaside.archipelago.tracking.datatype
import com.ngc.seaside.archipelago.tracking.datatype.SystemTrackIdentifier
import com.ngc.seaside.archipelago.common.datatype.StateVector
data SystemTrack {
SystemTrackIdentifier identifier
StateVector state
}
The feature file can now reference these files.
The state vector is effectively a 3x3 matrix. Specifying these values directly in the feature file can be messy. As an
alternative, we can store these values in another data file such as a CSV file. In our example, we store state
information for specific tracks in a CSV. Data that is referenced in tests is stored under the directory
src/test/resources/data
. Common data that is reused for multiple tests can be stored directly in this directory. By
convention, data that is referenced only by tests in a single package is placed inside that package. For our tests,
we’ll create a CSV in the directory src/test/resources/data/com/ngc/seaside/archipelago/threat
. The name of this file
doesn’t manner so we’ll use DefendedAreaSystemTrackData.csv
for consistency. The contents of this file might look
something like this:
identifier | state.ecefPosition.x | state.ecefPosition.y | state.ecefPosition.z | tate.ecefVelocity.x | state.ecefVelocity.y | state.ecefVelocity.z | state.ecefAcc.x | state.ecefAcc.y | state.ecefAc |
---|---|---|---|---|---|---|---|---|---|
0 | 22.58422 | 90.86906 | 97.60875 | 21.01349 | 93.01078 | 45.71687 | 45.1883 | 88.56428 | 80.38899523 |
1 | 9999.347 | 123444.5 | 33.71943 | 19.43223 | 13.03871 | 73.78143 | 28.03295 | 70.35227 | 44.99680078 |
2 | 3194442 | -3194442 | 4487380 | 99.12156 | 52.96208 | 84.78288 | 5.413042 | 39.38792 | 81.13938798 |
3 | 3194442 | -3194442 | -4487380 | 93.89776 | 28.81023 | 78.46608 | 38.39385 | 89.20086 | 68.58333996 |
We can use a Background step to reference the CSV file when the tests for the feature file are executed. The
DefendedAreaTrackPriorityService.calculateTrackPriority.feature
would look something like this:
DefendedAreaTrackPriorityService.calculateTrackPriority.feature
Feature: DefendedAreaTrackPriorityService calculateTrackPriority
Background:
Given a csv file "com.ngc.seaside.threateval.DefendedAreaSystemTrackData.csv" of SystemTrack objects
And an absolute tolerance of 0.01
Scenario Outline: Defended Area Priority list.
Compute the priority by multiplying the priority calculated from the state vector/kinematics
to with the impactProbability of the impact assessment.
Given a SystemTrack object
And the identifier is <id>
And the StateVector of the SystemTrack object is retrieved
And an ImpactAssessment object
And the identifier is <id>
And the impactProbability is 1.0
When the SystemTrack object is received by the service
And the ImpactAssessment object is received by the service
Then the service should respond with a TrackPriority object
And the trackId should be <id>
And the priority should be <priority>
And the sourceId should be "service-com.ngc.seaside.threateval.DefendedAreaTrackPriorityService"
Examples:
| id | priority |
| 0 | 0.50 |
| 1 | 0.25 |
| 2 | 1.00 |
| 3 | 0.75 |
The background step references the CSV file. Using the fully qualified name with the package. The remaining tests can
now reference tracks by using only the track ID which is also used to index the CSV data. This makes it easier to
initialize data into structures without having to include all this detail in the tests. The remaining steps for the
test closely follow the steps from the EngagementTrackPriorityService
. Once again, we can expand this test with
additional cases:
DefendedAreaTrackPriorityService.calculateTrackPriority.feature
Feature: DefendedAreaTrackPriorityService calculateTrackPriority
Background:
Given a csv file "com.ngc.seaside.threateval.DefendedAreaSystemTrackData.csv" of SystemTrack objects
And an absolute tolerance of 0.01
Scenario Outline: Defended Area Priority list.
Compute the priority by multiplying the priority calculated from the state vector/kinematics
to with the impactProbability of the impact assessment.
Given a SystemTrack object
And the identifier is <id>
And the StateVector of the SystemTrack object is retrieved
And an ImpactAssessment object
And the identifier is <id>
And the impactProbability is 1.0
When the SystemTrack object is received by the service
And the ImpactAssessment object is received by the service
Then the service should respond with a TrackPriority object
And the trackId should be <id>
And the priority should be <priority>
And the sourceId should be "service-com.ngc.seaside.threateval.DefendedAreaTrackPriorityService"
Examples:
| id | priority |
| 0 | 0.50 |
| 1 | 0.25 |
| 2 | 1.00 |
| 3 | 0.75 |
Scenario Outline: Defended Area Priority list with variable impact probability.
Given a SystemTrack object
And the identifier is <id>
And the StateVector of the SystemTrack object is retrieved
And an ImpactAssessment object
And the identifier is <id>
And the impactProbability is <impactProbability>
When the SystemTrack object is received by the service
And the ImpactAssessment object is received by the service
Then the service should respond with a TrackPriority object
And the trackId should be <id>
And the priority should be <priority>
And the sourceId should be "service-com.ngc.seaside.threateval.DefendedAreaTrackPriorityService"
Examples:
| id | impactProbability | priority |
| 0 | 1.0 | 0.50 |
| 1 | 0.5 | 0.125 |
| 2 | 0.25 | 0.25 |
| 3 | 0.75 | .5625 |
Conclusion
We have only created features files for two services but the remaining feature files are created in the same way. Note
that a feature files for the ThreatEvaluation
system itself can also be created. These feature files are used to test
perform automated tests with all the integrated components together. Data that is needed for the test can be stored in
files external to the feature files and referenced within a test step.