Software verification, testing and maintenance are essential to minimise the risks of using technology. Verification and testing is best conducted well before the system is used for a 'live' electoral event (both on its own and in conjunction with associated hardware and communications). After successful testing, software systems will require appropriate maintenance to ensure they will perform effectively when needed.
The level of importance of the technology will impact on the degree of rigour applied to verifying, testing and maintaining software systems. For a system to be used for a crucial electoral function, such as an electronic voting system, the degree of rigour needed will be high.
While this and the two accompanying sections separate hardware, software and communications into three topics, their operation is often interdependent, and the following verification, testing and maintenance procedures may need to be carried out with all three elements in combination.
Software verification
For a highly important system such as an electronic voting system, it is appropriate to employ an independent testing authority to perform system verification tests. For less important systems, system verification could be conducted in-house.
Software verification tests (otherwise known as qualification tests) could include:
- testing of software to ensure that appropriate standards are met and that the software performs its intended functions, including audits of code (see below)
- ensuring system documentation is adequate and complete
- verifying that systems are capable of performing under expected normal conditions and possible abnormal conditions
- ensuring that security measures are in place and that they conform to appropriate standards
- ensuring that appropriate quality assurance measures are in place
Audits of software code may need to be conducted, particularly where the software is being used for a crucial system. Software audits are generally most effective when carried out by experts who are independent of the authors of the code. Measures included in a software audit could include:
- verifying that the code is logically correct
- ensuring the code is of modular design (that is, that the code is made up of discreet modules that can be separately tested and evaluated)
- verifying there is no 'hidden' code intended to perform unauthorised functions
- checking that the code is straightforward and relatively easy to understand
- ensuring the code is designed for easy testing - that is, that it includes features to allow testing of flow of data within and between modules
- verifying that the code is robust, so that it includes error trapping and error correction features that will allow immediate detection of errors and prevent loss of data through error
- ensuring the code incorporates security features that will prevent unauthorised access and/or detect and control any attempts at unauthorised access
- ensuring that the system is useable without the need for complex or obscure procedures
- ensuring that the software can be easily installed in the live environment
- verifying that the software can be easily maintained, and that errors or defects can be easily identified, corrected and validated after installation
- checking whether the software can be easily modified to add new features
Software testing
After software has been verified, it needs to be thoroughly tested to ensure that every component of the system is operating as it should, and that the system is performing exactly in accordance with the specific local requirements.
For an important system such as an electronic voting system, a structured system testing program can be established to ensure all aspects of a system are tested. Testing measures that could be followed include:
- developing a set of test criteria
- applying functional tests to determine whether the test criteria have been met
- applying qualitative assessments to determine whether the test criteria have been met
- conducting tests in 'laboratory' conditions and conducting tests in a variety of 'real life' conditions
- conducting tests over an extended period of time, to ensure systems can perform consistantly
- conducting 'load tests', simulating as close as possible a variety of 'real life' conditions using or exceeding the amounts of data that could be expected in a real situation
- verifying that 'what goes in' is 'what comes out', by entering known data and checking that the output agrees with the input
Software maintenance
After a software system has been verified, tested and implemented, it must continue to be maintained. Maintenance routines will vary depending on the type and complexity of the technology. Many software systems will come with a maintenance schedule or program recommended by the developer. Maintenance could be provided by the developer as part of the purchase agreement for the technology.
Systems will need to be maintained to ensure that they continue to perform to the level demonstrated during the system testing stage. If systems deteriorate, there is a risk that the systems will not perform to the required standard.
Ongoing monitoring or testing systems may be installed to ensure that maintenance needs are identified and met where necessary. Where systems are in long-term use, a system can be designed to monitor feedback from users and conduct any modifications or maintenance as needed.
Where modifications to software are made as a result of system maintenance or upgrades, it may be necessary to instigate further rounds of system verification and testing to ensure that standards are still met by the modified system.
Reference: Performance and Test Standards for Punchcard, Marksense, and Direct Recording Electronic Voting Systems, [United States] Federal Election Commission, US Government Printing Office, Washington DC, January 1990
