TrafficInfraTech Magazine Linking People Places & Progress
In Step 5, the decision is made on the best evaluation approach. If an ‘experimental’ design is selected, a ‘baseline’ or ‘control’ is defined, against which the C-ITS impacts can be compared; this should ensure that any changes measured can be attributed to the C-ITS rather than to other factors such as changes in the economy or improvements in vehicle safety. Alternatively, a theory-based approach may be used, in which the effects of the C-ITS are observed and measured without the experimental control. Compared with an experimental approach, this will involve a longer evaluation period and much greater stakeholder involvement. A combination of both approaches may also be used.
Finally, Step 6 involves refining the evaluation approach. For example, by refining the indicators to ensure that they are compatible with the evaluation approach selected. The Connected Corridor project has yet to define the evaluation in detail.
The methodology to identify indicators and determine suitable measurement metrics involves three phases: preparation; data collection and quality checking; and analysis and reporting.
Phase II – Data Collection and Quality Checking
At this stage in the development and deployment of C-ITS, the most extensive method of evaluation currently in use is the ‘Field Operational Test’ (FOT). This involves a rigorous evaluation of the services in their intended environment by their intended users, on a scale and with a duration that is sufficient for statistically robust conclusions to be drawn.
The data collection methodology is usually developed in detail as part of the C-ITS trials design. The data required for impact assessment will typically include quantitative measurements, quantitative assessments (such as rating scales) and qualitative data (such as feedback from stakeholders and reports from focus groups of users). A data collection plan will define key elements such as naming, labelling and storage of data, and the ethical and legal requirements.
The data collection may also include metadata on the trial such as the technical performance of the C-ITS (like incidence of faults and system availability), and monitoring of service delivery and vehicle availability.
Quality checking is planned as part of the data collection plan and begins as soon as data collection has begun. Dynamic ‘sense checks’ are carried out on the data while it is being collected, to ensure that the data being recorded is as expected. More comprehensive quality checks on the complete data are then carried out before the analysis begins (such as checks for gaps in data, usage of the correct units and the values being within feasible ranges). Procedures for ‘cleaning’ the data prior to analysis are then designed on the basis of these quality checks.
Phase III – Analysis and Reporting
Using established data analysis methodology, the following types of analysis are carried out:
• Aggregating raw data over appropriate time scale
• Combining aggregated data into the previously identified indicators and reporting confidence limits according to standard statistical practice
• Traffic modelling and judgement to determine input parameters for estimation and modelling of outcomes relative to the baseline
• Testing of hypotheses relating to the intervention logic
• Reporting on outcomes such as safety, traffic flow and environmental performance
• Estimating distributional effects of benefits and outcomes for different Stakeholders including industry, road operators and road users
• Evaluating risks and sensitivities.
Reporting of C-ITS field trials covers three core areas: Technical performance of the technologies; Description of data obtained and quality achieved; and Key outcomes in terms of system objectives such as safety, traffic flow, environmental improvements etc.
Other areas may also need to be reported on, such as recommendations for future roll out, bundling of services, other services to be considered and the evaluation approach used.
(Acknowledgements: This article draws on several projects, including work for Highways England on the UK Connected Corridor Project and technical paper ITS-EU-TP0239 presented at the ITS World congress in Australia during 2016 . © Transport Research Laboratory, 2017).
