School Work

Evaluating Government Communication Activity. Standards and Guidance

Categories
Published
of 27
All materials on our website are shared by users. If you have any questions about copyright issues, please report us to resolve them. We are always happy to assist you.
Share
Description
Evaluating Government Communication Activity Standards and Guidance Contents Introduction 3 PROOF: five guiding principles for evaluation 4 The big IDIA: the four-stage evaluation process 5 Stage 1: Identify
Transcript
Evaluating Government Communication Activity Standards and Guidance Contents Introduction 3 PROOF: five guiding principles for evaluation 4 The big IDIA: the four-stage evaluation process 5 Stage 1: Identify 6 Stage 2: Develop 8 Stage 3: Implement 17 Stage 4: Analyse and report 23 Conclusions 27 Appendix: Recommended metrics 28 2 Introduction As government communicators, we re all aware of the need to make every piece of work that we produce as effective and efficient as possible. To do this, we need to understand what s already working well and where there s room for improvement. This in turn requires us to evaluate our work and apply the learning from this evaluation to future activity. We re also increasingly required to demonstrate how we re applying evaluation in our day-to-day jobs, through the plans and reports that we submit to the Efficiency and Reform Group (ERG) for activities with a spend of 100,000 or more and the information that we supply to feed into the annual communication plan. The revised Government Communication Network (GCN) Core Skills for Government Communicators will set out clear evaluation standards that we should all follow, based on our grade and the discipline that we work in. To help us evaluate our activity effectively, in line with expected standards, we need clarity on what good evaluation practice looks like. This guide sets out an approach to evaluation that should be followed as a minimum for all government communication activity, regardless of size, discipline or budget. This approach is pragmatic and focuses on helping you to produce the best possible evaluation given the scope of your activity and the time, resource and budget that you have available for evaluation. By following the guide, you can be confident that you will produce an evaluation that meets the required standards for your activity and for your role. If you re new to evaluation, use the guide to help you get started, recognising that it will take time to build up your approach and gather what s working. Remember, a partial evaluation is almost always better than no evaluation at all. If you re already evaluating your activity effectively, ensure that you re following the standards required of you, making modifications as necessary. Think about how you can share what you ve learned with others to help them evaluate more effectively. By evaluating the activity that we carry out, we will be able to improve the effectiveness and efficiency of our work over time in the future. We will also be able to demonstrate the contribution that well-planned and executed communication activity makes to government overall and hence justify further investment in our work. 3 PROOF: five guiding principles for evaluation Whatever the size or scope of your communication activity, following these guiding principles will help to ensure that your evaluation is as effective as possible. P Pragmatic best available within budget, not best ever A pre-planned, but partial, evaluation is better than no evaluation at all. Be transparent: acknowledge the gaps in your evaluation and the implications of these gaps. Even if you are not able to fully quantify the effect of your communication activity, you will still be able to draw valuable learning from the evidence that you have obtained. R Realistic prove what you can, acknowledge what you can t Don t worry if you can only collect a small amount of data in the short term. By establishing a robust evaluation framework that is linked to a clear set of communication objectives, you will be able to interpret and analyse whatever information you gather. Over time, you will be able to build on this knowledge, increasing the amount of data that you collect from each subsequent activity. O Objective approach your evaluation with an open mind Be honest and constructive about what was achieved, so that we can all learn for the future. Learn from your successes and from things that didn t work as well as you d hoped. Use this to refine your future strategy. O Open record and share as much as possible Share your learning as widely as possible so that colleagues can also benefit from your experience. Work with GCN to develop a detailed case study. F Fully integrated integrate evaluation into activity planning and delivery Plan ahead. Start thinking about how to evaluate your activity as soon as possible, ideally well before it begins. This will help you to put the right mechanisms in place for subsequent measurement and data collection. Retrospective evaluation is often less effective because the right data may not have been collected or objectives may not be measurable. 4 The big IDIA: the four-stage evaluation process You can evaluate all kinds of communication activity, including press and media management, marketing and internal communication. Follow the four-stage process set out in Figure 1 below and you will carry out an effective evaluation in line with the five guiding principles for evaluation. Figure 1: The four-stage evaluation process Identify the scope of your project Develop your evaluation plan Implement source data to measure performance Analyse and report performance against the plan 5 Stage 1: Identify the scope of your project Checklist Task: to define what you need to evaluate by asking: ç What activity am I evaluating? ç What do I already know? ç What is my evaluation expected to achieve? Output: summary of your proposed evaluation approach What activity are you evaluating? Begin by establishing exactly what you are going to evaluate. What activities do you need to include? Are these activities part of a wider communication strategy? Identify the time period over which you will evaluate the activity. For a one-off piece of activity with clear start and end dates, this is generally straightforward. For ongoing activity, you need to identify and agree appropriate time periods for evaluation, i.e. how often you will assess performance against objectives. Examples of types of activity Press Marketing Internal communication For ongoing reputation management work, it s often most effective to track the effect of your work over time. Identify the key messages that you want to track and monitor coverage on a regular basis, providing monthly or quarterly updates rather than reporting on individual activities. If the activity that you are evaluating runs across a range of channels (which might include, for example, paid-for activity, partnerships, leaflets, a website or social media), check that you ve included all of them in your plan. If you are evaluating the effect of a change management programme, ensure that you include all the elements of the programme in your evaluation plan (e.g. briefings to senior staff, communication via and the intranet, events, training etc). 6 Stage 1: Identify What do you already know? Review similar previous activity that your team or others have run to see what you can learn. This will give you a benchmark to measure performance against and may help to refine your objectives and your approach to the activity that you are about to run. $ Useful tip gathering evidence for ERG exemption requests If your activity is subject to ERG approval, use your previous evaluation results as evidence for why you believe it will meet its objectives in sections 3 and 10 of the exemption request form. What is your evaluation expected to achieve? Ask the following questions to help identify what is expected from your evaluation: What are the key questions that your evaluation report needs to answer? What level of detail is expected from your report and what format does it need to be in? Is any budget available for your evaluation? Who will do the work? This might include you and others on your team, research, analysis and evaluation specialists from within your hub or the Shared Communications Service, an external agency or a combination of all of these. $ Useful tip ERG evaluation standards If your activity is subject to ERG approval, there is a standard format that you must use for your evaluation report. A report template can be found here. Ensure that your evaluation plans are designed with this in mind Summarising your approach to evaluation Before beginning work, set out your proposed approach to evaluation, basing this on the information that you have gathered so far. Depending on the scope of the activity that you re evaluating and the person for whom you are carrying it out, this may be a detailed proposal to stakeholders in your department, an to your manager or a note to yourself. Whatever format you use, you can find a useful template here that sets out the questions you should consider. $ Useful tip managing expectations Sometimes, there may be a gap between the time, resource and budget that you have available and others expectations of what your evaluation can achieve. Discuss whether it s better to increase available resource, budget or time or to lower expectations. Decide this early in the process. Stage 1: Identify 7 Stage 2: Develop your evaluation plan Checklist Task: to define how you ll measure success by: ç Setting out activity objectives ç Defining target audiences ç Mapping out how your activity will work ç Setting performance metrics ç Agreeing metrics and targets Output: draft evaluation plan Setting out activity objectives Clear and measurable communication objectives are the cornerstone of any evaluation plan. They set out what your activity aims to achieve and the overall goals against which you should judge success. Your objectives should already be identified in the communication strategy for the activity you are evaluating. It may be useful to summarise them as shown in Figure 2, clearly demonstrating how each sub-objective links to the overall communication and departmental objective. Start by identifying the overall departmental objective and the issue it is designed to address. Your activity will be part of a set of interventions which link back to this objective. Next, identify the communication objective. This is the overall role that communication is expected to play in achieving the policy objective. Different channels or activities may have distinct roles to play in achieving the overall communication objective. Each of these should be set out as a separate communication sub-objective. You may only be evaluating the effectiveness of one communication sub-objective, but it is important to understand how this is expected to contribute towards the overall communication objective. Ensure that all communication objectives are clear and set out what each activity was put in place to achieve, together with a measure of success. Always consider whether or not you will be able to prove a communication objective has been met. If not, it will need to be revised. You may find it useful to map out your objectives as shown in Figure 2. 8 Stage 2: Develop Figure 2: Hierarchy of objectives Departmental objective Put in place to address a specific issue Includes: policy development, policy delivery, reputation management Communication objective Role that communication will contribute to achieving departmental objective Communication sub-objectives Role that individual activities or channels will play in meeting the overall communication objective Sample objectives might include: Press Departmental objective: To ensure that compliance with a new tax regulation is above 80%. Communication objective: To ensure that the majority of the general public understands the reasons why complying with the regulation benefits the economy. Press-specific sub-objective: To ensure that the public is given a fair and balanced view of the policy, via the media. Marketing Departmental objective: To get 10,000 more people working as community service volunteers in your area. Communication objective: To get 40,000 people in the area to register as potential volunteers on your community website. Sub-objective 1: To increase the proportion of the public who recognise the value of volunteering from 20% to 40%. Sub-objective 2: To get 80,000 people to visit the website and find out more about how to volunteer. Sub-objective 3: To secure 40,000 incremental registrations. Internal communication Departmental objective: To ensure that unauthorised staff absences fall by 50%. Communication objective: To ensure that all staff are able to follow the correct processes for reporting absences from work. Sub-objective 1: To ensure that all staff recognise that there is a policy for reporting absences from work and that they must follow this. Sub-objective 2: To ensure that all staff understand how to access the guidance on how to report absences. Stage 2: Develop 9 Defining target audiences All communication activity has an end audience the people at whom it is ultimately targeted. But an activity may also have an intermediary audience a group of people targeted so that they will deliver the message to the end audience on your behalf. Typical intermediaries include the media, stakeholders, such as non-governmental organisations and charities involved in delivering a policy objective, and commercial partners working with you to deliver a piece of marketing activity. If your activity includes an intermediary audience, you should evaluate: how effectively the intermediary was engaged by the activity how effectively the intermediary communicated the message to the end audience. Audiences might include: Press Where you are using a media engagement or PR campaign to try and raise volunteer levels among the public overall: Intermediary (the press): How did the media react to the activity targeted at them? Did they feature your messages, what volume and quality of coverage did you get for the story? End audience (the general public): How many people volunteered as a result? Marketing Where you are trying to engage a partner to run events promoting volunteering on your behalf: Intermediary (partner): How effectively did you engage the partner and how many events did they run as a result? End audience (the general public): How many people volunteered as a result? Internal communication If you are training staff on how to communicate better with members of the public so that customer satisfaction improves: Intermediary (staff): How effectively did you train them? Did they put their skills into practice in their interaction with the public? End audience (the general public): Did they notice that they received a better service from trained staff? Were they more satisfied? $ Useful tip intermediary audiences If your activity includes an intermediary audience, make sure you include performance metrics that enable you to: (1) evaluate the effect of your activity on the intermediary; and (2) evaluate the effect of their activity on your end audience. 10 Stage 2: Develop Mapping out how your activity will work Spend some time thinking about how your activity is expected to achieve its objectives. If it is successful, what messages will the target audience(s) see, what will they think or feel and what will they do? Mapping out the steps to success will help you identify the right performance metrics to evaluate your activity. Draw on any behavioural insight 1 modelling or customer journey work that has already been done. Setting performance metrics Having identified the objectives and target audiences for your activity and mapped out how you expect it to work, you need to build a set of performance metrics. These are the measures you will use to assess the activity s performance against its objectives and to identify which elements of the activity were most and least successful. Make sure that your evaluation plan includes a range of performance metrics from the following five categories: Figure 3: The five types of performance metrics for evaluation 1. Inputs The activity carried out 2. Outputs How many people had the opportunity to see or hear your activity? 3. Out-takes What was its immediate effect on them? 4. Intermediate outcomes Did they do anything as a result of your activity? 5. Outcomes Did you achieve your overall objective? Using these five categories as a guide will help you to pick the right performance metrics for your activity. The categories can be applied equally to the full range of press, marketing and internal communication activity. 1 For more on behavioural insight, see MINDSPACE: Influencing behaviour through public policy (Institute for Government/Cabinet Office Stage 2: Develop 11 Inputs Inputs include details of the actual activity that has been undertaken, including the channels that you used to communicate. The channels to include will have been identified at Stage 1. Examples of input metrics Press Marketing Number of press releases sent out or engagement work carried out Paid-for media plan Website or digital space created Number of partners contacted, types of message shared or requests made of partners (intermediary audience) Internal communication Number of staff events organised Number of briefings or training sessions organised Web content created and put on to the intranet Include the costs of carrying out the activity in your input metrics. Include all external and internal costs and time spent (including staff time). When comparing results for more than one piece of activity, use a consistent methodology to record the costs and time spent against each one. This will be essential for calculating return on marketing investment. 2 $ Useful tip choosing the right performance metrics At this stage, don t worry about whether you can get data for the performance metrics you choose. Pick those that you would need for an effective evaluation. Stage 3 looks at how to secure data and deal with gaps. 2 For more information refer to Evaluating the financial impact of public sector marketing communication: An Introduction to Payback, Return on Marketing Investment (ROMI) and Cost per Result (https://gcn.civilservice.gov.uk/wp-content/uploads/2012/12/intro-to-payback-romi-and-cpr.pdf) 12 Stage 2: Develop Outputs Output metrics measure the number of people who had the opportunity to see or hear your activity, regardless of whether they recall or recognise it. Try to include: Reach the total number of people or organisations in your target audience who were exposed to your activity. Frequency the number of times they saw or heard the activity. Examples of output metrics Press Number of pieces of coverage achieved Frequency of exposure to coverage by end audience Marketing Proportion of the target audience reached by media activity Number of impressions (one page-view) on the web page Number of stakeholders you contacted and number of contacts made Internal communication Number of staff attending events and training sessions Number of impressions on the intranet $ Useful tip activity mapping Use the activity map that you created earlier to help you identify the right out-take, intermediate outcome and outcome performance metrics for your activity. Out-takes Out-take metrics look at the impact that the activity had on your target audience s awareness, understanding and attitude. Think about what you wanted people to recall, think or feel about your activity and include performance metrics that allow you to measure this. $ Useful tip other metrics Performance metrics for out-takes, intermediate outcomes and final outcomes cannot be standardised in the same way as those for inputs and outputs. They will need to be tailored to reflect how you expect your activity to work and what it is trying to achieve. Stage 2: Develop 13 Examples of out-take metrics Recall Think Feel How many people are aware of your activity or the message(s) that it is promoting? How many people understand the key messages that your activity is trying to get across? What effect has your activity had on people s attitudes? Do they intend to behave differently as a result of your communication? Intermediate out
Search
Related Search
We Need Your Support
Thank you for visiting our website and your interest in our free products and services. We are nonprofit website to share and download documents. To the running of this website, we need your help to support us.

Thanks to everyone for your continued support.

No, Thanks