‘I love this site! …This is perhaps the finest example of a government website in the history of the Internet.’
Member of the public
‘What idiot thought a single web site was a good idea? The separate ones are bad enough.’
Civil servant
These are genuine comments at the extreme ends of the feedback we received for the Inside government beta. Over the six weeks of the beta we received a lot more in between, and we were grateful for every last item of praise and criticism.
This post is about how we captured that feedback, what we learned from it and what we are going to do as a result.
Recap
We released Inside government as a beta on February 28th 2012, following 24 weeks of iterative development. The idea being to test – on a limited scale – the site that may come to accomodate all departmental corporate websites. The beta ran for six weeks in the public domain and involved 10 pilot departments (BIS, Cabinet Office, DCLG, Defra, DFID, DH, FCO, HMRC, MOD and MOJ).
Inside government ‘department’ page on an Ipad
By releasing Inside government we were testing a proposition (‘all of what government is doing and why in one place’), and two supporting products (a frontend website and a content management system). With this in mind, we wanted to ensure that we captured feedback from the public and from colleagues across government.
We wanted to know if – having used the site – people thought it was a good idea and whether it should be developed further. And besides testing the viability, we hoped that feedback would prove a rich source of ideas and steering on what was important for us to concentrate on in subsequent iterations.
Open feedback channels
The most obvious and noisy sources of feedback were, of course, ‘open channels’ which included Twitter, email, our service desk, our GetSatisfaction forum, and some people even had our phone numbers.
The value of these routes lies in their diversity and because they provided the opportunity for the project team to engage directly with the end users. They were particularly popular with public users, but we were pleased to see that civil servants also took up these opportunities to put forward their views and get into discussion with others.
Here’s a cross-sample of the comments that came in:
‘Love the product! The concept of GOV.UK is the right way to go.’
Member of the public
‘The design feels far more people-friendly, the language (of the site architecture as well as the content) feels like it has the right balance between being friendly and expert’
Member of public
‘I have looked through the test website and I think it is an excellent idea. I like the news reports and I like the idea of one website for all departments.’
Civil servant
‘I think the concept of gov.uk is sensible but there’s a way to go yet to get this website working well.’
Civil servant
‘Looks very basic and home made’
Civil servant
‘I don’t know why you’d need a new website. Why not just add a section on how government works to direct.gov (sic)’
Civil servant
Our open channels were a particularly rich source of product enhancement ideas. Examples included: adding a section on the mechanics of how government works, incorporating section-specific searches, and suggestions of what data to include in feeds.
While there was lots coming through the open channels, much of it was granular and from an engaged audience predisposed to take an interest and have an opinion. Highly valued stuff but only part of the picture, and so our evaluation squeezed three further tests into the time available.
User interviews
Inside government should be open and accessible to everyone and we expect the core users to be people with a professional or deep thematic interest in the policies and workings of government.
To get qualitative insights into how these users used and rated gov.uk/government, we arranged 12 face-to-face interviews (with professionals from academia, charities, media and the private sector). Each participant knew that they would be asked about their internet usage but they were not aware which site(s) they would be discussing.
Interviews were conducted on a one-to-one basis by a trained facilitator and began with a discussion of how the individual used central government’s current websites. It was evident that although a user gets to know their way around a particular section of a site, when they move off that section or onto another department’s site the inconsistencies present real frustrations.
‘It is tricky because currently you have to go to each department individually and its only done with civil servants in mind.’
Member of public
Department pages on Inside government beta (top) vs current departmental sites
The facilitator then pulled up Inside government. The participants began with a cursory browse and were then set tasks designed to move them through the site’s content and functionality. As they carried out the tasks, the facilitator asked them to comment on how well they felt the site was performing.
We learned that these professionals found the user interface clear and intuitive. There were common criticisms; a number of which were arguably down to the ‘rough’ nature of the beta, but important to heed as the site develops and takes on more content. These included issues with long lists, the visibility of ‘related content’ in columns, and the depth of content (some participants worrying that it was being ‘dumbed down’). Perhaps the most interesting finding being that participants wanted more of a departmental lead in the navigation, rather than the thematic approach that we were trialling.
Fundamentally, when asked, the participants were able to articulate who is responsible for the site, what its purpose is and who the users are likely to be. In this sense, the proposition of Inside government was clear and, they said, with some development this was a product that they welcomed because it would make their work easier.
Usability testing
With insights from the core users bagged, we were hankering after feedback from general public users (who we expect to be infrequent visitors to corporate sections of government websites and who have basic or no prior knowledge of the machinery of government). What would they think of the proposition and would they be able to find what they wanted easily?
Using the GDS’ summative test methodology, we put Inside government to the scrutiny of a panel of 383 users. The respondents were a mix of ages and gender, they were geographically-spread throughout the UK, participated in their home environment and there was no moderator or facilitator.
The tests use a range of measures to assess performance (such as journey mapping and completion times). Participants were prompted (by software) to find specific information (facts and figures) on the site through five tasks (which were tracked) and at the end of the tasks they were asked a series of questions about their experience.
We were pleased to see that on all but one of the tasks the successful completion rate was above 60%, and more than 50% of participants said they found it very or quite easy to complete the tasks. That was a positive overall trend on an unfamiliar and content-heavy site.
Below the overall trend, a few issues of concern were flagged up. Just over a third of participants found it difficult to complete the tasks, especially when it came to finding a specific piece of information on a page. Most participants described the content as ‘straightforward’, ‘to the point’ and ‘up to date’, while some said it was ‘longwinded’ and ‘complicated’. Over 50% of participants said they thought Inside government contained ‘the right amount of information’ but 39% thought there was too much.
From these findings, we were satisfied that in the beta we had build a site usable by the general public, but there is clearly a great deal of work still to be done to produce an excellent ‘product for all’.
Comparative CMS Tests
We wanted to tackle one of the common complaints government digital teams have about their digital operations: cumbersome, convoluted and costly content management systems. So we decided to try building a CMS from scratch that would be stable, cheap to run, include only the functionality required to manage the Inside government site, and be easy to use by even the most inexperienced government staff.
A view of the Inside government CMS
To test what we produced, we ran structured testing in departments, using their standard IT and setting Gov.uk in head-to-head tasks with their existing CMS. The aim was to assess the performance and usability of the beta publishing application, and understand where government publishers found it to be better, lesser or equivalent quality compared with what they were currently using. And, we wanted to get their ideas for further iteration.
We ran tests in six departments against five incumbent products with seven participants, who were from digital publishing teams. The participants were a mix of those with previous or no experience of using the Gov.uk publishing app, and each was given up to five tasks representing common ‘everyday’ CMS functions.
In all but one task, users completed tasks faster or in comparable times. The reason for the slower task was down to the users lack of experience with markdown formatting method (and even then it was only in one specific area – bulleted lists).
Taking into consideration all the tasks they were asked to complete during the tests, the users made positive comments about their overall experience of the Gov.uk publishing app. For most it was easier to use, better laid out and faster than their current content management system because it was customised to their specific professional needs.
‘The CMS is a dream – especially compared with the current [product name removed] system. It’s fast, user friendly and intuitive. It’s also easier to use visually.’
Civil servant
What we learned
We didn’t get to do as much testing as we would have liked. Time was against us. But from what we did do, we learned a lot.
Problems include:
- Big improvements in findability need to be made if Inside government is to be able to cope with the weight of content from all departments and agencies
- We need to adjust to make departments more prominent in the navigation and on pages
- In trying to improve the readability and comprehension of the corporate content, Inside government needs to be careful not to over-simplify
Noted.
Overall, we tested well and the positives rang out louder:
- People understood the concept and valued the proposition
- The site design was applauded
- The publishing app was lean and easily stood up to the pressures of real use
People were impressed by what GDS and the 10 participating departments had achieved and wanted to see more. No doubt, there is development to do but this thing can work.
Crucially, by conducting these tests we now have benchmarks against which to measure the performance of future releases of Inside government. This data was not previously available to the Inside government team in the way it was for those involved in the other betas to replace Directgov and Business Link.
We will plough the learning into forthcoming iterations and we will continue to run testing regularly and report back on the findings.
If you want to tell us about your experience of the Inside government beta, leave a comment or drop us an email.
Ross Ferguson is a Business Analyst for the Inside government project.
