PRESENTATION C14

EVALUATING INFORMATION AND KNOWLEDGE SERVICES USING NARRATIVE TECHNIQUES - A CASE STUDY

Nerida Hart
Land & Water Australia, Braddon, ACT

Aims:

In late 2005, the Australian Families and Communities Services (FaCS) Knowledge and Information Service (KIS) was due for a review. It was obvious that the usual techniques for evaluating services being the collection of statistics, did not give a true indication of the value to the client groups (five Australian Government Departments) of the services provided and would not give a true indication of where we could improve.

We had already conducted a couple of projects, including and evaluation of a digital repository project, using narrative techniques and decided that this would be an appropriate approach in the case of the evaluation of the KIS services.

Methods:

It was decided to use both traditional evaluation techniques (i.e. a survey) in combination with a technique which was not as well known. This technique is a narrative technique and uses a methodology called 'anecdote circles' and 'sensemaking' followed by 'intervention design' for continuous improvement. The technique was originally used by David Snowden of the Cynefin Centre in Wales and two of the senior staff of KIS had undertaken training in this methodology.

Results:

The results were very interesting in that the survey showed that all clients were very happy with the products and services provided by the Knowledge and Information Services Section of FaCS (now called Families, Housing, Community Services and Indigenous Affairs).

On the other hand, the narrative techniques, while validating the very good services indicated by the survey evaluation, exposed some gaps in the services. The KIS team members had originally felt that they could improve but needed hard evidence of where this improvement could best be targeted. The 'narrative technique' exposed the gaps and the 'sensemaking' and 'intervention design' gave the team a way forward to make the continuous improvements they were keen to apply. It also gave the KIS team some very sound evidence of the value they added to all their client groups and the work of the five Australian Government Departments.

Conclusion:

When evaluating a service area it is important to apply both quantitative and qualitative methodologies to the evaluation. The richness of the information and knowledge gained from using narrative techniques could never have been gleaned from the survey methodology. It also gave the KIS team members a solid plan for continuous improvement, which allowed for both major and minor incremental actions which were all achievable within the KIS budget.