Making a case for quality evaluation in youth work

Introduction

If quality evaluation would be an iceberg, what I am doing in this article is just to offer you a map of where its visible part is. I do not write this article from the perspective of an expert in quality evaluation in youth work. I am writing it from the point of view of a research-minded person who is also aware, at some extent, of the grassroot realities in youth work in different countries across Europe. This is why the article will not have a lot of references to theoretical or empirical work in the field and takes the form of an opinion.

My aim is not to ‘teach’ readers how to implement a quality system, but to make a case for starting the discussion about measuring the quality of the youth work we do at different levels and to offer some directions for this. I am aware this is not an easy task and there is a lot of learning that needs to happen if we have never done it before in a systematic way. Nonetheless, I strongly believe that we need to always question the quality of our work in general and to find strategic ways to assess it for the sake of our own development. Accepting there is room for improvement and embarking on this journey for continuous learning will make us feel our working time is invested well and will give us a sense of direction.

Briefly about youth work

Before starting the discussion about quality in youth work, I think it is important to clarify what we mean by youth work and what are its core features. As defined by an EU Expert Group in the field, youth work is represented by “actions directed towards young people regarding activities where they take part voluntary, designed for supporting their personal and social development through non-formal and informal learning” (p.12, European Commission, 2015). Its core features are illustrated in Figure 1.

Quality in youth work

Quality refers to the degree to which something fulfils its function. Therefore, taken together, quality in youth work refers to the extent to which youth work fulfils its function of supporting personal and social development of young people through voluntary participation in activities based on non-formal and informal learning.

Discussion about quality of youth work should have as starting point the overall aims and the core features of youth work. This is important because, due to various reasons, we can get side-tracked into focusing on more general results such as “more young people in education”, “lower crime rates among young people”, “more people doing voluntary work” which can be outcomes of quality youth work but should not be the focus!

The simple reason for it is that they are outcomes of the interplay between many more factors and sectors than youth work alone. The focus of good quality youth work is to provide spaces for personal and social development for young people and when talking about quality of youth work we should be looking at the degree to which we succeed to provide such spaces bearing in mind the core features mentioned previously.

Lovely indicators and the magic quality circle

Quality assurance, evaluation, assessment, criteria, system, indicators, standards, did you fall asleep already? What if I use research, purpose, values, meaning, change, impact, curiosity, development, learning? Better? Maybe some are even more comfortably sleeping after this second batch of words. I know the topic of quality evaluation can seem dry or boring if you are not patient enough to understand its true value. But take a coffee and let’s try to break this wall.

The main point is that it doesn’t mean you cannot do good youth work without indicators, but you could neither prove it, nor know whether the quality has improved or not. I can list many reasons for taking the time to develop a system to evaluate the quality of your work, but here are a few:

1. To gain an understanding of what you are doing from different perspectives.
2. To learn about what you are doing well and what you need to improve.
3. To be able to improve your work in a systematic way and check whether the changes you implemented are working.
4. To show to external bodies the outcomes of your work.

When going a bit more in-depth with what quality evaluation is, we discover that for every objective we set, there are a few elements that we need to ‘attach’ indicators to in order to understand how well we are doing things and where we need to improve:

1. Preconditions (i.e., what needs to be there for the objective to be tackled such as budget, facilities, youth worker competences).
2. Work processes (e.g., how we set aims, how we map youth needs and interests, how we document our work).
3. Outcomes which are quantitative (e.g., number of participants, of events, of hours) and qualitative (e.g., attitudes changed, skills developed, knowledge gained).

If we make a quick stop at the “outcomes”, it is important to mention that youth work is (too) often described in quantitative terms: how many young people were reached, how many events organised for them, how many hours worked. Because of the variety of youth work activities and the approach and context of youth centres and youth NGOs, these quantitative indicators are sort of useless if you want to look a bit further than numbers or when you are interested in the actual change made. But this happens often and we know we all do it, sometimes for the very reason that it is rather easy and it looks nice and neat.

In contrast, trying to capture something less quantifiable, but I would argue much more important, can be challenging. That is why having qualitative indicators in place to look for BEFORE starting any kind of activity is so crucial, not only for the external stakeholders, but also for internal checks. Qualitative measures would look for perceived experiences and feelings of young people, changes in their attitudes, developed soft and hard skills and gained knowledge.

The figure showing the quality circle suggests some basic steps to develop a system for quality evaluation, to which I have added some pre-steps with questions for reflection:

1. Define your youth work: What kind of activities do you do that can go under this umbrella? Try to really think this questions through if you are doing a variety of activities; maybe not all of them are youth work and evaluating them as youth work will not make a lot of sense.
2. Connect these activities with the core features of youth work mentioned above: Which ones would you like to focus on? What indicators will you be looking for? Design your evaluation plan alongside your activities, not at the end: What tools will you use to gather your data? Who will do it?
3. Analyse the information you gather: Who will do it? Who will you discuss these results with?
4. Plan the change: What will you do once you get the results? What would you like to change?
5. Action plan: Who will implement these changes and how? How will you track the changes you planned?
6. Gather new data: You are back in the loop at point 3.

Awareness about different quality evaluation tools and how the quality circle works is very much needed for organisations to ensure they themselves understand the nature of their work and how to improve it, but also to be able to show it to external parties. However, there is no one-fits-all solution for the quality system and the job of each organisation is to carefully choose their own compilation of quality tools depending on their desired outcomes and based on their resources and realities.

Start where you are (& other clichés)

We are aware that the reality varies a lot in different countries, in different organisations and so on. The differences are manifold, but I will focus on attitudes towards evaluation in general and the actual resources invested in youth work which I think can pose some challenges in being able to implement a quality system.

Attitudes. Arguably, because of our previous experiences, evaluation rarely has a positive vibe associated with it. For me it was either that I would get a bit scared of being assessed and marked on something or that I used to find evaluation rather boring, especially when it comes with a table of indicators and bullet points of abstract stuff. But, as mentioned by Chelimsky (1997), evaluation takes place with three main purposes:

1. For accountability: to measure results of a process and how efficient it was, for instance.
2. For development: to provide help to strengthen and improve a process.
3. For knowledge: to gain more in-depth understanding about a process.

By looking at these, we can easily see that not being that open towards evaluation is related to having an unbalanced focus on its accountability purpose. Indeed, Ellis (2008, cited in Cooper, 2014) reported that people working in the voluntary sector hold the belief that evaluation is rather for the funders and regulators and not necessarily for their own benefit. Cooper (2014) emphasised that the challenge is to encourage practitioners to engage with the evaluation process as “researchers of their own practice” and not as “data collectors”. The point here is that there is a clear need to shift the view on evaluation from something done mainly for external bodies towards its learning and development purposes for those who are actually doing the work.

Warning! We should be aware that on this path we might find things we do not necessarily like – we all like to think that we do wonderfully, but being prepared to find out we might not do so well in some areas is super important. The motivation for evaluating quality should be related to a desire to improve which implies that we will need to change our old ways and remain open for continuous learning. No matter what your experience with evaluation is, it is never too late to start building a learning culture within your organisation. With this in mind, when leading an evaluation process, experts advise us to be:

Curious; Open-minded; Humble; Constructively critical; More interested in saying what we do not know than demonstrating what we know; More interested in asking questions than giving answers; Aware that no one knows better, but that we know different things; Focused on solutions, not on who provides them.

Limited resources. It is understandable that when the resources allocated for youth work are limited and it is often that the work is done by unpaid volunteers, the task of managing an ongoing evaluation process in parallel with the actual work is rather daunting. It is also important to acknowledge that resources allocated to youth work in different countries vary a lot and while in some countries this is mainly done on a voluntary basis, in other countries there is more support from local/national governments as well as from other sectors. This has implications for the level at which we are able to do the work and different barriers we might face that will require extra energy to overcome.

However, before rejecting the idea of thinking about the quality of our work when overwhelmed with many other daily tasks, we should reflect for a bit at the bigger picture and our WHY. We are most probably doing this work to provide opportunities for the personal and social development of young people. So knowing whether our work is achieving its objectives or not and how we can improve is something that should be of great interest.

After acknowledging all these, I would argue that creating a quality system is necessary if you are interested in the value of the time you are investing in youth work. And if you are reading this article, I assume you are. So here you go, start where you are! I can go on with a lot of clichés: If you were waiting for a sign, this is it. No better time than now to start thinking about how you will evaluate the quality of the youth work you are doing. If you have never done it, start thinking about it, do a bit of reading (from the resources linked at the end of the article), reflect on what is the nature of the youth work you do and what would you like to focus on from the core features mentioned earlier. Start small and try gathering some data and implement small changes based on what you discover.

If you are measuring the quality of your youth work already, start thinking about systematic ways to support change based on what you find. Do you have an overview that includes all different aspects of youth work? Are your tools the most appropriate to use for your purposes? Start to optimise. If you already have a quality system in place, first of all well done, you’ve come a long way! But that is not a reason to stop improving, you can go into… meta-evaluation which means you can start thinking about the quality of your quality system. Are you doing it in the most efficient way? Are you learning from the evaluation process itself as well? Do you involve all stakeholders (youth, youth workers, project managers, policy makers, funding bodies)? If yes, wow! But then, you need to teach us, the rest, on how to get there! So start where you are and never stop, because that is the beauty of this never-ending journey.

Author of the article
Carmen F. Ionita, MSc
PhD student in Psychology at the University of Manchester
International Project Manager at “Un Strop de Fericire”
Contact: carmen-ionita@hotmail.com
LinkedIN

Download the article in PDF (with references and useful resources)