Self review guidelines
For an organisation to grow and improve the quality of its services it is important it monitors its performance. Ngā Arohaehae Whai Hua/Self-review Guidelines for Early Childhood Education were created with this in mind.
Section 2: The review process
"Whatu ngā whenu a takapa, kia tāpui, kia ita.
Weave the strands of the takapa so they may become intertwined and strong."
This whakatauākī emphasises the importance of the strands of the takapa being woven together in a way that will make them strong and enhance the mat. In the context of review, following a process can support us to engage in review that is relevant and empowering. As we work through this section, we can ask ourselves, “How effective are our review processes in providing evidence that we can use to inform and transform our practice?”
Preparing, gathering, making sense, and deciding form the basis for review. Like a weaving, they overlap and interlink, but they are all important parts of the process. We move back and forth between the parts of the process according to the pattern of our curriculum whariki - which aspects of our practice we want to look at in review and why.
A review story from Aratika Educare (fictitious name for this service) illustrates the process of preparing, gathering, making sense, and deciding. Their review took place over a three-month period.
Preparing - Te whakarite
In preparing for review, we want to be clear about what we are looking at and why. Preparing for review involves:
- identifying the trigger for review
- developing a focus
- establishing indicators to guide us
- identifying sources of information
- preparing a plan.
Review can be either planned or spontaneous.
Planned review can be triggered by:
- our review schedule or plan
- an Education Review Office recommendation
- an outcome of a previous spontaneous review
- an issue arising from a previous planned review that prompts us to look at an aspect of our practice that sits outside our schedule
- our vision and strategic planning goals.
Spontaneous review can be triggered by:
- an incident
- a comment
- an event or issue
- a question that we want to answer
- an observation
- a new piece of information that challenges what we do now.
Whether a review is spontaneous or planned, preparation requires us to prioritise aspects of our practice. Once we have identified which area of our practice we want to target for review, we can develop a review focus. (Three areas of practice that can be reviewed are outlined in Section 1 under 'What do we review?')
A review focus can be expressed as a question or statement that helps us to be more specific about what aspect of practice we want to look at. This focus continues to be at the forefront of our thinking throughout the review process and helps us to target our findings.
Questions can assist us in developing a focus for review.
The principles of Te Whāriki can be used to frame questions. The reflective questions from Te Whāriki can be utilised in a similar way. The Education Review Office also offers some useful evaluative questions to help us focus a review (see Framework and Resources for Early Childhood Education Review(external link)).
There are a number of publications that assist early childhood education services in evaluating practice. There are specific documents, such as these guidelines and a wide range of books and articles that can be accessed to support us in preparing to review by offering different methodologies to try. These can include research texts, which can be adapted for review.
How will we be clear about what we want to find out? - Me pēhea tātou e mōhio ka tika, ki tā tātou e hiahia ana?
In preparing for review, we select or develop indicators that allow us to shape our information gathering. Later in the process, we can check the information we have gathered against the indicators to help us make judgments.
Indicators are usually expressed as a statement of what we would expect to see. They are written using active language, such as “Educators are …” or “Adults respond …”. There can be more than one indicator for a review focus. We can select indicators that link to our vision, goals, and priorities if we want to provide a strategic element to review (see Section 3).
The process of developing indicators as we prepare gives us an opportunity to talk to one another about what is valued and why.
Indicators work best in review when they:
- focus on what is important
- can be observed
- are easily understood by everyone involved
- facilitate reflection.
There are indicators already available that can be used to guide our review:
- evaluation indicators from 'Evaluation Indicators for Education Reviews in Early Childhood Services'
- performance indicators from 'Professional Standards for Kindergarten Teachers'
- indicators from 'Evaluation Indicators for Education Reviews in Kohānga Reo(external link)'
- a range of indicators from 'Booth and Ainscow's (2004) 'Index for Inclusion: Developing Learning, Participation and Play in Early Years and Childcare'
- the cultural input exemplar from Bevan-Brown's (2003) 'The Cultural Self-review: Providing Culturally Effective, Inclusive Education For Maori Learners(external link)'.
Each set of indicators offers a way of looking at aspects of practice in review.
We can also develop our own indicators for review. In doing so, we ask ourselves, “What would this aspect of practice look like if it were effective?” The benefit of developing indicators for ourselves is that they are grounded in the knowledge and skills that are valued within our learning community.
In preparing, we identify what information we need to source for review. The information we identify will be determined by our review focus and associated indicators. There is a range of records, resources, and processes available that will enable us to gather the most relevant information. These include records kept within the service that can be accessed as potential sources of information, for example, meeting minutes, children's profiles, or other assessment records. When we consider using this information, we ask, “What do we already know, or think we know, about this review focus?” and “What sources of information already exist that can help us learn more?”
There are also processes of enquiry that we can utilise to provide information about aspects of our practice. These include face-to-face meetings (such as hui, fono), interviews (such as telephone, individual, or focus-group), questionnaires or surveys, and observations of one another. When we choose one of these processes, we are intentionally seeking new information that can shed light on aspects of our practice. We ask, “What more do we need to find out about our practice, and how will this information help?”
We can also use resources to present aspects of review.
The most effective sources of information in review are:
- relevant for our service
- relevant to our purpose in review
- able to get "under the surface"
- easy to use
- easily accessible.
Several potential resources that can support review are outlined in Appendix 2.
A review plan can assist in organising our review. Not only will a plan make the review accessible for everyone, but it also provides a means of clarifying responsibilities and processes. A review plan has a place even in spontaneous review. Taking the time to stop and plan helps us to maintain our focus.
The following plan was used by Aratika Educare and is provided as a template for us to use or adapt in Appendix 2:
What is our review focus?
How effective are our communication and consultation processes with parents/whanau?
What indicators will we use?
- Teachers are communicating with parents about their children's learning in consistent, authentic ways and at regular intervals.
- Parents have multiple opportunities to contribute to their child's learning at the centre.
- Teachers are available for parents when they need to talk about their child.
- Parents regularly seek opportunities to share information about their child's learning.
Who will be involved?
Five teachers and five parents from the under-twos and the over-twos areas, as well as the Centre Director, will form a review team. All parents will be invited to complete the survey. Children's voices will be heard through their profile records.
How long will we take?
The review will take place over a 2-month period.
When will we start?
What resources will we use?
Children's profiles - the review team will investigate the content of these by asking questions such as “Was there continuity from teachers?” and “How much were parents contributing to the profile?” We planned for the teachers on the team to do this investigation and to report back on their findings without referring specifically to individual teachers or children.
A survey in which parents will be asked how satisfied they are with communication between management and staff and with the way that information about their child's progress is communicated to them.
How will we document this review?
The review team will record the review process in a report format that will then be analysed at a centre management meeting. The results will then be presented to the centre community in a booklet.
Gathering - Te kohikohi
Gathering involves collecting together information that will help us to explore our review focus. The quality of the information we gather is more important than the quantity. We want to make sure that the information we gather provides us with the evidence we need to evaluate our practice in terms of what works well and what doesn't, thus providing direction for change.
How will we find out? - Me pehea tātou e whai mohiotanga?
We gather together the information we need by using the records and/or the processes we identified in preparing.
For example, we might seek feedback on the effectiveness of a particular policy or teaching strategy by asking others, or we could collect information by observing one another. We might also look through our daily diary to see the sorts of comments parents make about a particular issue, or we could revisit children's profiles to check that we really do involve children in documenting their own learning.
There are two main ways we can gather information.
We can gather it quantitatively by counting or measuring number-based information. Processes such as questionnaire scales, multi-choice questions, and duration or event recordings of observations can assist us in gathering information quantitatively.
We can gather it qualitatively by discussing and recording ideas or comments. Processes such as open-ended questions, recorded conversations, or narrative recordings of observations can assist us in gathering information qualitatively.
When gathering information, we draw on a range of information sources to ensure that we have a sound evidence base from which to make judgments about our practice. In gathering information, we will want to consider issues such as:
- Are we sure that this information will give a fair representation of this aspect of practice?
- Do we have confidence that this information will provide evidence to inform our judgments?
We monitor the gathering process to make sure that the information we gather and the way we gather it represents an authentic view of our practice. It would be unwise to gather information that was incomplete or flawed in some way. We want to make sure that the time and place where information was gathered represents an ongoing reality rather than a one-off event. For example, Aratika Educare could have chosen to survey only the parents on their management committee or to select profiles from the most experienced teachers. However, in doing so, their selection would not have represented the views or practices of the whole centre. Section 3 provides further examples of issues to consider when gathering valid and reliable information to use as evidence for review.
Making sense - Te whai matauranga
Making sense involves us in a process of analysis. We scrutinise each aspect of the information we have gathered in order to create meaning. In doing so, the information becomes a source of evidence that informs our review and allows us to make judgments about our practice.
This question leads us to look closely at each piece of information we have gathered. We want to know what it can tell us about our practice in relation to the review focus. We look for:
- issues that come up again and again
- emerging trends or patterns
- anything seemingly insignificant that we need to be careful not to overlook
- one-off or unexpected pieces of information.
There are approaches to analysis that we can select from in making sense of the information we have gathered.
A descriptive approach allows us to make sense of information we have gathered qualitatively by:
- looking for themes or patterns in clusters or groups of ideas and issues in our information
- looking for contradictions where we have pieces of information that seem to be inconsistent with the rest
- using predetermined categories where we have a specific purpose in mind and will have designed our gathering accordingly - this is a targeted approach and may have links to conceptual frameworks or indicators that we have developed as part of our preparation.
A descriptive approach to analysis allows us to draw conclusions from the information in a wide range of ways. We may find exceptions or challenges that we have not previously considered. If we are responsive to such findings, we have the opportunity to explore new ways of looking at what we do and why.
A conceptual approach allows us to interpret our information in relation to existing key ideas or theories (or statements that we have developed through indicators). There are several conceptual frameworks that we can access. These are usually identified at the beginning of the review process. For example, the Child's Questions (developed by Carr, May, and Podmore, 1998) is a conceptual framework provided by Te Whāriki.
The Focuses of Analysis is another conceptual framework based on the work of Barbara Rogoff (2003). Templates for both are provided in Appendix 2. In the case of Aratika Educare, the reflective questions in Kei Tua o te Pae provided a conceptual framework for analysis of the profiles.
A numerical approach allows us to work with quantitative information. It can involve counting how often different episodes occur, comparing sets of figures and interpreting the meaning of clusters of numbers in questionnaires, surveys, or observations in which scales or measures have been used (McMillan and Meade, 1986). A numerical approach to analysis allows us to make comparisons between pieces of information. Combined with descriptive or conceptual approaches, a numerical approach to analysis can provide a strong evidence base to inform our judgments.
Each of the three approaches to making sense of information (descriptive, conceptual, and numerical) allows us to look at information in different ways. We may choose to use several different approaches to make sense of the same piece of information. Alternatively we can use one approach to look at a range of information.
Making sense of information in the review process provides us with the evidence that will inform our judgments. As a result of the analysis we have completed, we are in a position to identify which aspects of our practice are positively impacting on children's learning and which aspects of our practice need improving. We draw together all of our analysis.
How do we make judgements? - Me pehea ta tatou whakatau?
The process of making sense of the information we have gathered provides us with the evidence we need to make sound judgments about our practice.
- Information - The information is what we have identified, gathered and now analysed
- Evidence - The evidence is the analysed information we use to inform our judgements
In bringing together our evidence, we now ask:
- What aspects of practice are we doing well?
- What aspects of practice might we need to improve?
Making judgments about what we are doing well and what we might need to improve is often quite challenging. It is a collective process in which the emphasis is on sharing responsibility rather than on apportioning blame. We want to be sure that our judgments are informed by the evidence that has been generated from the review process rather than based on speculation or a hunch.
Our ability to reach effective judgments through making sense of the information is underpinned by the elements of effective review. These are:
- the strength of our relationships in challenging and listening to one another
- our ability to use the evidence to inform our practice
- the extent to which we have a shared vision for our work
- the commitment we share to improvement
- our ethical awareness that contributes to everyone feeling safe
- the wisdom we share in making sound judgments.
(See Section 3 for a discussion of these elements.)
Sometimes we identify other aspects of our practice that need consideration as a result of our making sense of the information that we have gathered. Alternatively, we discover that more evidence or careful scrutiny of all the information is needed before judgments about our practice can be made.
Deciding - Te whiriwhiri
Deciding requires us to consider what we should do as a result of what we have learned about our practice and to act accordingly. It involves:
- deciding on what needs to happen, based on our judgments
- planning to implement and monitor any changes that arise
- sharing the outcomes of our review.
We are responsible for the decisions we make and the actions we subsequently take in review. Our decisions are informed by the judgments we have made. Knowing what we do well and what we need to improve, we can then plan to bring about improvement. As a result of review, we may have to do things differently, learn some new ways, or let go of practices we were comfortable with. We may develop some recommendations to guide us as we move forward or take immediate action.
Through the review, we also learn about the effectiveness of our review process in helping to evaluate our practice. We have an opportunity to consider which aspects of our review process worked well and which need further development. These findings lead us to consider ways in which we can improve our future reviews:
Developing a plan is an important aspect of review because it helps us transform our thinking into action. We act on decisions we have made. To do this, we have to come up with practical solutions that will address any issues raised through review and/or take us into another review.
We want to be sure that our resulting actions are both reasonable and practical. If there are resourcing or budget implications, we will need to explore these before we make plans that cannot be realised. In addition, we want to ensure that we identify the changes to practice that arise from the review and monitor these over time. Ultimately we are concerned to ensure that any changes that we make to practice have a positive impact on children and their learning.
Ongoing monitoring of our plan helps us to know the impact of our actions on practice over time. We can do this by revisiting our plan at regular intervals to check that agreed actions have been taken. We may also want to undertake a subsequent review to find out the extent to which these changes are contributing to positive learning outcomes for children.
Sometimes, through review, we identify a need to make a change that is likely to have a significant impact on our service. This will require us to look at how we might manage this sort of change. It is important that we also recognise and celebrate our successes. Celebrating helps us to balance the challenges that come with change.
The nature of our review will determine who we share the outcomes of our review with and how. We ask:
- Who do we need to share with?
- What will we share?
- When do we share our review findings?
- How will we share our findings?
We consider the best ways of conveying the most important outcomes of the process. This will differ for each review. Some services have specific ways of sharing findings using templates, while others develop approaches that suit their particular context. For example, Aratika Educare chose to present their findings in the form of a booklet:
- What resources did we use to help us? How did the resources help us? Are there other resources that we could explore for next time?
- How prepared were we? What more could we do to prepare next time?
- What indicators did we access/develop, and how well did they help us to focus our review? Did we draw on them in our analysis?
- Did we gather the most appropriate information? How could we develop our review process to support this?
- What kind of analysis did we do and why? Is there another approach that we could consider that would provide a wider perspective that we hadn't thought about?
- How well did we draw on our analysis to inform our judgments?
- To what extent does our evidence support our review findings?
- What consideration did we give to sharing our decisions?
- How will we plan for actions to take place as a result of our review?
- How will we ensure that these actions take place?