By Maya L. Kawailanaokeawaiki Saffery, Curriculum Specialist & Assessment Coordinator for
Kawaihuelani Center for Hawaiian Language at the University of Hawaiʻi at Mānoa

At the Kawaihuelani Center for Hawaiian Language at the University of Hawai‘i at Mānoa, our faculty regularly reflect on curriculum and assessment matters. The learning improvement story we have chosen to share tells of how our graduate faculty utilized rubrics and implemented intentional curriculum interventions that led to improved student learning in our Masters (MA) program.

After our graduate faculty collaboratively worked to develop our MA program Student Learning Objectives (SLOs) and mapped our curriculum to these SLOs in 2010, we started on a journey to create a process that would allow us to assess how our students were meeting these objectives at the time of their graduation. Through collaborative discussion, we decided to conduct an assessment of our MA program SLOs through evaluation of our students’ final theses and oral defenses by their MA committees using faculty-developed rubrics.

Our graduate faculty developed two rubrics based on our program SLOs – one for the theses and one for the oral defenses – along with accompanying scoresheets and an implementation process for the collection and review of these pieces of student work. It is important to note here that we intentionally chose the final thesis and defense because our students were already required to create them at the end of their graduate studies and our faculty were already evaluating them, therefore adding a rubric to this evaluation for program-level assessment would not add an extra burden on our faculty.

Once the rubrics and processes were finalized in Spring 2012, we started to implement them. Any time a student defended their research or completed their final papers, their committee used the rubrics to assess their work in relation to our MA program SLOs. After four years of successful implementation, we synthesized and analyzed all our rubric results over that period (2012-2016). We were happy to find out that the majority of students assessed during those four years were rated as “competent” by their committees in all SLO categories and overall for their final theses and defenses.

The findings from this exercise served as the catalyst for productive discussions among our graduate faculty about how to use these findings to not only celebrate our students’ achievements but also make informed programmatic decisions for improved student learning.
A trend we had been noticing over the first four years of our rubric assessment process and identified again in our data synthesis & analysis was that the least amount of exemplary ratings were given to students in the Culture SLO category for both their defenses and final papers (23% for defenses; 10% for papers were rated at exemplary).  The overarching Culture SLO: demonstrate the ability to apply cultural norms in a range of communicative events. While we understand that reaching an exemplary level of achievement is difficult for any second language learner, especially at the master’s level, our faculty did feel that there were practical and plausible things we could do to support our students' ongoing journey to understanding the cultural worldview of our ancestors as expressed through our Native language.

The primary way we worked to address this was to offer our students more opportunities to practice and develop their language skills in culturally-rich contexts. This took the form of developing new courses and then creating a set schedule for offering our MA courses that incorporated the regular offering of those courses in our curriculum that most directly address our Culture SLO. For example, we began offering a new course every year called HAW 653 ʻŌlelo Niʻihau. It offers intensive, advanced study and analysis of the traditional Hawaiian Ni'ihau dialect of Hawaiian through face-to-face conversations with Niʻihau native speakers, listening to audio recordings, and watching video recordings.

Mānaleo or Native speakers of Hawaiian are one of the main sources of cultural knowledge we have left, most of whom are born and raised on the island of Niʻihau. This course was developed and is taught by one of our professors who is also a native speaker from Niʻihau, Dr. Annette Kuʻuipolani Wong. She provides opportunities for students to engage with other native speakers throughout the course through a variety of methods: in-person, virtual, via video and audio recordings, and an annual on-campus event called Lā Mānaleo in which HAW 653 students help to plan, host, and interview active speakers from Niʻihau in Hawaiian about their life, community, experiences, and cultural practices.

After this new course began to be offered every year and we implemented additional interventions based on our four-year data analysis, we saw an improvement in our rubric results. When compared to our overall findings from the first four years of our rubric assessment in 2012-2016, our rubric results from 2016-2020 showed a positive upward trend across all SLO categories. 

In terms of our culture SLO, the percentage of students who were rated at the exemplary level increased greatly:

• Oral Defense: 23% of students in 2012-2016 →  56% in 2016-2020

• Thesis Papers: 10% of students in 2012-2016 →  44%  in 2016-2020

It was reassuring to see that the interventions and programmatic decisions we made were having an impact on student learning.

Through the assessment-for-learning-improvement process, we learned many lessons, including:

• Choose artifacts already being created and evaluated to limit the burden on participants;

• Involve faculty at all levels of the process, including the development of the process and tools, evaluating and assessing student work, and participating in data analysis and decision-making; and

• Revise rubric assessment processes to provide insight for areas that are working and areas that still need to develop further.

At times, assessment can be viewed as an extra task that requires a lot of work with little return value. However, after revisiting, synthesizing, and analyzing our assessment findings over a period of time, it was clear that our graduate faculty could see the value and usefulness of this data in implementing changes that can positively affect our students and program. We look forward to continuing with this process of data synthesis, analysis, and further action towards learning improvement and student success.

Additional Context

Cohort 1 (2012 - 2016) included 13 students. Cohort 2 (2016 - 2020) included 18 students. We assessed all students accepted into our program who had gone through the program and reached the final stage of writing and defending their research. 12 graduate faculty participated in the development and implementation of the rubrics and in the analysis of the results over the course of this particular assessment process.

​

The University of Hawai‘i at Mānoa is a public institution with 20,000 students (2022) and is classified as doctoral, very high research activity (“R1”).

​

Suggested Citation

Saffery, M. L. K. (2022). Improving a Culture Learning Outcome in an MA Language Program. Learning Improvement Community. Retrieved from https://www.learning-improvement.org