Carteret Community College Title III Grant

Aggie Bonfire Memorial

(Texas A&M…a marker for each student who lost her/his life, positioned so that they are looking toward that student’s hometown)

9th Annual Assessment Conference
Texas A&M University
Feb 22-24, 2009
Conference Title:  Using Assessment to Demonstrate Improvement

I’m off to College Station, Texas to take part in this well-known, well-established conference on assessment in higher education.  I am really looking forward to it because of the array of sessions that have direct relevance to the assessment activities that we are putting in place (through Title III) at Carteret Community College.  I’m going to use this space over the next couple of days to report – both during and after – the sessions I am attending.    With a look at the agenda, here are the sessions that I plan on attending.

Sunday February 22
Pre-Conference Workshop (2:00-5:00)
Program Assessment: What to Know, Do, and Avoid
Instructor: Dr. Susan Hatfield; Winona State University

Why I’ve signed up for this workshop –
The brief description in the conference program promotes the workshop as a review of “best practices and bad ideas in program level assessment,” with an opportunity for participants to “consider what to assess, how to do it, and what it all might mean.”  This description is attractive to me on a number of levels.

First, I have spent the last two years working on the Program Review process at CCC, and a lot of it has been learn-as-you-go.  This workshop will (I hope), give me an opportunity to explore this process (most likely in part, not in whole) in retrospect in a setting where ideas are shared by people from across the country who are in my shoes.

Second, if it only focuses on Program Outcomes, that’s OK as well.  This is still a critical piece of the Program Review process, and it will play a larger role as we focus more creating a logical connection between program outcomes and funding.

Finally, I am looking forward to this workshop because, as I mentioned above, I am keen on hearing how other schools are approaching this task – how they are defining Program Outcomes, how they are assessing them, and how they are meeting the challenges inherent in establishing such a systematic program.

My post-session impressions –
This was not as much about Program Review as it was about program level learning outcomes (PLLOs). Indeed, she spent a few minutes arguing how PLLOs were not program assessment [I can agree with the not program assessment part, but she did make it sound like never the twain shall meet…which I believe is inaccurate…PLLOs can very easily be a critical component of the review process…but that’s an argument for another day].

The workshop presenter laid out a coherent description of why PLLOs, how to write PLLOs, and how to assess them.  She talked primarily from the perspective of one who must inspire faculty to action regarding PLLOs, including making persuasive speeches to faculty regarding the connections between, for instance, his/her activities and the program’s need/desire to receive funding for a particular project.

We also talked a lot during this two-hour period about writing useful/articulate/concise outcomes.  What Susan spent a good deal of time on was curriculum mapping and rubrics.  She stressed the need to ensure that Program student learning outcomes were taught and assessed across the program. In connection to this, she talked about orphan outcomes or orphan courses – in either case, the PLLO is not assessed in a particular course (vertically – when looking at the completed map) or across a program (horizontally).  We are doing this to some extent, vis a vis the Gen Ed outcomes (ILLOs), and under Dr. Emory’s leadership and tutelage.

For me, there were two important take-aways from this session: 1) the discussion on rubrics.  Susan talked about different types of rubrics (e.g. Analytic & Holistic…summative and developmental), but she also gave us a great resource, which is the link to her office’s website that has a great repository of rubric samples:
http://www.winona.edu/air/rubrics.htm [While you’re at it, check out their Resources page…a little bit of everything IE, whether you are IE or are a part of the process:
http://www.winona.edu/air/resources.htm ].

2) The second take-away was, as it always is at conferences (which is why we go in the first place), the others at the round table.  The 200(?) people in the room sat at round tables, where they had randomly chosen to sit.  There were folks from Virginia, South Carolina, and Texas at our table. Rich discussions around all of the talking points took place. But the real gem for me was meeting the Assessment Specialist from the Office of Institutional Effectiveness at the University of Richmond (VA).  Ashley showed me a copy of their Assessment Workbook (and told me how to find download it from their website: http://oir.richmond.edu/Assessment.htm (it’s under “Assessment of Academic Programs”).   Their plan is similar to the one we’ve developed, but I like the extra-added dimension of dissemination of findings.  In other words, let’s not let this sit on a shelf…let’s have a discussion.

Below is a list of the sessions I intend to attend for the rest of the conference.  I’ll be checking in periodically and filling in the spaces with information about the sessions, along with my thoughts and Take-Aways.

Monday February 23

Plenary Speaker (9:00-10:30)
Mary Allen,
Prof. Emeritus, California State U.
Former Director, CSU System Institute for Teaching & Learning

Topic:  Assessing General Education Programs

Description: designing and implementing effective General Education assessment programs.

Powerpoint and handout can be downloaded HERE.

Dr. Allen provided an engaging, 1.5 hour presentation on developing an effective Gen Ed assessment plan on campus.  She provided many definitions around assessment, talking about Program Assessment, learning outcomes, the curriculum map/matrix, assessment plans, and rubrics.

Much of Dr. Allen talked about was familiar, and on the surface, not new.  However, for each of the areas she spoken on, she would add an interesting perspective or dimension.  When she talked about assessment (formative vs. summative) of learning outcomes, she framed it as a focus on “deep and lasting learning” versus “shallow and short-term.”  She stressed that assessment plans be: “Meaningful, Manageable, and Sustainable.”  The meaningful  and manageable seemed to be the thread that ran through the conference as most of the sessions addressed the need for buy-in if an assessment plan were to be successful.  Dr. Allen really emphasized these two points in order to make the process sustainable.

Other important points she made (for me, at least) were:
• Assessment plans should be multi-year and reasonable (again, the notion of creating buy-in).
• Assessment plans should be perceived as “a series of incremental improvements to the program.”  This is critical because often people want to large gains, quick.  Slow and steady wins the race was her message here.
• And the take away message for me was: When asking faculty to help read and score papers (for ILLOs, for instance), keep the numbers of papers low, get them in a room for an afternoon, train them, have them score the papers, discuss results (and use of) and let them go.  Get it done in the shortest time possible, and make sure they feel that they have done something meaningful.
Session 1 (10:45-11:45)
Ending the Paper Chase: Moving from Paper to Electronic Assessment Reporting

SACS Roundtable
12:00-1:30
General Education Competencies: Reviewer Expectations

Session 2 (1:30-2:30)
Building a Technology-Supported Culture of Assessment: Software Readiness Considerations

Session 3 (2:45-3:45)
Vertical Assessment: From Assent/Ascent to Descent/Dissent

Session 4 (4:00 – 5:00)
Achieving Institutional Effectiveness with a Multi-Layer Strategic Planning Process

Alternative:  Assessment to Promote and Sustain Online Programs

Communities of Practice Dinner
Classroom & Program Assessment

Session 5 (8:00-9:00)
QEP

David Carter of SACS, spoke on the QEP.
He talked primarily about the pitfalls that schools experience with their QEP plans.  I thought this one-hour presentation was both useful and interesting.  Particulary from the perspective of a school who is submitting it’s QEP plans within the next week (we hope!).

You can simply check out his powerpoint slides at his website.  Or, if you want the full multi-media experience, I did record his talk.  You can listen to it at the CCC iTunes site (or download it to your ipod and listen to it during your next jog).

To get the audio, go to:  The CCC iTunes Site
Scroll down to the Title III repository (under CCC Community).  Click on the Title III Grant icon.  Then, click on the Title III Grant icon again and it will take you to a list of audio files.  The last one, the “QAP”  is the one you want.  Click on it to listen, or click on “get” to download it to your computer.

p.s. if you go Dr. Carter’s website, you can download his lunch time plenary that he delivered on the day before on Gen Ed Competencies.  I don’t have the audio for this one.

Plenary (9:15-10:30)
Gloria Rogers
Using Assessment to Drive Improvement without Driving Faculty Crazy!

NC3ADL presentation ppt

Click HERE to download the power point.

Carteret Community College offered its first online course in the spring of 1998.  In Fall 2008, the college offered 61 sections of full internet courses and 15 hybrid courses. [By Fall 2009, all courses at the college will have a web presence].

CCC is not different from most colleges with burgeoning distance learning programs. For most of the last decade, with corresponding increases in the number of distance learning courses and students, attention has been trained on providing faculty with the skills necessary to deliver effective online instruction [via Blackboard]. In particular, in 2006, the college was fortunate to have been awarded a 5-year Title III grant from the US Department of Education.  Of the three primary objectives for the grant, distance learning is one of them.

This grant has given the college the resources to train all full and part-time faculty to effectively teach online.  Faculty are certified to teach online through a three-module Blackboard Bootcamp, developed in-house.  This professional development provides instructors with the tools to design and develop online courses, including the utilization of rich-media content (e.g. podcasts, enhanced podcasts, teachertube videos, etc…).

The missing link thus has far has been an eye toward assessment of online delivery. While assessment has taken place by the instructors in the courses, we have not really systematically examined online delivery as an effective means for providing instruction (i.e. are students learning). [I’ll digress for a moment here to say that we are only looking at Full Internet and Hybrid courses…i.e. not the so-called Web-Based courses – primarily because there is such variation on what a WB course entails in terms of online instruction/learning]

This presentation is a description of our attempt to do just that.   I begin with some background regarding DL, particularly the quantitative information (enrollments, success rates, etc…).  I then talk about the four primary ways that we are looking at DL courses:
1) The Quality Assessment Plan: a peer-evaluation process of online courses before they are even allowed to “go live.”  (download the QAP by clicking here:  http://web.carteret.edu/keoughp/TitleIII/FinalQAP.xls)
2)  Grade Distributions: This is a broad-brush approach to see how students fared in any one semester (by way of final grades…i.e. “success”  and D, F, W rates).
3) ILLOs & PLLOs: How did students do on specific assignments that were related to either Institutional Level Learning Outcomes or Program Level Learning Outcomes.  For instance, how did students in both online and seated sections of CIS 110 (the course in which we assess our ILLO for Computer Literacy) score on their final class project?
4) Same instructor. Same course.  Teaching both seated and online sections of that course. We looked at common assignments for both sections and examined how students performed.

The Bottom Line
While there were some spots where there was a significant difference between how students performed in an online setting vs. a traditional/seated setting, these were probably more one-offs than evidence of a trend. To me, the most intriguing pieces to look at will be  A) Hybrid courses – Full internet courses for the last three fall semesters have seen an almost consistent 64% success rate.  Hybrid courses, on the other hand, have seen 80% success rates.  Should we be doing more Hybrids (instead of full onlines) is the obvious question.   And B) We need to drill deeper.  What student characteristics can we use as predictors of success when it comes to online instruction.  Should every student be allowed to take DL courses?  Yes.  But, should we look at which ones students should be advised to Not take, or should there be policies around the number of DL courses a student can take at a time…or the number they can take the semester after dropping a DL course(s).  These are issues that we need to explore further.

(click HERE to download ppt)

In 2006, the Title III – Distance Learning team began making plans for offering student services online.  There were a number of key factors driving this decision.  First, there is the issue of accreditation.  As distance learning becomes a more integral aspect of education, it is garnering greater attention from accrediting bodies (in our case, SACS – the Southern Association of Schools & Colleges).  SACS  mandates that, “students have adequate access to the range of services appropriate to support the programs.”  For this reason (and the fact that we are up for reaffirmation in 2009-2010) inspired us to look closely at developing online services, accessible to all of our students (and not just online students).

But, we believe that the more compelling reason to begin exploration of online services is because it’s the right thing to do.  First, the number of DL students is increasing all the time.  So, yes, we need to give them access to the same services that all students who come to campus can access. True, a great many of our DL students come to campus anyway, which leads us to the second reason for developing online services.  Our students are changing.  This may be a rural community, but a great many of our students are exhibiting many of the characteristics that define the so-called millennial generation.  One of these traits is a blurring of time and space.  Students (of all ages, really) are not space bound (i.e. learning takes place as much off campus as on), nor are they time bound (i.e. just because the class or day is over does not mean that they don’t want access to the college at any time they have a free moment, no matter what time it is).

Finally, we are driven by cold, hard facts.  Through grades and attrition, the numbers tells us that our students need a broad support net to ensure their success in class, and throughout the whole college experience.  Therefore, we are working hard to develop and put online services for tutoring, counseling, career coaching, and financial aid assistance.