Scientific journals/reviewers routinely - and for good reasons - ask for reliability statistics for the research instruments. While for traditional questionnaires this is easy to do (usually Cronbach's alpha), it gets very complicated when the psychological construct under consideration is complex. We demonstrate this for the data we collected using "performance assessments" that measure critical thinking in college students. It is misleading in this case to merely report the interrater reliability as it is most commonly done.